Port 25: The Open Source Community at Microsoft : HPC, Windows Server

Virtualizing Free Linux Distributions in Windows Server 2008 R2

Peter Galli — Tue, 11 Aug 2009 00:00:00 GMT

Jason Perlow, a columnist over at ZDNet, has written a comprehensive review on virtualizing free Linux distributions in Windows Server 2008 R2.

In his Tech Broiler column, Perlow notes that the updated Hyper-V bare-metal hypervisor virtualization layer in Microsoft's upcoming Windows Server 2008 R2, which is due to be released August 14th to MSDN and Technet customers, now has support for SUSE Linux Enterprise Server 11 and Red Hat Enterprise Linux 5.3.

"Additionally, Linux support and performance has greatly improved over the initial Hyper-V release. Microsoft also recently released its Hyper-V Linux Integration Components (Linux ICs) under the GPLv2 Open Source License," Perlow says.

The Linux ICs for Hyper-V, which are in Release Candidate status, provide synthetic device drivers that enhance I/O and networking performance when Linux OSes are virtualized under Hyper-V.

"The source code for the Linux IC's were accepted into the Linux Driver Project and should become part of the Linux Kernel within two subsequent releases and code merges - 2.6.32 is expected to be when they will be integrated, and all Linux distributions using that kernel code base going forward should be Hyper-V enabled out of the box. Yes, you heard that correctly, Microsoft is now an official Linux Kernel contributor," Perlow says.

You can read the rest of Perlow's column here.

Real Mission Critical

Mark Stone — Mon, 01 Jun 2009 16:15:00 GMT

The 1.0 release of WinBioinfTools might seem like a modest event; as of this writing, the project has
44 downloads. High Performance Computing (HPC) is a small community, granted, and the number of HPC
applications for bioinformatics is a small subset of that. Let's not confuse popularity with importance,
however.

We use the phrase "mission critical" very frequently and somewhat casually within software development. In
talking to a friend about the swine flu outbreak, I was reminded that the phrase has its origin in
military history: an aspect of a mission so critical that failure in that aspect would result in the
loss of life. In the developing world where medical infrastructure can be a fragile thing, information about
the origins or genetic makeup of a virus can be vital. It can be mission critical.

Historically, the developing world has been dependent on developed western countries to do their research for them.
Open source is beginning to level that playing field, though. Using a cluster environment and software
projects like CoCoNUT for gene sequencing and comparison, even university research centers with modest x86
server environments can play in the HPC space. This is important because the research priorities for a
university in a developing country may be very different from the research priorities of a major western
research university. At its best, this is exactly the kind of lowering of barriers to entry that open
source should facilitate.

For all its value, CoCoNUT has two significant limitations. Its license is an academic license, not a fully
open source license. And it runs only on Linux/Unix systems. The latter is particularly important. Research
scientists are not IT professionals, and they should not have to care about the underlying platform on which
their software runs. The spirit of open source is to make software as widely available as possible, and there
is no way to meet that spirit without including Windows Server among the target platforms. Mission critical
demands no less.

So WinBioinfTools makes important steps forward on both fronts. The team at Nile University has released a
GPL-licensed project that "contains a number of programs for Bioinformatics running over Windows Cluster running
Windows HPC server 2008. The current version includes the CoCoNUT system for pairwise genome comparison,
parallel global sequence alignment, and parallel BLAST."

This is a great example of a local software community using open source to make their needs a priority,
and delivering a project that will benefit local software communities in other developing countries with
similar needs. WinBioinfTools puts us one step closer to making scientific computing software platform
neutral, and closer to making Windows Server a first class citizen in the open source world of HPC.

Brazilian Students Set Their Own Course

Mark Stone — Mon, 16 Mar 2009 19:05:00 GMT

I’m going to tell a story that starts in Indiana, but really it’s about Brazil.

Once upon a time “scientific computing” was nearly synonymous with “Fortran”. Today, though, just about any high level language can be used to write High Performance Computing (HPC) applications. These days that language choice also includes C#.

At Indiana University, the Open Systems Lab has pioneered work to implement Message Passing Interface (MPI) support for .Net, so that MPI applications can be written in C#. The project is MPI.Net, and you can find it on Codeplex. It is open source, about three years old, has reached a 1.0 release, and is compatible with two other important open source projects, OpenMPI and Mono. The principle developers behind the project are Andrew Lumsdaine at Indiana University and his former student, Douglas Gregor, who is now on the faculty of Rensselear Polytechnic Institute.

This is the kind of open source work that’s really exciting to see because of the way it expands choices for the developer and the end user. A C# developer should not be closed off from writing HPC applications if that’s what they want to do. And a research scientist should not have to think about whether their lab is running Linux or Windows Server. Both of these individuals are working enough layers above the operating system that somebody else’s operating system choice should not be a constraint.

So I was very excited to learn that students in Brazil at Federal University of Rio Grande do Sul were doing work on MPI, and excited to talk with them about their work. One of their projects is MPI#, also open source and also hosted on Codeplex.

MPI# builds on top of the work of MPI.Net, adding some functionality not yet present in MPI.Net. Specifically, quoting from the project description:

The goals of this project would be to build upon MPI.NET in order to complement it with the features that are missing, mainly regarding collective communication. Either they could benefit from C# native support for such communication, either they could be programmed on top of the provided MPISend/MPIRecv encapsulations. C# and .NET features such as fault tolerance or dynamicity support would be studied, in other to turn the MPI# implementation robust in large, dynamic and heterogeneous platforms.

Two of the students working on MPI# are Ismael Stangherlini and Fernando Afonso. They are graduate students in computer science, working on projects affiliated with the Brazilian Interoperability and Open Source Software Development Nucleous. When I talked to them about their work on MPI# I was curious what their communication with Indiana University had been like. Their response: they had never been in contact with Indiana University; they simply downloaded the code for MPI.Net and started working on their own.

That’s the magic of open source: that they can, in fact, just download the code on their own and start coding against it. They may make an important contribution to MPI.Net. Or their code may be entirely disregarded. Or they may move on to other projects and somebody else may or may not pick up where they left off. At this stage it’s too early to tell. But the fact that all of these scenarios are possible demonstrates why, as a methodology, open source is so nimble and adaptive. A top-down product development process, or a top-down standards development process can only execute on the innovations envisioned by the few at the top, and at the speed of the slowest decision-makers in the process. But a bottom-up open source process enables every innovation that anyone at the grass roots level can see.

Is High Performance Computing naturally Open Source (ie. for tinkerers)?

anandeep — Wed, 18 Jun 2008 13:17:00 GMT

I have always been fascinated by clusters. Some people envision working with desktops or workstations when they think of “working with computers”. For me working with computers was always with a large collection of computers in a back room somewhere. And how cool if you could make all those computers collaborate with each other working to solve cool things like genome mapping, movie special effects, simulations of car crashes or simulations of molecules being formed!

So you can imagine I jumped at the chance to work with the Windows High Performance Computing team. This is the same team that builds Windows HPC Server 2008.

I think most of the people working in the team are from the “large collection of computers in back room somewhere” school. Would be really different in the Mac software division I assume!

I work with the Open Source Software Lab and we are all things “Open Source” to the rest of the company. The HPC Server team wanted us to make sure that their product played nice with Linux infrastructure and vice-versa. The usual suspects like AD, Samba, LDAP. CIFS etc were involved. We had to make sure that these recurrent interoperability themes were addressed in the HPC environment. I also got a chance to dig into ROCKS, OSCAR, MPI stacks and job schedulers etc etc.

This was a very rewarding experience not only for the technology exposure that I got but the pervasiveness of knowledge of Open Source within the team. They were far ahead of the other product groups in this regard and “got” the Open Source ethos. In fact, prior to my interactions with them they had released an open source MPI stack based on Argonne National Lab’s MPI implementation.

The other reason was that a lot of their customers were relentlessly open source! The conventional wisdom is that HPC applications and infrastructure require a lot of tinkering. Of course, there are some applications like FEM and CFD and that are well understood, but the general feeling was that complete control and access to the underlying infrastructure is a must for getting the most performance out of a cluster. And performance is the main thing in “High Performance Computing”.

Linux is seen providing that access by HPC customers and there is a large base of Linux for HPC in academia, the national labs and other institutions that use large clusters for doing their thing.

But is this really true?

I think that HPC has gone through a typical evolution – it starts with a few people who have a pressing need. There is a cross disciplinary team formed that builds software to do their job and a community grows around it. The community reaches critical mass and people start making building tools to make it more convenient. ROCKS is an example of this. Great skill, knowledge and ability is needed to get the job done.

However, these skilled people now become overloaded. The tools and the infrastructure that they created become so popular that everyone, including people who do not have background that was assumed before, wants to use it for their ends. So the community responds and builds standardized, easy-to-use infrastructure pieces that start to fit seamlessly together. Some control is lost, but ease-to-use is the primary focus.

The infrastructure for HPC has reached that stage (ROLLS with ROCKS). Windows HPC Server 2008 is built for this ease-of-use too.

However, the applications have not reached the stage of ease-of-use. They have to be coded with a lot of domain knowledge and have to built from scratch to truly scale while running on clusters. That means that the application writers demand more control of the underlying infrastructure and want more access to it than the users and maintainers want.

I am going out on a limb and making a prediction here – soon end users will be able to specify instead of coding applications, be it genome comparison or physics simulation. This is similar to accountants finding spreadsheets. There will probably be a few different models for different types of applications but that stage will come pretty quickly.

The infrastructure that runs these user-specified applications will be adaptive and will take these specifications and automatically tune them for high performance on the clusters.

This is where the perception of needing control to the lowest levels will be moot. The best adaptive infrastructure will be the one adopted.

Bold enough for you?

Code Parallel or Die, Part 2

Frank Chism — Tue, 15 Apr 2008 19:40:00 GMT

It’s multicore time. Do you know where your parallelism is?
Do you know where your parallelism is? Well you better find it! This is because you must or your users will not see the doubling of performance that they have seen for the last seven or eight generations of microprocessors. Why? Well the chip makers ran out of physics, clocks just couldn’t go a lot faster every few months the way they used to and more cores, even if they double or triple the performance of the processor do not lead to a faster feeling system if all the applications are serial. So, is your application serial? It better not be.

Two examples from my own life demonstrate the user and buyer frustration we are about to hit head on. First my sister, a long time Mac user, bought a new dual socket dual core Mac. On paper this system was easily three to five times “more powerful” then her previous system, a dual processor Mac. Her first complaint was, “It doesn’t feel faster. It certainly doesn’t feel ‘wicked fast’ compared to my five year old system.” I pointed out to her that what she had was more power, not more ‘faster’. I told her to try doing more than one thing at once. It took awhile for her to get the hang of this, but one day recently she called and started raving about how ‘powerful’ her new system was compared to her old one. What she had learned how to do was to get three or four long running tasks to run in the background while continuing to do her interactive day to day tasks without any noticeable degradation in interactive performance!

As a second example, for Christmas I built my wife a new dual core system because she was complaining about her system being too slow and crashing all the time. It was a thing of beauty with twice the CPU, twice the memory, and five times the capacity and twice the bandwidth of her old disks. The processor was two generations newer with a 20% faster clock and over four times the memory bandwidth. I was ready for a big hug and profuse thanks for my intelligence and craftsmanship. Boy was I in for a disappointment. What I got was, “It isn’t any faster.” I am a victim of the core war!

The lights are on, but there’s no parallel home.
Well my personal tragedy was nothing compared to what will happen to any software developer or corporation that depends on their fortune or fame to flow from people using their product and liking the experience. That is, it will be tragic for those who do not start to think parallel for every step of every phase of every program that they write and design in parallel not try to graft it in afterwards. Just as with security, and network awareness, you don’t get good results trying to add in parallel later. You must start in parallel and continue in parallel and only go serial as a last drastic measure and then only for the shortest possible time. Sadly, even in my own backyard of High Performance Computing the number of programs that have tried to graft in parallel rather than design it in is appalling. What they end up with is often what I call the ‘Silence of the Lames’. That is, a lame parallel port that doesn’t scale or doesn’t speed up their program nearly as much as is possible.

Now you might say that this only works for scientific programs with massive amounts of data. Well, sorry Charlie, if your product is something as ‘serial’ as a document processing program like Microsoft Word, you had better be thinking how you can use those extra cores to improve the user experience. Why? Well as I said in the beginning: Code parallel or die!

For the want of a nail, the shoe was lost; for the want of a shoe the horse was lost; and for the want of a horse the rider was lost, being overtaken and slain by the enemy, all for the want of care about a horseshoe nail. - Benjamin Franklin

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my ‘Parallel Imperative’ and what you can personally do about it.
So, never stop studying and I’ll blog at you later.

- Frank

Code Parallel or Die

Frank Chism — Mon, 14 Apr 2008 17:22:00 GMT

“What we have here is a failure to communicate.”
- Luke in “Cool Hand Luke”

A fist full of cores in a server rich environment.
If you haven’t noticed that the world has changed, you had better wake up and smell the coffee. The era of killer apps written in serial code is ending. Like the norm in species extinction, the reason many ISVs will not survive this change is that the environment in which we develop code has changed rapidly and not all species can adapt to this new world. The change was preceded by a hint of ‘The Shape of Things to Come’ in the Beowulf and compute cluster revolution. This revolution created the first generation of commodity clusters based on microprocessor nodes connected by commodity networks. An evolutionary message from this generation is that if you are building a cluster you really ought to use server class nodes. You _can_ build a cluster from desktop parts, but the higher you scale your node count the more you want the convenience and reliability of server parts. I remember seeing a look of outrage on the face of an Intel marketing person when I commented that I thought their then brand new Itanium Tiger platform was a really nice part for building a cluster. He thought of the new platform as a High Performance Server, not as a ‘part’. Well get used to it. We don’t build computers out of discreet components like transistors any more (which is good for him because Intel makes really big Integrated Circuits, not transistors) and we have just moved up to the point where the level of integration is the server, not the processor.

And Then There Were Cores
So into this environment where servers are parts and many high performance computing applications were written to a distributed memory communications aware parallel programming model, the microprocessor has changed in the most radical way I can remember. Our equilibrium has been punctuated (see: "Punctuated equilibria: an alternative to phyletic gradualism" (1972) pp 82-115 in "Models in paleobiology", edited by Schopf, TJM Freeman, Cooper & Co, San Francisco. Eldredge, N. & Gould, S.J.) and we can expect rapid change to a whole new kind of parallelism. The change I refer to is the introduction and near ubiquity of multiple core processors. For years software developers have been able to count on dramatic increases in CPU speed and processing power on a Moore’s Law schedule. But, Moore did not say that the processing speed and power of CPUs would double every eighteen months, he said that the number of gates on a single chip would double every eighteen months. So, all was good for a long time because that doubling of gates meant smaller _and_ faster gates and greater architectural complexity and thus ‘faster’ CPUs. So by about 2005, just about every good idea for a better computer processor architecture ever thought of was incorporated into the major microprocessors on the market. So, when 2006 rolled around the obvious thing to do was to use the next doubling of gates to double the number of ‘processors’ on a microprocessor. Dual core was here and the trend will continue for at least a couple of doublings. Who knows what strange beasties will emerge when we get to the point where ‘many core’ processors become ‘too many core’ processors. Until then, though we can be sure: Multicore killed the serial star....

More on this tomorrow.

What Lies Beneath: Setting up underlying HPC tools

kishi — Thu, 21 Dec 2006 22:34:00 GMT

This blog continues what I started writing about w/ Thinking About HPC Infrastructure and what Frank wrote in about in Overloading Clusters.

After reading thru the previous blogs on HPC, someone might ask “What are some of the core components of HPC ?”. After all, once you’ve seen the outside of a Maserati or a Pantera DeTomaso, you’re not going to be satisfied just by ogling at it. Even after a test drive, the engineer in you will want to pop the hood and see what’s inside. Taking a similar approach let’s uncover some underlying HPC technologies by looking at any basic HPC setup. Once all the provisioning has been completed, the HPC system will be physically deployed with an OS and relevant drivers, utilities etc. Yet, before the actual HPC application can get installed across, there remains a critical step in the process, i.e. configuration of cluster and file system along with any tools and interfaces such as MPI (Message Passing Interface) etc. After peeling through the HPC application layer, its worthwhile to do a “deep-dive” into what really runs the HPC clusters. A broad category of these tools are:

Cluster Management tools e.g. CSM
Job Scheduling tools e.g. SCALI, Maui
Resource Management tools e.g. Torque

If you’re trying to understand the “WHY” behind the existence of these tools and their importance, take a look at Cluster Management for example. Cluster configuration, installation and management can be difficult and requires intimate familiarity with the HPC hardware, OS, underlying architecture etc. Without specific tools that attend to and manage specific underlying HPC sub-components, HPC just won’t be what it is. So, it is worthwhile to understand the unique installation experience of the tools, such as the ones listed above to understand the complexity of HPC systems. Ready – let’s dive in to the installation and function of these tools:

1. SCALI: The SCALI management and MPI software packages provide deployment, monitoring and job scheduling services for a cluster. After you deploy this software, you will be able see all the compute nodes that may have been preconfigured or are configured on your system. Scali will enable you to monitor the systems and run jobs using the SCALI graphical interface. In order to license the SCALI software, you must utilize the scainstall command to produce a license request file. This file can then be sent to SCALI to receive a permanent key. For those that need some hand-holding through this, luckily SCALI provides very comprehensive documentation on their website. A large portion of the SCALI Manage User’s Guide is dedicated to pre-setup planning and configuration of the cluster and the network. The documentation provides detailed recommendations about how you can set up their Ethernet-based network environment and out-of-band management network. The documentation also provides a general overview about how to install and configure higher performance interconnects, including bonded Ethernet, Infiniband, Myrinet and SCI. The SCALI Manage interface provides simple tools to assist in configuring and testing DET, Infiniband, and Myrinet devices for use with the SCALI MPI implementation. The SCALI MPI software supports multiple Infiniband stacks including Mellanox, Topspin, Voltaire and Infinicon.

2. HP-MPI: HP-MPI is Hewlett-Packard’s Linux-based implementation of the Message Passing Interface (MPI). Many of the utilities distributed with HP-MPI are similar to other common MPI utilities such as MPICH - e.g. mpicc, mpirun, etc. In order to utilize the HP-MPI software, a license is required for each CPU core in the cluster. To obtain a license file you are required to obtain the MAC address from each node (typically eth0) and input that information into a form at licensing.hp.com. The resulting file can then be copied to the compute node. The HP-MPI software is non-functional until licensing files are generated for the nodes

3. CSM (Cluster Systems Management): The CSM software suite is designed to automate the deployment and management of cluster nodes. Nodes can be remotely installed with an operating system as well as the CSM software for later monitoring. The CSM software supports RedHat and Novell on multiple platforms. In order to obtain and install the CSM software one must register with IBM’s website and download the required RPMs. In order to configure CSM, it can remotely install the operating system and/or the CSM software on the compute nodes. Much like Platform ROCKS, CSM makes use of PXE functionality and RedHat’s kickstart or the autoyast software to remotely install the operating system. The CSM software provides multiple methods for defining the nodes that should be deployed and managed:

a. The first method involves creating a hostname mapping (hostmap) file, which is a colon-delimited file that defines a number of attributes of each node
b. The second method also involves manually creating and editing a “node definition” (nodedef) file. This is the method suggested by the documentation for use with small clusters

Proper remote power and remote console capabilities greatly ease the administration and deployment of the compute nodes, however according to the CSM FAQ remote power management is not absolutely required. All the compute nodes must be rebooted (remotely or manually). They are then PXE booted and installed with RHEL4 using the kickstart installation system.

4. Maui and Torque: Both Torque and Maui are free software which must be compiled from the source distribution on the head node. Maui is an open-source job scheduler for compute clusters. It supports a number of task management features not found in other parallel batch processing software including policy-based scheduling and prioritization of tasks. Torque is an open-source resource manager for managing compute nodes and scheduled jobs. It can integrate with Maui to provide additional features for scheduling and managing scheduled tasks. Installation of Torque can be done using the guidance available in the Torque 2.0 Admin Manual .

5. Platform Rocks: Platform Rocks is a cluster deployment software that facilitates the deployment of various software stacks (“rolls”) onto the compute nodes. The software is capable of deploying the base operating system and utilities required for cluster administration, management and scheduling. The software can also manage configuration and updates to ensure consistency throughout the cluster. Platform Rocks is a suite of utilities that are packaged together as separate installable rolls. One of the main goals of the software is to allow for easy installation and integration of third-party rolls and applications. One unique aspect to the Platform Rocks installation approach is that the software installs an operating system on the head node, and also installs all the required rolls at the same time. The software can also automatically set up the subsystem required to install an operating system and other packages on the compute nodes (such as management agents, etc).

That about does it for a quick “deep-dive”. Let me insert a gentle reminder that these are not the only cluster or resource management technologies out there in the HPC space but rather the ones most prevalent. If you have additional tools that you have worked with, we’d like to hear from you and thank you for tuning in to Port 25. HAPPY HOLIDAYS!