Port 25: The Open Source Community at Microsoft : HPC, Server Center

Virtualizing Free Linux Distributions in Windows Server 2008 R2

Peter Galli — Tue, 11 Aug 2009 00:00:00 GMT

Jason Perlow, a columnist over at ZDNet, has written a comprehensive review on virtualizing free Linux distributions in Windows Server 2008 R2.

In his Tech Broiler column, Perlow notes that the updated Hyper-V bare-metal hypervisor virtualization layer in Microsoft's upcoming Windows Server 2008 R2, which is due to be released August 14th to MSDN and Technet customers, now has support for SUSE Linux Enterprise Server 11 and Red Hat Enterprise Linux 5.3.

"Additionally, Linux support and performance has greatly improved over the initial Hyper-V release. Microsoft also recently released its Hyper-V Linux Integration Components (Linux ICs) under the GPLv2 Open Source License," Perlow says.

The Linux ICs for Hyper-V, which are in Release Candidate status, provide synthetic device drivers that enhance I/O and networking performance when Linux OSes are virtualized under Hyper-V.

"The source code for the Linux IC's were accepted into the Linux Driver Project and should become part of the Linux Kernel within two subsequent releases and code merges - 2.6.32 is expected to be when they will be integrated, and all Linux distributions using that kernel code base going forward should be Hyper-V enabled out of the box. Yes, you heard that correctly, Microsoft is now an official Linux Kernel contributor," Perlow says.

You can read the rest of Perlow's column here.

Is High Performance Computing naturally Open Source (ie. for tinkerers)?

anandeep — Wed, 18 Jun 2008 13:17:00 GMT

I have always been fascinated by clusters. Some people envision working with desktops or workstations when they think of “working with computers”. For me working with computers was always with a large collection of computers in a back room somewhere. And how cool if you could make all those computers collaborate with each other working to solve cool things like genome mapping, movie special effects, simulations of car crashes or simulations of molecules being formed!

So you can imagine I jumped at the chance to work with the Windows High Performance Computing team. This is the same team that builds Windows HPC Server 2008.

I think most of the people working in the team are from the “large collection of computers in back room somewhere” school. Would be really different in the Mac software division I assume!

I work with the Open Source Software Lab and we are all things “Open Source” to the rest of the company. The HPC Server team wanted us to make sure that their product played nice with Linux infrastructure and vice-versa. The usual suspects like AD, Samba, LDAP. CIFS etc were involved. We had to make sure that these recurrent interoperability themes were addressed in the HPC environment. I also got a chance to dig into ROCKS, OSCAR, MPI stacks and job schedulers etc etc.

This was a very rewarding experience not only for the technology exposure that I got but the pervasiveness of knowledge of Open Source within the team. They were far ahead of the other product groups in this regard and “got” the Open Source ethos. In fact, prior to my interactions with them they had released an open source MPI stack based on Argonne National Lab’s MPI implementation.

The other reason was that a lot of their customers were relentlessly open source! The conventional wisdom is that HPC applications and infrastructure require a lot of tinkering. Of course, there are some applications like FEM and CFD and that are well understood, but the general feeling was that complete control and access to the underlying infrastructure is a must for getting the most performance out of a cluster. And performance is the main thing in “High Performance Computing”.

Linux is seen providing that access by HPC customers and there is a large base of Linux for HPC in academia, the national labs and other institutions that use large clusters for doing their thing.

But is this really true?

I think that HPC has gone through a typical evolution – it starts with a few people who have a pressing need. There is a cross disciplinary team formed that builds software to do their job and a community grows around it. The community reaches critical mass and people start making building tools to make it more convenient. ROCKS is an example of this. Great skill, knowledge and ability is needed to get the job done.

However, these skilled people now become overloaded. The tools and the infrastructure that they created become so popular that everyone, including people who do not have background that was assumed before, wants to use it for their ends. So the community responds and builds standardized, easy-to-use infrastructure pieces that start to fit seamlessly together. Some control is lost, but ease-to-use is the primary focus.

The infrastructure for HPC has reached that stage (ROLLS with ROCKS). Windows HPC Server 2008 is built for this ease-of-use too.

However, the applications have not reached the stage of ease-of-use. They have to be coded with a lot of domain knowledge and have to built from scratch to truly scale while running on clusters. That means that the application writers demand more control of the underlying infrastructure and want more access to it than the users and maintainers want.

I am going out on a limb and making a prediction here – soon end users will be able to specify instead of coding applications, be it genome comparison or physics simulation. This is similar to accountants finding spreadsheets. There will probably be a few different models for different types of applications but that stage will come pretty quickly.

The infrastructure that runs these user-specified applications will be adaptive and will take these specifications and automatically tune them for high performance on the clusters.

This is where the perception of needing control to the lowest levels will be moot. The best adaptive infrastructure will be the one adopted.

Bold enough for you?

What Lies Beneath: Setting up underlying HPC tools

kishi — Thu, 21 Dec 2006 22:34:00 GMT

This blog continues what I started writing about w/ Thinking About HPC Infrastructure and what Frank wrote in about in Overloading Clusters.

After reading thru the previous blogs on HPC, someone might ask “What are some of the core components of HPC ?”. After all, once you’ve seen the outside of a Maserati or a Pantera DeTomaso, you’re not going to be satisfied just by ogling at it. Even after a test drive, the engineer in you will want to pop the hood and see what’s inside. Taking a similar approach let’s uncover some underlying HPC technologies by looking at any basic HPC setup. Once all the provisioning has been completed, the HPC system will be physically deployed with an OS and relevant drivers, utilities etc. Yet, before the actual HPC application can get installed across, there remains a critical step in the process, i.e. configuration of cluster and file system along with any tools and interfaces such as MPI (Message Passing Interface) etc. After peeling through the HPC application layer, its worthwhile to do a “deep-dive” into what really runs the HPC clusters. A broad category of these tools are:

Cluster Management tools e.g. CSM
Job Scheduling tools e.g. SCALI, Maui
Resource Management tools e.g. Torque

If you’re trying to understand the “WHY” behind the existence of these tools and their importance, take a look at Cluster Management for example. Cluster configuration, installation and management can be difficult and requires intimate familiarity with the HPC hardware, OS, underlying architecture etc. Without specific tools that attend to and manage specific underlying HPC sub-components, HPC just won’t be what it is. So, it is worthwhile to understand the unique installation experience of the tools, such as the ones listed above to understand the complexity of HPC systems. Ready – let’s dive in to the installation and function of these tools:

1. SCALI: The SCALI management and MPI software packages provide deployment, monitoring and job scheduling services for a cluster. After you deploy this software, you will be able see all the compute nodes that may have been preconfigured or are configured on your system. Scali will enable you to monitor the systems and run jobs using the SCALI graphical interface. In order to license the SCALI software, you must utilize the scainstall command to produce a license request file. This file can then be sent to SCALI to receive a permanent key. For those that need some hand-holding through this, luckily SCALI provides very comprehensive documentation on their website. A large portion of the SCALI Manage User’s Guide is dedicated to pre-setup planning and configuration of the cluster and the network. The documentation provides detailed recommendations about how you can set up their Ethernet-based network environment and out-of-band management network. The documentation also provides a general overview about how to install and configure higher performance interconnects, including bonded Ethernet, Infiniband, Myrinet and SCI. The SCALI Manage interface provides simple tools to assist in configuring and testing DET, Infiniband, and Myrinet devices for use with the SCALI MPI implementation. The SCALI MPI software supports multiple Infiniband stacks including Mellanox, Topspin, Voltaire and Infinicon.

2. HP-MPI: HP-MPI is Hewlett-Packard’s Linux-based implementation of the Message Passing Interface (MPI). Many of the utilities distributed with HP-MPI are similar to other common MPI utilities such as MPICH - e.g. mpicc, mpirun, etc. In order to utilize the HP-MPI software, a license is required for each CPU core in the cluster. To obtain a license file you are required to obtain the MAC address from each node (typically eth0) and input that information into a form at licensing.hp.com. The resulting file can then be copied to the compute node. The HP-MPI software is non-functional until licensing files are generated for the nodes

3. CSM (Cluster Systems Management): The CSM software suite is designed to automate the deployment and management of cluster nodes. Nodes can be remotely installed with an operating system as well as the CSM software for later monitoring. The CSM software supports RedHat and Novell on multiple platforms. In order to obtain and install the CSM software one must register with IBM’s website and download the required RPMs. In order to configure CSM, it can remotely install the operating system and/or the CSM software on the compute nodes. Much like Platform ROCKS, CSM makes use of PXE functionality and RedHat’s kickstart or the autoyast software to remotely install the operating system. The CSM software provides multiple methods for defining the nodes that should be deployed and managed:

a. The first method involves creating a hostname mapping (hostmap) file, which is a colon-delimited file that defines a number of attributes of each node
b. The second method also involves manually creating and editing a “node definition” (nodedef) file. This is the method suggested by the documentation for use with small clusters

Proper remote power and remote console capabilities greatly ease the administration and deployment of the compute nodes, however according to the CSM FAQ remote power management is not absolutely required. All the compute nodes must be rebooted (remotely or manually). They are then PXE booted and installed with RHEL4 using the kickstart installation system.

4. Maui and Torque: Both Torque and Maui are free software which must be compiled from the source distribution on the head node. Maui is an open-source job scheduler for compute clusters. It supports a number of task management features not found in other parallel batch processing software including policy-based scheduling and prioritization of tasks. Torque is an open-source resource manager for managing compute nodes and scheduled jobs. It can integrate with Maui to provide additional features for scheduling and managing scheduled tasks. Installation of Torque can be done using the guidance available in the Torque 2.0 Admin Manual .

5. Platform Rocks: Platform Rocks is a cluster deployment software that facilitates the deployment of various software stacks (“rolls”) onto the compute nodes. The software is capable of deploying the base operating system and utilities required for cluster administration, management and scheduling. The software can also manage configuration and updates to ensure consistency throughout the cluster. Platform Rocks is a suite of utilities that are packaged together as separate installable rolls. One of the main goals of the software is to allow for easy installation and integration of third-party rolls and applications. One unique aspect to the Platform Rocks installation approach is that the software installs an operating system on the head node, and also installs all the required rolls at the same time. The software can also automatically set up the subsystem required to install an operating system and other packages on the compute nodes (such as management agents, etc).

That about does it for a quick “deep-dive”. Let me insert a gentle reminder that these are not the only cluster or resource management technologies out there in the HPC space but rather the ones most prevalent. If you have additional tools that you have worked with, we’d like to hear from you and thank you for tuning in to Port 25. HAPPY HOLIDAYS!

Thinking about HPC Infrastructure

kishi — Fri, 01 Dec 2006 19:21:00 GMT

I started the first HPC blog (See “previous blog“) with an understanding that HPC is an area where there has been a surge of activity from a development/investment standpoint. This segment of Information Technology has experienced a heightened level of engagement from OEM’s and partners, all trying to meet the growing computing needs of their customers. So after getting a basic understanding behind the importance of why HPC matters, the next logical step that needed uncovering was “How to think” about HPC Infrastructure and tap into the “wisdom” behind managing it. You might ask why this is relevant. For starters, setting up HPC Infrastructure is an experience that, just like any other infrastructure, be it Network or Storage, requires intricate planning and intimate familiarity with its individual contributing components. In case of HPC, let’s just say you really need to know your nodes J. Let’s talk more about what’s involved in setting up an HPC Infrastructure and how to think about it as a whole:

1. Investment Impetus: To successfully plan and design an HPC Infrastructure, the first and foremost step should be to “look beneath the surface” . This simply means to understand, the primary reason for investing in HPC. The demand for HPC equipment, linked to a set of business objectives should have clear purpose around the outcome and expectation. This is specially true today than at any other moment in time because the consumption of HPC cycles, specifically in the research and development areas across all verticals has seen a steady 70% growth over the past four years (Source: primeur ). Despite this tremendous growth in the proliferation of HPC technology, the growth pattern itself is sporadic. One of the reasons for it may be the complexity, not only in terms of design but also in terms of consumption as well. Take the case of SHARCNET in Southern Ontario that developed a long range plan around adoption and implementation of HPC technology. According to the report, some of the elementary challenges around planning for HPC emerge from the fact that “it is an enabling technology for an extremely diverse set of researchers”. This embodies the essence of the sentiment behind the complexity and diversity predominant in the HPC space.

2. Planning and Designing Hardware: While thinking about planning and designing an HPC infrastructure implementation, I spoke to several folks in this area, drew from a decade and a half of my experience as an Infrastructure Architect and thought of some key areas that I would consider. These include:

a. Facility considerations (Rackspace, Power and Cooling): Talk to any enterprise level Datacenter manager what his/her top 10 pain-points are and you are bound to hear the words “rackspace, power and cooling” in what follows. Dig deeper and you’ll realize that in any datacenter, there’s a fixed number of colo’s (Colocation) you can populate based on the HVAC designs. This means that rackspace is what’s at a premium in each of these colo’s with every “u” accounted for. Packing in dense chipsets in small form-factor server add to existing power and cooling challenges
Translation – you need more outlets and more airflow per rack than what you did a decade ago with a handful of 4 and 5u servers taking up the entire rack
b. Physical Plant planning: Quoting the resident HPC Guru Frank Chism who says “I cannot over emphasize the importance to planning for physical plant in HPC deployments. Things like room and raceways for well managed and planned cabling. HPC uses more cable than anything except maybe SAN. Also, pay attention to floor loads, air flow, clean and redundant power. Finally, never never forget out-of-band management. Deep subfloor really helps with all that cabling”.

Translation – Effective HPC performance calls for an effective HPC design, which includes tweaking hard as well as soft components. These components can be as covert as chip-design or as overt as subfloor depth.
c. Hardware and Processing Power: Pushing the envelope on hardware and processor architectures today translates to increased performance (the heart and soul of HPC). Adding energy efficient hardware on top of the architecture amounts to greater investment in raw computing power, which in turn translates to building a sound HPC infrastructure. The key advantages one needs to look for in this scenario are faster data access and increased instructions. The word “performance” is repeated throughout the theme of this topic because it IS what HPC is all about, the ability to reduce the number of cycles to process data. Addressing the hardware and processing specs as part of core requirements ensures a smoother build-out.

3. Implementing HPC Tools and Software: Like any other piece of hardware, a HPC cluster is just that until software and tools exploit the underlying architecture to drive results and performance to do what it does best – compute. When thinking of some core elements of HPC tools and software, here’s how I thought to break them up:

a. Setup and deployment systems: Setting up HPC clusters goes back to what I said earlier in Section 1 – what do you want to do with it? Although there are various ways and methods that allow you to drive the software and installation experience of an HPC system, the bottom line is that this depends to a great extent of what components make up the genetic composition of the HPC cluster you ordered. Taking a look at some HPC software setup and deployment tools out there, a few mainstream ones are SCALI and HP-MPI (HP’s message passing interface). These packages provide deployment, monitoring and job scheduling services for managing and administering an HPC cluster just like IBM’s CSM (Cluster Systems Manager). In the Open Source space, there’s Maui and Torque, that work as job scheduler and resource managers for managing compute nodes and clusters. Platform Rocks is another suite of utilities that allow installation and integration of third party apps

b. Parallel FS: This is truly what I think is going to be the frontier for some intense activity over the next few years. Using Wikipedia’s description, “Distributed parallel file systems stripe data over multiple servers for high performance. Some of the distributed parallel file systems use object storage device (OSD) (In Lustre called OST) for chunks of data together with centralized metadata servers such as Ceph Scalable, Distributed File System from University of California, Santa Cruz. (Fault-tolerance in their roadmap.), Lustre from Cluster File Systems. (Lustre has failover, but multi-server RAID1 or RAID5 is still in their roadmap for future versions.) and Parallel Virtual File System (PVFS, PVFS2)”.

Deep-Dive: At Base, parallel file systems are global namespaces for files that achieve high bandwidth via parallelism. That bandwidth comes in three dimensions, high aggregate bandwidth, high single stream bandwidth, and high metadata operations per second. No one seems to have achieved high performance in all of these dimensions. Don’t forget that the volumes of data are so large that backup is a major undertaking and thus, reliability is required as well. Further, nobody seems to be able to make a parallel file system that performance well for high-speed data for short I/Os, like say you do when compiling a major application

c. Multiple Networks: A final comment on implementation of HPC is that HPC often has multiple networks. For example, it does little good to have a parallel file system that delivers gigabytes per second of data to single nodes if the network can’t handle that much bandwidth!

So in conclusion, here’s a recap on the learning behind setting up HPC Infrastructure:

Comprehensive understanding beneath WHY you’re investing in HPC and what you expect as an outcome
Deep familiarity with the core HPC Hardware and design components
Facility and Physical plant considerations to ensure adequate cabling and subfloor space
Visibility into prominent HPC based software and toolsets
Understanding the three dimensions of bandwidth
And finally accommodating the concept of “Multiple Networks” into node design to accommodate the required bandwidth

Look forward to getting back to you with more on HPC over the new few weeks again. Until then “Happy Computing”!!

Overloading 'Clusters'

Frank Chism — Fri, 20 Oct 2006 17:06:00 GMT

"`When I use a word,' Humpty Dumpty said, in rather a scornful tone, `it means just what I choose it to mean -- neither more nor less.' "

Through the Looking Glass
Lewis Carroll

Where’s the glory?
I work in the cluster business. I can tell you that all too often I have felt like Alice trying to hold a conversation with Humpty Dumpty in Looking Glass Land. This usually occurs when I’m talking to someone new to cluster computing or someone who comes from a different tread of the industry than I do. My roots are in a thread that used number crunching to mean serious floating point arithmetic done by Fortran programs to simulate physical processes. Of course, some of the support routines and tools and even the operating system might be written on C, but Fortran ruled. Imagine my surprise when I found there was a ‘Number Crunchers Users Group’ in Seattle and they got together to discuss using spreadsheets. “Now where’s the glory in that?” I thought to myself.

Time marches on, but technology runs as fast as it can just to stay in one place.
Fortunately for me, the object oriented police have provided me with just the right jargon to describe my predicament. Just consider that in any modern object oriented language it is possible that + can mean any number of things. Humpty would be proud. In OOP + means just exactly what the developer chooses it to mean. This is called overloading an operator. That may be OK for a compiler, but what about me? When I use cluster I am thinking of something that descended from the original Beowulf. No, not the King of the Geats. I mean the seminal work of those oft sung NASA nerds who put together the first Beowulf compute clusters. When I say nerds, I am here to praise cluster creators, not heap dirt on them or their work. After all, they ain’t dead yet.

For example, I work for a company that has several cluster offerings. There’s failover clusters, and load balancing scale out clusters, and my baby compute clusters. Now that’s overloading. You can usually tell what kind of cluster we mean by the type of work we talk about feeding it. If you had one type of cluster in mind and I had another and we kept talking long enough we’d either figure out the root cause of the confusion or dismiss our conversational partner as an idiot.

But wait. It gets worse.
Within my own little compute centric world, two new terms have come into common usage. They are farm and grid. So how do I tell a farm from a cluster if both are eating compute intensive programs? And worse yet, how is a cluster or a farm related (or not) to a grid? I was recently told by a co-worker to not tell our customer that he had a cluster, because as far as he was concerned it was a grid. This is proof that technical correctness is not nearly as important as political correctness. As in politics, so in life.

I can’t claim to have invented farms, but I can certainly claim to be one of the first of the render farmers. I was working at an early Computer Generated Images (CGI) site that was falling behind schedule for a major (OK, it was a big deal to us) Hollywood movie. If we were to finish in time for the planned release, we needed to get our CGI effects generated at just about twice the rate we were running at on our current machine. Fortunately the little ol’ mainframe we were using, a Cray-1, had just been superseded by the Cray X/MP, which had two CPUs instead of one and each CPU was about 50% faster than the Cray-1 CPU. In an example of embarrassingly parallel render farming, we ran odd numbered frames on one thread, even numbered frames on another and ran a third thread to collate the frames and send them to the camera.

I can’t be blamed for grid at all. Well yes, some of the computers my company sold were ‘on the grid’, but I never thought of the grid as anything other than a route for users to do cool things with our machines. In fact I wasn’t sure that grid was anything other than a buzz word used to get NSF funding. Now, thanks to the efforts of the hardworking and unpaid volunteers at Wikipedia, I have at least one fixed mark to guide my wondering barking.

If a cluster on the grid failed over and no one was there to farm it, would it make any sense?
So, can we all agree on one set of definitions for clusters (several flavors to be sure), farms and grids? If not, I’m sure I’ll hear from the more assertive of the Port 25 readers and perhaps we can reach a group consensus and I can start quoting the group mind of an entire community in defense of my own use of these terms without sounding too much like Humpty Dumpty making up meanings as I see fit.

Cluster: Making more than one computer behave as a single resource.

Failover or High Availability Cluster: A cluster specifically designed to perform functions in a manner that makes the service it provides continuous, even in the event of individual computer failures.

Load balancing or Scale out Cluster: Generally a high availability cluster that in addition to offering resiliency against individual computer failures also offers addition ability to deliver more of the intended service.

Compute Cluster: A cluster that is built as a single unit and treated as a single system and tuned to perform compute intensive tasks either as a capacity engine, that is to run lots of single node jobs or many low scale parallel jobs, or a capability engine, that is to run much bigger parallel jobs than a single node can accommodate.

Compute Farm: A cluster that uses a collection of computers, generally in a centralized location, to run many similar jobs in parallel for improved time to completion of a particular process. This is very similar to a Compute Cluster in capacity mode but the farm is not necessarily built to look like a single system.

Compute Grid: A heterogeneous farm that is spread out across a wider network or even the Internet but more importantly that is controlled by and conforms to the standards, concepts, and tools originating in the Global Toolkit. It can be used in both capacity and capability mode but is generally a distributed collection of resources, not a single system.

I tried to turn the handle but—

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my proposed definitions and how they can be improved. Other items on my blog-fodder list are ‘The Parallel Imperative’ and ‘What the Heck is Parallel I/O Anyway?’

So, never stop studying and I’ll blog at you later.
- Frank

Introducing Frank Chism: High Performance Computing Blogger on Port 25

MichaelF — Wed, 18 Oct 2006 22:00:00 GMT

Today we are pleased to introduce Frank Chism. Frank is a Technology Specialist at Microsoft who has worked in the High Performance Computing space for 41 years. He is joining our team of bloggers and will contribute a monthly blog regarding HPC. In this interview Sam and Frank talk about Frank's background as well as some of his insights on HPC both past and present.

Look for Frank's first blog later this week or early next.

Video: Introducing Frank Chism: High Performance Computing Blogger on Port 25

Alternate Video Format
-Download MPEG4 Video

Compute Cluster Server: Sam interviews Ryan Waite to discuss Compute Cluster Server

MichaelF — Fri, 18 Aug 2006 16:47:00 GMT

Sam interviews Ryan Waite, Group Program Manager for HPC, who was recently involved in the development and release of Compute Cluster Server. Ryan and Sam discuss how Open Source influenced CCS through the inclusion of Open Source in the product and contributions back to the community.

Alternate Video Format
-Download MPEG4 Video

Video: Compute Cluster Server: Sam interviews Ryan Waite