Port 25: The Open Source Community at Microsoft : HPC

Virtualizing Free Linux Distributions in Windows Server 2008 R2

Peter Galli — Tue, 11 Aug 2009 00:00:00 GMT

Jason Perlow, a columnist over at ZDNet, has written a comprehensive review on virtualizing free Linux distributions in Windows Server 2008 R2.

In his Tech Broiler column, Perlow notes that the updated Hyper-V bare-metal hypervisor virtualization layer in Microsoft's upcoming Windows Server 2008 R2, which is due to be released August 14th to MSDN and Technet customers, now has support for SUSE Linux Enterprise Server 11 and Red Hat Enterprise Linux 5.3.

"Additionally, Linux support and performance has greatly improved over the initial Hyper-V release. Microsoft also recently released its Hyper-V Linux Integration Components (Linux ICs) under the GPLv2 Open Source License," Perlow says.

The Linux ICs for Hyper-V, which are in Release Candidate status, provide synthetic device drivers that enhance I/O and networking performance when Linux OSes are virtualized under Hyper-V.

"The source code for the Linux IC's were accepted into the Linux Driver Project and should become part of the Linux Kernel within two subsequent releases and code merges - 2.6.32 is expected to be when they will be integrated, and all Linux distributions using that kernel code base going forward should be Hyper-V enabled out of the box. Yes, you heard that correctly, Microsoft is now an official Linux Kernel contributor," Perlow says.

You can read the rest of Perlow's column here.

Real Mission Critical

Mark Stone — Mon, 01 Jun 2009 16:15:00 GMT

The 1.0 release of WinBioinfTools might seem like a modest event; as of this writing, the project has
44 downloads. High Performance Computing (HPC) is a small community, granted, and the number of HPC
applications for bioinformatics is a small subset of that. Let's not confuse popularity with importance,
however.

We use the phrase "mission critical" very frequently and somewhat casually within software development. In
talking to a friend about the swine flu outbreak, I was reminded that the phrase has its origin in
military history: an aspect of a mission so critical that failure in that aspect would result in the
loss of life. In the developing world where medical infrastructure can be a fragile thing, information about
the origins or genetic makeup of a virus can be vital. It can be mission critical.

Historically, the developing world has been dependent on developed western countries to do their research for them.
Open source is beginning to level that playing field, though. Using a cluster environment and software
projects like CoCoNUT for gene sequencing and comparison, even university research centers with modest x86
server environments can play in the HPC space. This is important because the research priorities for a
university in a developing country may be very different from the research priorities of a major western
research university. At its best, this is exactly the kind of lowering of barriers to entry that open
source should facilitate.

For all its value, CoCoNUT has two significant limitations. Its license is an academic license, not a fully
open source license. And it runs only on Linux/Unix systems. The latter is particularly important. Research
scientists are not IT professionals, and they should not have to care about the underlying platform on which
their software runs. The spirit of open source is to make software as widely available as possible, and there
is no way to meet that spirit without including Windows Server among the target platforms. Mission critical
demands no less.

So WinBioinfTools makes important steps forward on both fronts. The team at Nile University has released a
GPL-licensed project that "contains a number of programs for Bioinformatics running over Windows Cluster running
Windows HPC server 2008. The current version includes the CoCoNUT system for pairwise genome comparison,
parallel global sequence alignment, and parallel BLAST."

This is a great example of a local software community using open source to make their needs a priority,
and delivering a project that will benefit local software communities in other developing countries with
similar needs. WinBioinfTools puts us one step closer to making scientific computing software platform
neutral, and closer to making Windows Server a first class citizen in the open source world of HPC.

Brazilian Students Set Their Own Course

Mark Stone — Mon, 16 Mar 2009 19:05:00 GMT

I’m going to tell a story that starts in Indiana, but really it’s about Brazil.

Once upon a time “scientific computing” was nearly synonymous with “Fortran”. Today, though, just about any high level language can be used to write High Performance Computing (HPC) applications. These days that language choice also includes C#.

At Indiana University, the Open Systems Lab has pioneered work to implement Message Passing Interface (MPI) support for .Net, so that MPI applications can be written in C#. The project is MPI.Net, and you can find it on Codeplex. It is open source, about three years old, has reached a 1.0 release, and is compatible with two other important open source projects, OpenMPI and Mono. The principle developers behind the project are Andrew Lumsdaine at Indiana University and his former student, Douglas Gregor, who is now on the faculty of Rensselear Polytechnic Institute.

This is the kind of open source work that’s really exciting to see because of the way it expands choices for the developer and the end user. A C# developer should not be closed off from writing HPC applications if that’s what they want to do. And a research scientist should not have to think about whether their lab is running Linux or Windows Server. Both of these individuals are working enough layers above the operating system that somebody else’s operating system choice should not be a constraint.

So I was very excited to learn that students in Brazil at Federal University of Rio Grande do Sul were doing work on MPI, and excited to talk with them about their work. One of their projects is MPI#, also open source and also hosted on Codeplex.

MPI# builds on top of the work of MPI.Net, adding some functionality not yet present in MPI.Net. Specifically, quoting from the project description:

The goals of this project would be to build upon MPI.NET in order to complement it with the features that are missing, mainly regarding collective communication. Either they could benefit from C# native support for such communication, either they could be programmed on top of the provided MPISend/MPIRecv encapsulations. C# and .NET features such as fault tolerance or dynamicity support would be studied, in other to turn the MPI# implementation robust in large, dynamic and heterogeneous platforms.

Two of the students working on MPI# are Ismael Stangherlini and Fernando Afonso. They are graduate students in computer science, working on projects affiliated with the Brazilian Interoperability and Open Source Software Development Nucleous. When I talked to them about their work on MPI# I was curious what their communication with Indiana University had been like. Their response: they had never been in contact with Indiana University; they simply downloaded the code for MPI.Net and started working on their own.

That’s the magic of open source: that they can, in fact, just download the code on their own and start coding against it. They may make an important contribution to MPI.Net. Or their code may be entirely disregarded. Or they may move on to other projects and somebody else may or may not pick up where they left off. At this stage it’s too early to tell. But the fact that all of these scenarios are possible demonstrates why, as a methodology, open source is so nimble and adaptive. A top-down product development process, or a top-down standards development process can only execute on the innovations envisioned by the few at the top, and at the speed of the slowest decision-makers in the process. But a bottom-up open source process enables every innovation that anyone at the grass roots level can see.

Is High Performance Computing naturally Open Source (ie. for tinkerers)?

anandeep — Wed, 18 Jun 2008 13:17:00 GMT

I have always been fascinated by clusters. Some people envision working with desktops or workstations when they think of “working with computers”. For me working with computers was always with a large collection of computers in a back room somewhere. And how cool if you could make all those computers collaborate with each other working to solve cool things like genome mapping, movie special effects, simulations of car crashes or simulations of molecules being formed!

So you can imagine I jumped at the chance to work with the Windows High Performance Computing team. This is the same team that builds Windows HPC Server 2008.

I think most of the people working in the team are from the “large collection of computers in back room somewhere” school. Would be really different in the Mac software division I assume!

I work with the Open Source Software Lab and we are all things “Open Source” to the rest of the company. The HPC Server team wanted us to make sure that their product played nice with Linux infrastructure and vice-versa. The usual suspects like AD, Samba, LDAP. CIFS etc were involved. We had to make sure that these recurrent interoperability themes were addressed in the HPC environment. I also got a chance to dig into ROCKS, OSCAR, MPI stacks and job schedulers etc etc.

This was a very rewarding experience not only for the technology exposure that I got but the pervasiveness of knowledge of Open Source within the team. They were far ahead of the other product groups in this regard and “got” the Open Source ethos. In fact, prior to my interactions with them they had released an open source MPI stack based on Argonne National Lab’s MPI implementation.

The other reason was that a lot of their customers were relentlessly open source! The conventional wisdom is that HPC applications and infrastructure require a lot of tinkering. Of course, there are some applications like FEM and CFD and that are well understood, but the general feeling was that complete control and access to the underlying infrastructure is a must for getting the most performance out of a cluster. And performance is the main thing in “High Performance Computing”.

Linux is seen providing that access by HPC customers and there is a large base of Linux for HPC in academia, the national labs and other institutions that use large clusters for doing their thing.

But is this really true?

I think that HPC has gone through a typical evolution – it starts with a few people who have a pressing need. There is a cross disciplinary team formed that builds software to do their job and a community grows around it. The community reaches critical mass and people start making building tools to make it more convenient. ROCKS is an example of this. Great skill, knowledge and ability is needed to get the job done.

However, these skilled people now become overloaded. The tools and the infrastructure that they created become so popular that everyone, including people who do not have background that was assumed before, wants to use it for their ends. So the community responds and builds standardized, easy-to-use infrastructure pieces that start to fit seamlessly together. Some control is lost, but ease-to-use is the primary focus.

The infrastructure for HPC has reached that stage (ROLLS with ROCKS). Windows HPC Server 2008 is built for this ease-of-use too.

However, the applications have not reached the stage of ease-of-use. They have to be coded with a lot of domain knowledge and have to built from scratch to truly scale while running on clusters. That means that the application writers demand more control of the underlying infrastructure and want more access to it than the users and maintainers want.

I am going out on a limb and making a prediction here – soon end users will be able to specify instead of coding applications, be it genome comparison or physics simulation. This is similar to accountants finding spreadsheets. There will probably be a few different models for different types of applications but that stage will come pretty quickly.

The infrastructure that runs these user-specified applications will be adaptive and will take these specifications and automatically tune them for high performance on the clusters.

This is where the perception of needing control to the lowest levels will be moot. The best adaptive infrastructure will be the one adopted.

Bold enough for you?

Code Parallel or Die, Part 2

Frank Chism — Tue, 15 Apr 2008 19:40:00 GMT

It’s multicore time. Do you know where your parallelism is?
Do you know where your parallelism is? Well you better find it! This is because you must or your users will not see the doubling of performance that they have seen for the last seven or eight generations of microprocessors. Why? Well the chip makers ran out of physics, clocks just couldn’t go a lot faster every few months the way they used to and more cores, even if they double or triple the performance of the processor do not lead to a faster feeling system if all the applications are serial. So, is your application serial? It better not be.

Two examples from my own life demonstrate the user and buyer frustration we are about to hit head on. First my sister, a long time Mac user, bought a new dual socket dual core Mac. On paper this system was easily three to five times “more powerful” then her previous system, a dual processor Mac. Her first complaint was, “It doesn’t feel faster. It certainly doesn’t feel ‘wicked fast’ compared to my five year old system.” I pointed out to her that what she had was more power, not more ‘faster’. I told her to try doing more than one thing at once. It took awhile for her to get the hang of this, but one day recently she called and started raving about how ‘powerful’ her new system was compared to her old one. What she had learned how to do was to get three or four long running tasks to run in the background while continuing to do her interactive day to day tasks without any noticeable degradation in interactive performance!

As a second example, for Christmas I built my wife a new dual core system because she was complaining about her system being too slow and crashing all the time. It was a thing of beauty with twice the CPU, twice the memory, and five times the capacity and twice the bandwidth of her old disks. The processor was two generations newer with a 20% faster clock and over four times the memory bandwidth. I was ready for a big hug and profuse thanks for my intelligence and craftsmanship. Boy was I in for a disappointment. What I got was, “It isn’t any faster.” I am a victim of the core war!

The lights are on, but there’s no parallel home.
Well my personal tragedy was nothing compared to what will happen to any software developer or corporation that depends on their fortune or fame to flow from people using their product and liking the experience. That is, it will be tragic for those who do not start to think parallel for every step of every phase of every program that they write and design in parallel not try to graft it in afterwards. Just as with security, and network awareness, you don’t get good results trying to add in parallel later. You must start in parallel and continue in parallel and only go serial as a last drastic measure and then only for the shortest possible time. Sadly, even in my own backyard of High Performance Computing the number of programs that have tried to graft in parallel rather than design it in is appalling. What they end up with is often what I call the ‘Silence of the Lames’. That is, a lame parallel port that doesn’t scale or doesn’t speed up their program nearly as much as is possible.

Now you might say that this only works for scientific programs with massive amounts of data. Well, sorry Charlie, if your product is something as ‘serial’ as a document processing program like Microsoft Word, you had better be thinking how you can use those extra cores to improve the user experience. Why? Well as I said in the beginning: Code parallel or die!

For the want of a nail, the shoe was lost; for the want of a shoe the horse was lost; and for the want of a horse the rider was lost, being overtaken and slain by the enemy, all for the want of care about a horseshoe nail. - Benjamin Franklin

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my ‘Parallel Imperative’ and what you can personally do about it.
So, never stop studying and I’ll blog at you later.

- Frank

Code Parallel or Die

Frank Chism — Mon, 14 Apr 2008 17:22:00 GMT

“What we have here is a failure to communicate.”
- Luke in “Cool Hand Luke”

A fist full of cores in a server rich environment.
If you haven’t noticed that the world has changed, you had better wake up and smell the coffee. The era of killer apps written in serial code is ending. Like the norm in species extinction, the reason many ISVs will not survive this change is that the environment in which we develop code has changed rapidly and not all species can adapt to this new world. The change was preceded by a hint of ‘The Shape of Things to Come’ in the Beowulf and compute cluster revolution. This revolution created the first generation of commodity clusters based on microprocessor nodes connected by commodity networks. An evolutionary message from this generation is that if you are building a cluster you really ought to use server class nodes. You _can_ build a cluster from desktop parts, but the higher you scale your node count the more you want the convenience and reliability of server parts. I remember seeing a look of outrage on the face of an Intel marketing person when I commented that I thought their then brand new Itanium Tiger platform was a really nice part for building a cluster. He thought of the new platform as a High Performance Server, not as a ‘part’. Well get used to it. We don’t build computers out of discreet components like transistors any more (which is good for him because Intel makes really big Integrated Circuits, not transistors) and we have just moved up to the point where the level of integration is the server, not the processor.

And Then There Were Cores
So into this environment where servers are parts and many high performance computing applications were written to a distributed memory communications aware parallel programming model, the microprocessor has changed in the most radical way I can remember. Our equilibrium has been punctuated (see: "Punctuated equilibria: an alternative to phyletic gradualism" (1972) pp 82-115 in "Models in paleobiology", edited by Schopf, TJM Freeman, Cooper & Co, San Francisco. Eldredge, N. & Gould, S.J.) and we can expect rapid change to a whole new kind of parallelism. The change I refer to is the introduction and near ubiquity of multiple core processors. For years software developers have been able to count on dramatic increases in CPU speed and processing power on a Moore’s Law schedule. But, Moore did not say that the processing speed and power of CPUs would double every eighteen months, he said that the number of gates on a single chip would double every eighteen months. So, all was good for a long time because that doubling of gates meant smaller _and_ faster gates and greater architectural complexity and thus ‘faster’ CPUs. So by about 2005, just about every good idea for a better computer processor architecture ever thought of was incorporated into the major microprocessors on the market. So, when 2006 rolled around the obvious thing to do was to use the next doubling of gates to double the number of ‘processors’ on a microprocessor. Dual core was here and the trend will continue for at least a couple of doublings. Who knows what strange beasties will emerge when we get to the point where ‘many core’ processors become ‘too many core’ processors. Until then, though we can be sure: Multicore killed the serial star....

More on this tomorrow.

What Lies Beneath: Setting up underlying HPC tools

kishi — Thu, 21 Dec 2006 22:34:00 GMT

This blog continues what I started writing about w/ Thinking About HPC Infrastructure and what Frank wrote in about in Overloading Clusters.

After reading thru the previous blogs on HPC, someone might ask “What are some of the core components of HPC ?”. After all, once you’ve seen the outside of a Maserati or a Pantera DeTomaso, you’re not going to be satisfied just by ogling at it. Even after a test drive, the engineer in you will want to pop the hood and see what’s inside. Taking a similar approach let’s uncover some underlying HPC technologies by looking at any basic HPC setup. Once all the provisioning has been completed, the HPC system will be physically deployed with an OS and relevant drivers, utilities etc. Yet, before the actual HPC application can get installed across, there remains a critical step in the process, i.e. configuration of cluster and file system along with any tools and interfaces such as MPI (Message Passing Interface) etc. After peeling through the HPC application layer, its worthwhile to do a “deep-dive” into what really runs the HPC clusters. A broad category of these tools are:

Cluster Management tools e.g. CSM
Job Scheduling tools e.g. SCALI, Maui
Resource Management tools e.g. Torque

If you’re trying to understand the “WHY” behind the existence of these tools and their importance, take a look at Cluster Management for example. Cluster configuration, installation and management can be difficult and requires intimate familiarity with the HPC hardware, OS, underlying architecture etc. Without specific tools that attend to and manage specific underlying HPC sub-components, HPC just won’t be what it is. So, it is worthwhile to understand the unique installation experience of the tools, such as the ones listed above to understand the complexity of HPC systems. Ready – let’s dive in to the installation and function of these tools:

1. SCALI: The SCALI management and MPI software packages provide deployment, monitoring and job scheduling services for a cluster. After you deploy this software, you will be able see all the compute nodes that may have been preconfigured or are configured on your system. Scali will enable you to monitor the systems and run jobs using the SCALI graphical interface. In order to license the SCALI software, you must utilize the scainstall command to produce a license request file. This file can then be sent to SCALI to receive a permanent key. For those that need some hand-holding through this, luckily SCALI provides very comprehensive documentation on their website. A large portion of the SCALI Manage User’s Guide is dedicated to pre-setup planning and configuration of the cluster and the network. The documentation provides detailed recommendations about how you can set up their Ethernet-based network environment and out-of-band management network. The documentation also provides a general overview about how to install and configure higher performance interconnects, including bonded Ethernet, Infiniband, Myrinet and SCI. The SCALI Manage interface provides simple tools to assist in configuring and testing DET, Infiniband, and Myrinet devices for use with the SCALI MPI implementation. The SCALI MPI software supports multiple Infiniband stacks including Mellanox, Topspin, Voltaire and Infinicon.

2. HP-MPI: HP-MPI is Hewlett-Packard’s Linux-based implementation of the Message Passing Interface (MPI). Many of the utilities distributed with HP-MPI are similar to other common MPI utilities such as MPICH - e.g. mpicc, mpirun, etc. In order to utilize the HP-MPI software, a license is required for each CPU core in the cluster. To obtain a license file you are required to obtain the MAC address from each node (typically eth0) and input that information into a form at licensing.hp.com. The resulting file can then be copied to the compute node. The HP-MPI software is non-functional until licensing files are generated for the nodes

3. CSM (Cluster Systems Management): The CSM software suite is designed to automate the deployment and management of cluster nodes. Nodes can be remotely installed with an operating system as well as the CSM software for later monitoring. The CSM software supports RedHat and Novell on multiple platforms. In order to obtain and install the CSM software one must register with IBM’s website and download the required RPMs. In order to configure CSM, it can remotely install the operating system and/or the CSM software on the compute nodes. Much like Platform ROCKS, CSM makes use of PXE functionality and RedHat’s kickstart or the autoyast software to remotely install the operating system. The CSM software provides multiple methods for defining the nodes that should be deployed and managed:

a. The first method involves creating a hostname mapping (hostmap) file, which is a colon-delimited file that defines a number of attributes of each node
b. The second method also involves manually creating and editing a “node definition” (nodedef) file. This is the method suggested by the documentation for use with small clusters

Proper remote power and remote console capabilities greatly ease the administration and deployment of the compute nodes, however according to the CSM FAQ remote power management is not absolutely required. All the compute nodes must be rebooted (remotely or manually). They are then PXE booted and installed with RHEL4 using the kickstart installation system.

4. Maui and Torque: Both Torque and Maui are free software which must be compiled from the source distribution on the head node. Maui is an open-source job scheduler for compute clusters. It supports a number of task management features not found in other parallel batch processing software including policy-based scheduling and prioritization of tasks. Torque is an open-source resource manager for managing compute nodes and scheduled jobs. It can integrate with Maui to provide additional features for scheduling and managing scheduled tasks. Installation of Torque can be done using the guidance available in the Torque 2.0 Admin Manual .

5. Platform Rocks: Platform Rocks is a cluster deployment software that facilitates the deployment of various software stacks (“rolls”) onto the compute nodes. The software is capable of deploying the base operating system and utilities required for cluster administration, management and scheduling. The software can also manage configuration and updates to ensure consistency throughout the cluster. Platform Rocks is a suite of utilities that are packaged together as separate installable rolls. One of the main goals of the software is to allow for easy installation and integration of third-party rolls and applications. One unique aspect to the Platform Rocks installation approach is that the software installs an operating system on the head node, and also installs all the required rolls at the same time. The software can also automatically set up the subsystem required to install an operating system and other packages on the compute nodes (such as management agents, etc).

That about does it for a quick “deep-dive”. Let me insert a gentle reminder that these are not the only cluster or resource management technologies out there in the HPC space but rather the ones most prevalent. If you have additional tools that you have worked with, we’d like to hear from you and thank you for tuning in to Port 25. HAPPY HOLIDAYS!

Thinking about HPC Infrastructure

kishi — Fri, 01 Dec 2006 19:21:00 GMT

I started the first HPC blog (See “previous blog“) with an understanding that HPC is an area where there has been a surge of activity from a development/investment standpoint. This segment of Information Technology has experienced a heightened level of engagement from OEM’s and partners, all trying to meet the growing computing needs of their customers. So after getting a basic understanding behind the importance of why HPC matters, the next logical step that needed uncovering was “How to think” about HPC Infrastructure and tap into the “wisdom” behind managing it. You might ask why this is relevant. For starters, setting up HPC Infrastructure is an experience that, just like any other infrastructure, be it Network or Storage, requires intricate planning and intimate familiarity with its individual contributing components. In case of HPC, let’s just say you really need to know your nodes J. Let’s talk more about what’s involved in setting up an HPC Infrastructure and how to think about it as a whole:

1. Investment Impetus: To successfully plan and design an HPC Infrastructure, the first and foremost step should be to “look beneath the surface” . This simply means to understand, the primary reason for investing in HPC. The demand for HPC equipment, linked to a set of business objectives should have clear purpose around the outcome and expectation. This is specially true today than at any other moment in time because the consumption of HPC cycles, specifically in the research and development areas across all verticals has seen a steady 70% growth over the past four years (Source: primeur ). Despite this tremendous growth in the proliferation of HPC technology, the growth pattern itself is sporadic. One of the reasons for it may be the complexity, not only in terms of design but also in terms of consumption as well. Take the case of SHARCNET in Southern Ontario that developed a long range plan around adoption and implementation of HPC technology. According to the report, some of the elementary challenges around planning for HPC emerge from the fact that “it is an enabling technology for an extremely diverse set of researchers”. This embodies the essence of the sentiment behind the complexity and diversity predominant in the HPC space.

2. Planning and Designing Hardware: While thinking about planning and designing an HPC infrastructure implementation, I spoke to several folks in this area, drew from a decade and a half of my experience as an Infrastructure Architect and thought of some key areas that I would consider. These include:

a. Facility considerations (Rackspace, Power and Cooling): Talk to any enterprise level Datacenter manager what his/her top 10 pain-points are and you are bound to hear the words “rackspace, power and cooling” in what follows. Dig deeper and you’ll realize that in any datacenter, there’s a fixed number of colo’s (Colocation) you can populate based on the HVAC designs. This means that rackspace is what’s at a premium in each of these colo’s with every “u” accounted for. Packing in dense chipsets in small form-factor server add to existing power and cooling challenges
Translation – you need more outlets and more airflow per rack than what you did a decade ago with a handful of 4 and 5u servers taking up the entire rack
b. Physical Plant planning: Quoting the resident HPC Guru Frank Chism who says “I cannot over emphasize the importance to planning for physical plant in HPC deployments. Things like room and raceways for well managed and planned cabling. HPC uses more cable than anything except maybe SAN. Also, pay attention to floor loads, air flow, clean and redundant power. Finally, never never forget out-of-band management. Deep subfloor really helps with all that cabling”.

Translation – Effective HPC performance calls for an effective HPC design, which includes tweaking hard as well as soft components. These components can be as covert as chip-design or as overt as subfloor depth.
c. Hardware and Processing Power: Pushing the envelope on hardware and processor architectures today translates to increased performance (the heart and soul of HPC). Adding energy efficient hardware on top of the architecture amounts to greater investment in raw computing power, which in turn translates to building a sound HPC infrastructure. The key advantages one needs to look for in this scenario are faster data access and increased instructions. The word “performance” is repeated throughout the theme of this topic because it IS what HPC is all about, the ability to reduce the number of cycles to process data. Addressing the hardware and processing specs as part of core requirements ensures a smoother build-out.

3. Implementing HPC Tools and Software: Like any other piece of hardware, a HPC cluster is just that until software and tools exploit the underlying architecture to drive results and performance to do what it does best – compute. When thinking of some core elements of HPC tools and software, here’s how I thought to break them up:

a. Setup and deployment systems: Setting up HPC clusters goes back to what I said earlier in Section 1 – what do you want to do with it? Although there are various ways and methods that allow you to drive the software and installation experience of an HPC system, the bottom line is that this depends to a great extent of what components make up the genetic composition of the HPC cluster you ordered. Taking a look at some HPC software setup and deployment tools out there, a few mainstream ones are SCALI and HP-MPI (HP’s message passing interface). These packages provide deployment, monitoring and job scheduling services for managing and administering an HPC cluster just like IBM’s CSM (Cluster Systems Manager). In the Open Source space, there’s Maui and Torque, that work as job scheduler and resource managers for managing compute nodes and clusters. Platform Rocks is another suite of utilities that allow installation and integration of third party apps

b. Parallel FS: This is truly what I think is going to be the frontier for some intense activity over the next few years. Using Wikipedia’s description, “Distributed parallel file systems stripe data over multiple servers for high performance. Some of the distributed parallel file systems use object storage device (OSD) (In Lustre called OST) for chunks of data together with centralized metadata servers such as Ceph Scalable, Distributed File System from University of California, Santa Cruz. (Fault-tolerance in their roadmap.), Lustre from Cluster File Systems. (Lustre has failover, but multi-server RAID1 or RAID5 is still in their roadmap for future versions.) and Parallel Virtual File System (PVFS, PVFS2)”.

Deep-Dive: At Base, parallel file systems are global namespaces for files that achieve high bandwidth via parallelism. That bandwidth comes in three dimensions, high aggregate bandwidth, high single stream bandwidth, and high metadata operations per second. No one seems to have achieved high performance in all of these dimensions. Don’t forget that the volumes of data are so large that backup is a major undertaking and thus, reliability is required as well. Further, nobody seems to be able to make a parallel file system that performance well for high-speed data for short I/Os, like say you do when compiling a major application

c. Multiple Networks: A final comment on implementation of HPC is that HPC often has multiple networks. For example, it does little good to have a parallel file system that delivers gigabytes per second of data to single nodes if the network can’t handle that much bandwidth!

So in conclusion, here’s a recap on the learning behind setting up HPC Infrastructure:

Comprehensive understanding beneath WHY you’re investing in HPC and what you expect as an outcome
Deep familiarity with the core HPC Hardware and design components
Facility and Physical plant considerations to ensure adequate cabling and subfloor space
Visibility into prominent HPC based software and toolsets
Understanding the three dimensions of bandwidth
And finally accommodating the concept of “Multiple Networks” into node design to accommodate the required bandwidth

Look forward to getting back to you with more on HPC over the new few weeks again. Until then “Happy Computing”!!

HPC - The way all computing will look...

MichaelF — Wed, 01 Nov 2006 22:37:00 GMT

I have been itching to write on this subject ever since I first met w/ Doug Lora and Frank Chism. High-Performance computing – Wow! The first time someone explained the concept to me, I couldn’t help but visualize a scene from the movie “Blade Runner” and the futuristic feel of how Supercomputers actually work. My interest took me deeper into the heart of HPC to try to get my head around what HPC really is all about. High-performance computing systems, also referred to sometimes as “Supercomputers” are more prevalent today across prominent verticals such as Oil and Gas, Bioinformatics, Finance and Entertainment than ever before. Wikipedia described HPC as “ Supercomputers and Computer Clusters i.e. computing systems comprised of multiple (usually mass-produced) processors linked together in a single system with commercially available interconnects. Usually, computer systems in or above the teraflop-region are counted as HPC-computers” . An HPC Cluster is usually implemented to provide increased performance by splitting a computational task across many different nodes in the cluster.

Applications that run on HPC systems are prevalent in heavy-duty research and experimentation to engineering scenarios including transactional processing, data warehousing, computational fluid dynamics, virtual prototype testing etc. The evolution behind clustering technology has a lot to do with the growing adoption of this technology as well. The additional element that has made this technology very attractive is price. An HPC cluster can be implemented at a fraction of the cost today as compared to 10-15 years ago. Take a Cray Y-MP c916 supercomputer that cost close to $40 million 15 years ago. Today, you can get computing power very close to that for almost $4,000. The proof of this adoption is in the fact that every industry vertical is deploying HPC. From a “mainframe” approach that existed decades ago, the implementation trend of this technology is gravitating towards decentralized grids and clusters.

So why do I say that this is the way all computing will look - HPC is already a $9 billion growing market (source HPCwire). Evolution in this sphere is occurring at a blazing speed and the demand for HPC systems across various verticals is expected to multiply. Bottom-line – HPC will play a very key role in how computing power is used, stacked and scaled. Not only that, the development of HPC propagated file-systems will be an area that we all should watch very closely over the next few years. Let’s also fully realize the “impact” of this technology in the business space. If done right, HPC clusters hold the key to superior systems performance, while maintaining reasonable economies of scale. Delving into the benefits of these clusters, which until recently was a domain of the scientific community, is literally like lighting a fuse to an explosive. I say this with a strong ethos because we have yet to recognize all possible uses of HPC clusters with their underlying potential. According to some researchers at INIST-CNRS, “Analytic methods, statistical modeling, and pattern searching algorithms that are common in scientific computing can now be applied to the vast amounts of operational and historical data generated by business transactions to extract knowledge that can be used for competitive advantage”.

A nagging question still remained in my head as to the WHY behind the importance of HPC? I kept looking for the single reason behind the heavy investment in this area and why it’s such a critical component of highly-complex computational analysis being done. The biggest advantage or theme that emerged from wherever I looked was that HPC is one of the few tangible technologies out there, whose sheer computing power helps solve highly complex computational workloads and problems. This is not to mention the solitary advantage of using this technology – time. Time that it takes to resolve highly complex workloads is greatly reduced with a faster outcome. And anyone reading this blog knows the value of time and how it is THE most valuable element of all. And everyone knows, no matter how evolved and fast hardware can get, there will always be bleeding-edge problems that will demand processing power beyond what the best clusters can provide.

My endeavor here at OSSL, is to understand this topic from the ground up, have an open discussion on the subject matter as well as educate the audience along the way. And how do I plan to do that – well, we have started venturing into doing more with HPC and understanding the various HPC platforms and technologies out there. Over the course of the next few months I’ll be sharing more on this subject with all of you including market trends, evolution of HPC, Grid Computing Scenarios, “chip” supercomputing etc.

-Kishi

Overloading 'Clusters'

Frank Chism — Fri, 20 Oct 2006 17:06:00 GMT

"`When I use a word,' Humpty Dumpty said, in rather a scornful tone, `it means just what I choose it to mean -- neither more nor less.' "

Through the Looking Glass
Lewis Carroll

Where’s the glory?
I work in the cluster business. I can tell you that all too often I have felt like Alice trying to hold a conversation with Humpty Dumpty in Looking Glass Land. This usually occurs when I’m talking to someone new to cluster computing or someone who comes from a different tread of the industry than I do. My roots are in a thread that used number crunching to mean serious floating point arithmetic done by Fortran programs to simulate physical processes. Of course, some of the support routines and tools and even the operating system might be written on C, but Fortran ruled. Imagine my surprise when I found there was a ‘Number Crunchers Users Group’ in Seattle and they got together to discuss using spreadsheets. “Now where’s the glory in that?” I thought to myself.

Time marches on, but technology runs as fast as it can just to stay in one place.
Fortunately for me, the object oriented police have provided me with just the right jargon to describe my predicament. Just consider that in any modern object oriented language it is possible that + can mean any number of things. Humpty would be proud. In OOP + means just exactly what the developer chooses it to mean. This is called overloading an operator. That may be OK for a compiler, but what about me? When I use cluster I am thinking of something that descended from the original Beowulf. No, not the King of the Geats. I mean the seminal work of those oft sung NASA nerds who put together the first Beowulf compute clusters. When I say nerds, I am here to praise cluster creators, not heap dirt on them or their work. After all, they ain’t dead yet.

For example, I work for a company that has several cluster offerings. There’s failover clusters, and load balancing scale out clusters, and my baby compute clusters. Now that’s overloading. You can usually tell what kind of cluster we mean by the type of work we talk about feeding it. If you had one type of cluster in mind and I had another and we kept talking long enough we’d either figure out the root cause of the confusion or dismiss our conversational partner as an idiot.

But wait. It gets worse.
Within my own little compute centric world, two new terms have come into common usage. They are farm and grid. So how do I tell a farm from a cluster if both are eating compute intensive programs? And worse yet, how is a cluster or a farm related (or not) to a grid? I was recently told by a co-worker to not tell our customer that he had a cluster, because as far as he was concerned it was a grid. This is proof that technical correctness is not nearly as important as political correctness. As in politics, so in life.

I can’t claim to have invented farms, but I can certainly claim to be one of the first of the render farmers. I was working at an early Computer Generated Images (CGI) site that was falling behind schedule for a major (OK, it was a big deal to us) Hollywood movie. If we were to finish in time for the planned release, we needed to get our CGI effects generated at just about twice the rate we were running at on our current machine. Fortunately the little ol’ mainframe we were using, a Cray-1, had just been superseded by the Cray X/MP, which had two CPUs instead of one and each CPU was about 50% faster than the Cray-1 CPU. In an example of embarrassingly parallel render farming, we ran odd numbered frames on one thread, even numbered frames on another and ran a third thread to collate the frames and send them to the camera.

I can’t be blamed for grid at all. Well yes, some of the computers my company sold were ‘on the grid’, but I never thought of the grid as anything other than a route for users to do cool things with our machines. In fact I wasn’t sure that grid was anything other than a buzz word used to get NSF funding. Now, thanks to the efforts of the hardworking and unpaid volunteers at Wikipedia, I have at least one fixed mark to guide my wondering barking.

If a cluster on the grid failed over and no one was there to farm it, would it make any sense?
So, can we all agree on one set of definitions for clusters (several flavors to be sure), farms and grids? If not, I’m sure I’ll hear from the more assertive of the Port 25 readers and perhaps we can reach a group consensus and I can start quoting the group mind of an entire community in defense of my own use of these terms without sounding too much like Humpty Dumpty making up meanings as I see fit.

Cluster: Making more than one computer behave as a single resource.

Failover or High Availability Cluster: A cluster specifically designed to perform functions in a manner that makes the service it provides continuous, even in the event of individual computer failures.

Load balancing or Scale out Cluster: Generally a high availability cluster that in addition to offering resiliency against individual computer failures also offers addition ability to deliver more of the intended service.

Compute Cluster: A cluster that is built as a single unit and treated as a single system and tuned to perform compute intensive tasks either as a capacity engine, that is to run lots of single node jobs or many low scale parallel jobs, or a capability engine, that is to run much bigger parallel jobs than a single node can accommodate.

Compute Farm: A cluster that uses a collection of computers, generally in a centralized location, to run many similar jobs in parallel for improved time to completion of a particular process. This is very similar to a Compute Cluster in capacity mode but the farm is not necessarily built to look like a single system.

Compute Grid: A heterogeneous farm that is spread out across a wider network or even the Internet but more importantly that is controlled by and conforms to the standards, concepts, and tools originating in the Global Toolkit. It can be used in both capacity and capability mode but is generally a distributed collection of resources, not a single system.

I tried to turn the handle but—

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my proposed definitions and how they can be improved. Other items on my blog-fodder list are ‘The Parallel Imperative’ and ‘What the Heck is Parallel I/O Anyway?’

So, never stop studying and I’ll blog at you later.
- Frank

Introducing Frank Chism: High Performance Computing Blogger on Port 25

MichaelF — Wed, 18 Oct 2006 22:00:00 GMT

Today we are pleased to introduce Frank Chism. Frank is a Technology Specialist at Microsoft who has worked in the High Performance Computing space for 41 years. He is joining our team of bloggers and will contribute a monthly blog regarding HPC. In this interview Sam and Frank talk about Frank's background as well as some of his insights on HPC both past and present.

Look for Frank's first blog later this week or early next.

Video: Introducing Frank Chism: High Performance Computing Blogger on Port 25

Alternate Video Format
-Download MPEG4 Video

Compute Cluster Server: Sam interviews Ryan Waite to discuss Compute Cluster Server

MichaelF — Fri, 18 Aug 2006 16:47:00 GMT

Sam interviews Ryan Waite, Group Program Manager for HPC, who was recently involved in the development and release of Compute Cluster Server. Ryan and Sam discuss how Open Source influenced CCS through the inclusion of Open Source in the product and contributions back to the community.

Alternate Video Format
-Download MPEG4 Video

Video: Compute Cluster Server: Sam interviews Ryan Waite