Port 25: The Open Source Community at Microsoft : Kishi Malhotra

Alvidaa

kishi — Tue, 06 Nov 2007 18:21:00 GMT

I have been working as a Senior Program Manager with the Open Source Software Lab since the fall of 2005. After spending two of the most eye-opening and fantastic years here, sadly, time has come for me to move on. I am taking on a role in a different division inside of Microsoft but having been attached to Port25 for such a long time, I didn’t want to leave without writing my parting thoughts. You see, when I started my work with the Open Source Software Lab, I had no idea who Bill Hilf was or his role at Microsoft. So when I first came to speak to him about this opportunity, I was driven purely by the job description, the first line of which read “Everything is connected”. After talking to Bill, when I came back and searched for his name/credentials on the web, needless to say, I felt like a total idiot. Here was someone, who was literally the Linux and Open Source “guy” within Microsoft and I had no clue about his background whatsoever....taught me that I should have done better homework . After going through the interview loops and meeting up w/ some sharp minds in OSSL, I was very attracted to the opportunity and came on board.

Anyway, I have had the pleasure of working with some amazing people on this time, Sam Ramji, Hank Janssen, Michael Francisco, Steve Zarkos, Tom Hanrahan, John Kew, Anandeep Pannu to name a few. In the process of understanding and learning about Linux and Open Source technologies, I also learnt a whole lot about driving change through people, technology and especially practices (Sam – Thank you!). In my two years with the OSSL, I got the opportunity to REALLY push the boundaries of conventional or deep-rooted thinking. I was able to work on my pet projects/areas of interest such as Systems Manageability and IT Operations. I spent this past summer building the Interop Lab in Cambridge, MA – something I enjoyed whole-heartedly. I got face time with thought leaders like Miguel De Icaza and rubbed shoulders with creative thinkers like Tom Hanrahan. The experience that I am walking away with is quite profound at many levels. Let me explain why: You see, this team is so unique in what it does, that it’s perhaps one of the few places which has the ability to drive change inward and outward. In my experience here, I have not only seen the ground shift beneath my feet but have also tremendous progress towards community involvement and understanding as it relates to Linux and Open Source. The wisdom I am walking away with can best be captured by something Margaret Mead wrote “Never under estimate the power of a few committed people to change the world “. I say that with the utmost passion because the intellectual horsepower, pure passion and pace that I have witnessed in this group is hard to ignore or imitate.

Some other thoughts that I am taking with me are how much effort goes into simply undoing misconceptions and misunderstandings. Working in this group and watching Bill, Sam, Hank and all these guys work – I realized how committed we are to building bridges and doing a great job of listening as well as being understood. So, after working with Open Source enthusiasts and Windows professionals side-by-side, I whole heartedly endorse something F. Scott Fitzgerald wrote a while ago “The test of a first-rate intelligence is the ability to hold two opposing ideas in mind at the same time and still retain the ability to function“

In conclusion, I would urge the Open Source Community to really look at how far we have come in the past two years alone. Don’t take my word for it, see for yourself the work done on Port25 and http://www.microsoft.com/opensource

As always, your thoughts and comments are ALWAYS welcome…………….. Alvidaa (That’s urdu for Farewell)

Systems Manageability Part 7 - Log Management and Analysis

kishi — Tue, 07 Aug 2007 15:57:00 GMT

Level-Set – Log Management: This section includes open-source technology directed primarily on host-based logging, log file rotation and log file analysis. Many of these tools are very common free and open-source software tools that are distributed and preconfigured with most of the major Linux systems, including major vendors such as RedHat and Novell.

I. Logrotate

Logrotate is a very popular application utilized in a number of Linux systems, including all RedHat and SUSE based systems. The logrotate utility typically runs periodically via cron, a task scheduling application. The utility will read a configuration file (/etc/logrotate.conf), and archive and compress log files according to the configuration. Administrators can configure when log files should be rotated based on age and size, and how long backlogs should be maintained. Older archived log files can then be swapped out and replaced with newer archives.

II. Syslogd and klogd

Typical Linux systems utilize a syslog daemon to capture log messages from userspace applications and write them to text-based log files or send them to a logging host over the network. The syslogd daemon is often accompanied by a klogd application which is designed to capture and log kernel messages.

The behavior of the syslog daemon can be configured via the /etc/syslog.conf configuration file. All messages captured by syslog are categorized by facility and priority. Messages can then be sent to particular log files or logging hosts, or dropped completely based on their facility and priority attributes.

Facilities

Priorities

- auth or security

- authpriv

- cron

- daemon

- kern

- lpr

- mail

- mark

- news

- syslog

- user

- uucp

- local0 through local7

- debug

- info

- notice

- warning or warn

- err or error

- error

- crit

- alert

- emerg or panic

List of syslog facilities and priorities.

III. Syslog-ng

The syslog-ng application aims to be an enhanced drop-in replacement for the traditional syslog daemon. It provides many of the same features of the standard syslog daemon, but includes additional features such as advanced message filtering based on content, remote logging via UDP or TCP, and the ability to write log files to a database such as MySQL or PostgreSQL. More recent SUSE-based systems such as SLES10 have switched to syslog-ng as the default syslog server.

IV. Viewing Logs

Most log files on a Linux system are stored in plain-text, which means they can be viewed and parsed using a number of different command-line tools. Typical utilities such as tail, head, grep, cat, less, more, sed and awk can be used to view and filter log messages via the command line.

There are also a myriad of utilities designed to parse and view log files via a GUI or web browser. Some utilities are even designed to handle specific log formats, such as those generated by Linux’s Netfilter firewall subsystem.

GNOME System Log Viewer

The GNOME system includes a GTK-based system log viewing application that displays system logs via the GUI.

YaST System Log Module

SUSE-based systems using YaST typically include a module called View System Log (called internally as view_anymsg). Similar to the GNOME System Log viewer, the YaST module allows an administrator to view many of the various system logs without using the command-line.

V. Log Analysis

LogWatch

The logwatch utility is designed to parse system logs and located any entries that might indicate security threat or system failure and send an email report to a designated address. Logwatch is distributed with RedHat Enterprise Linux systems. The following is an excerpt from the RPM description:

“LogWatch is a customizable log analysis system. LogWatch parses through your system's logs for a given period of time and creates a report analyzing areas that you specify, in as much detail as you require. LogWatch is easy to use and claims that it will work right out of the package on almost all systems. Note that LogWatch now analyzes Samba logs.”

LogWatch is typically executed periodically via cron, a task scheduling application.

LogCheck

The logcheck utility is a part of the Sentry Tools project that also includes portsentry, a utility designed to detect port scans. Similar to the LogWatch utility, the software is designed to parse system log files, find log entries that may indicate security problems and send an email to a preconfigured address. Also similar to the LogWatch utility, logcheck relies on the standard cron utility to be periodically executed.

That does it for Log Management and Analysis section. We have one last blog to go and certainly hope that you found the information we have captured for you useful. If you’re running any special toolsets or customizable scripts for log management and analysis and would like to share your experience with us, please send us your feedback and as always, THANK YOU for tuning into Port25.

Systems Manageability Part 6: Patch Management and Online Updates

kishi — Fri, 29 Jun 2007 17:44:00 GMT

Level-Set - Patch Management: Patch Management and Maintenance focuses on those solutions available to deploy and install software update on Linux systems, with a primary focus on Novell based Linux systems. This is going to be a very short blog because the only open source tool that I could find, which is used in a widespread manner, is YaST. I know there are tons of solutions out there, some proprietary like RHN and some custom built. YaST was the only common thread we could recognize. A deeper look at YaST and its online update abilities follows:

YAST Online Update Utility

Probably the most common and important modules in YaST are those related to software management (adding and removing software) and patch management. Software and updates for a typical SUSE system are obtained from software repositories, which can be local or remote software inventories from which new software or updates may be obtained. At a deeper level, the SLES9 package management system utilizes the common rpm utility to install, remove, and update packages and manage the package and dependency database. Although this subsystem is similar to RedHat’s, Novell has chosen a very different approach to distributing its patches, choosing to utilize what are called patch RPMs. With many RPM-based distributions, when a package needs to be updated for one reason or another the distributor will modify or patch the original source tree and recompile/repackage the software to produce a new RPM for that particular package. Therefore in these cases the new RPM will simply be an updated version of the original RPM.

Novell has taken a slightly different approach with patching via RPMs. Instead of updating and repacking the entire package, Novell updates the original source tree, recompiles, and then produces a delta (or a diff) between the original binaries in the package and the newly patched/recompiled binaries. The delta is a binary file that contains information about the differences between two binary files. The deltas will then be packaged within an RPM and distributed to clients. The patch RPM can then be manually or automatically installed in the same way a standard RPM would be installed. An advantage to this technique is that patches are often smaller in size – typically anywhere between 5KB and 8MB depending on the size of the package and the changes being applied. This often allows the update process to progress far faster than it would otherwise when using full RPMs – especially for large applications.

Major updates to the stable SLES9 branch are released as an installable “service pack”. Novell typically recommends installing the service pack files via YaST2, from either a CDROM or network location that contains the service pack files. One may also simply utilize the Online Update module of YaST2 to update the system manually or automatically. In this case, the service pack will be distributed as a large number of individual packages, similar to how RedHat distributes major updates (i.e. RHEL4 U4). Aside from a log file, SLES9 does not currently have an email mechanism to inform the administrator when a patch is automatically downloaded and installed (as RedHat does). However, a log file that contains information about each automatic update is maintained in /var/lib/YaST2/you/youlog. This log is generally very easy for an administrator to read and discover when, or if, a patch RPM was downloaded and installed.

There are other ways to find information about installed patches, however. By default, SLES9 archives each patch RPM that is downloaded and installed. Full RPMs will also be archived if they were installed via YaST2 after the original system installation. This functionality can be disabled with YaST2, of course, although it can sometimes be useful to maintain the archive if a patch ever needs to be reinstalled.

1. YaST Software/Update Repositories

Software repositories are typically added manually via the Installation Source module in YaST or can be scanned using SLP (Service Location Protocol). From this module, one may add references to locations from which to receive updates. These references typically take the form of a URI or a directory path. YaST supports the following software repository references:
FTPHTTP(S)
SMB/CIFS
NFS
CD or DVD
Local Directory
Using this methodology it is also quite common for an administrator to install a centralized repository for software and updates. Updates may then be obtained from Novell by a single server, and other servers on the LAN may then pull patches from the central patch server using one of the above protocols.

2. YaST Security

Although software repositories for SLES and SLED distributions are typically operated by Novell, it is quite possible to add third-party repositories to obtain software not offered by Novell, or even different versions of the same software packages. Novell warns against this, however, since adding repositories not controlled by Novell can result in the installation of untested or possibly malicious software, which ultimately could compromise security, but more likely may result in software instability and RPM package conflicts.
All official software and patches obtained by Novell are cryptographically signed, which can be verified with Novell’s public key. The public keys used to verify these signatures are typically obtained via the official SLES/SLED CDs or DVDs, but may also be obtained via Novell’s website. Once these public keys are accepted and imported, any software package or update obtained with an invalid signature will produce a warning and may not install without user intervention.

3. YaST Automatic Updates

Automatic updates can be configured via YaST’s Online Update Setup module, which allows a user to schedule updates to occur at a particular time either daily or weekly. On the backend, this module simply installs a new cron entry, a task scheduling application, which periodically runs another program to check for and install updates pushed out by Novell.
In earlier SUSE-based systems, YOU (YaST Online Update) had been used to automate the installation of updates packages. The cron utility would execute a shell script called /usr/bin/online_update which would automate the patch installation process. Newer versions of SUSE, including SLED10, utilize a similar process but instead of a shell script a utility called rug is used. The rug utility is the command-line interface to the ZENworks management agent that is present on new SUSE systems.

If you are running any open source based tools or applications in your environment to push patches and manage online update scenarios, we would REALLY like to hear what you have to say. As always THANK YOU for tuning into Port25

Systems Manageability Part Five: Monitoring

kishi — Thu, 21 Jun 2007 14:16:00 GMT

Background: This is Part 5, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the third part of the “ontology” which is “Monitoring”

Level-Set – Monitoring: Monitoring and other data collection tools are an essential component of any management strategy. The proper collection and organization of host data allows for manual and sometimes automated reactive corrective measures. This section outlines many of the open source and free software monitoring tools available on the Linux platform. Much of the analysis in this section is focused on the inner workings of these tools as data collection systems, rather than feature comparisons between the various monitoring applications. The WBEM/CIM overview has been placed in this section due to its basis as a data collection and management system, even though its use is not limited the confines of this category.

I.WBEM/CIM: The following section includes an overview of the WBEM initiative and the open-source CIM implementations that exist today. The Distributed Management Task Force (DMTF) classifies WBEM (Web Based Enterprise Management) as the following:

“[WBEM is] a set of management and Internet standard technologies developed to unify the management of distributed computing environments. WBEM provides the ability for the industry to deliver a well-integrated set of standard-based management tools, facilitating the exchange of data across otherwise disparate technologies and platforms.”

Core components and industry standards used in WBEM include CIM, CIM-XML, CIM Query Language, SLP (Service LocationProtocol, for WBEM Discovery) and WBEM URI (Universal Resource Identifier) mapping. The DMTF has also developed a WBEM Management profile template for the purpose of systems manageability. WBEM has been designed to be compatible with all the major existing management protocols, including SNMP, DMI, and CMIP. There are several open source implementations of WBEM including OpenWBEM, WBEM Services, OpenPegasus and SBLIM. These are discussed in more detail below. Additionally, there are both client and server implementations available for the WBEM standard:

WBEM clients include PyWBEM, an open-source WBEM library written in Python, and the Purgos open-source management client for Windows written in C++.
WBEM server implementations include OpenPegasus and OpenWBEM, an open-source client and server written in C++ (Novell has adopted this and added it to SLES9/10).

The following is an explanation of CIM, or Common Information Model, from the DMTF documentation:

“provides a common definition of management information for systems, networks, applications and services, and allows for vendor extensions. CIM’s common definitions enable vendors to exchange semantically rich management information between systems throughout the network. It is a conceptual information model for describing management that is not bound to a particular implementation. This allows for the interchange of management information between management systems and applications. This can be either "agent to manager" or "manager to manager" communications that provides for Distributed System Management.”

CIM includes two components; a specification and a Schema.

CIM Specification: This describes the language, naming, Meta Schema and mapping techniques to other management models such as SNMP MIBs, and DMTF MIFs etc. The Meta Schema is a formal definition of the model. It defines the terms used to express the model and their usage and semantics. The elements of the Meta Schema are Classes, Properties, and Methods. The Meta Schema also supports Indications and Associations as types of Classes and References as types of Properties. Essentially, the CIM specification
CIM Schema: This provides the actual model descriptions. The CIM Schema supplies a set of classes with properties and associations that provide a well-understood conceptual framework within which it is possible to organize the available information about the managed environment. The CIM Schema itself is structured into three distinct layers:

The Core Schema is an information model that captures notions that are applicable to all areas of management.
Common Schemas are information models that capture notions that are common to particular management areas, but independent of a particular technology or implementation. The common areas are systems, devices, networks, applications, metrics, databases, the physical environment, event definition and handling, management of a CIM infrastructure (the Interoperability Model), users and security, policy and trouble ticketing/ knowledge exchange (the Support Model). These models define classes addressing each of the management areas in a vendor-neutral manner.
Extension Schemas represent organizational or vendor-specific extensions of the Common Schema. These schemas can be specific to environments, such as operating systems (for example, UNIX® or Microsoft Windows®). Extension Schema fall into two categories, Technology-Specific areas such UNIX98 or Product-Specific areas that are unique to a particular product such as Windows.

WBEM (CIM) Architecture Diagram

OpenPegasus:

OpenPegasus is an open-source implementation of the DMTF CIM and WBEM standards being driven under the auspices of The Open Group. OpenPegasus is open source and is licensed under the MIT open-source license. The distribution is available via CVS, and as snapshot images in tar, zip, and (self-extracting) exe file formats on the OpenPegasus web site. Based on documentation posted on the site, simply put, Pegasus is an open-source CIM Server for DMTF CIM objects. It is written in C++ and includes the Object manager (CIMOM), a set of defined interfaces, an implementation of the CIM Operations over HTTP operations and their cimxml HTTP encodings, and Interface libraries for both clients and providers. It is maintained to be compliant with the DMTF CIM and WBEM specifications with exceptions noted in the documentation. It is designed to be portable and modular. It is coded in C++ and translates the object concepts of the CIM objects into a programming model. Pegasus is designed to be inherently portable and builds and runs today on most versions of UNIX(R), Linux, and Windows. OpenPegasus includes the following components:

A DMTF compliant CIM Server that processes CIM operations, CIM Indications, and includes class and instance repositories and interfaces for creating CIM Providers and CIM Clients.
Provider interfaces so that providers may be build in multiple languages (i.e. C++, C, Java).
A number of CIM Providers.
A MOF compiler.
A number of CIM Clients to provide usage examples, CIM Server test functions, and administrative functions

OpenWBEM On SLES10:

OpenWBEM is included in SUSE Linux Enterprise Server 9 and 10, allowing any WBEM enabled management console to access configuration information on the system. A CIM schema and a MOF compiler are also included as packages in SLES9 and 10, which can be used to create and import the schema.

## Create the namespace called /root/cimv2

SLES10:/etc/openwbem # owcreatenamespace -n /root/cimv2
Creating namespace (/root/cimv2)
## Import the CIM schema.
SLES10:/etc/openwbem # owmofc /usr/share/mof/cimv2.12/cimv212.mof
[ ... Lots of Output ... ]
Compilation finished. 0 errors occurred.
Compiling and Importing the CIM Schema

## Start the OpenWBEM Daemon.
SLES10:~ # /etc/init.d/owcimomd start
Using common server certificate /etc/ssl/servercerts/servercert.pem
Starting the OpenWBEM CIMOM Daemon done
## Check the status of the OpenWBEM service.
SLES10:~ # /etc/init.d/owcimomd status
Checking for service OpenWBEM CIMOM Daemon running
Starting the OpenWBEM Service on SLES10

II. NAGIOS: Nagios is a system monitoring application designed to monitor remote hosts and applications over a network. The application provides a web-based graphical display that allows one to view the status of nodes and particular applications running on the nodes. The following is an excerpt from the Nagios documentation listing some of Nagios’ feature set: Some of the many features of Nagios include:

Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
Monitoring of host resources (processor load, disk usage, etc.)
Simple plugin design that allows users to easily develop their own service checks
Parallelized service checks
Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
Contact notifications when service or host problems occur and get resolved (via email, pager, or user-defined method)
Ability to define event handlers to be run during service or host events for proactive problem resolution
Automatic log file rotation
Support for implementing redundant monitoring hosts
Optional web interface for viewing current network status, notification and problem history, log file, etc.

Nagios can poll servers and obtain data in a number of different ways. The most straight-forward method is to connect to a remote system directly and test to see if the host is available or if a particular service is running. Data internal to the host, such as free memory or processor usage, however, must be gathered using the Nagios agent, SNMP, another custom script or program or a Nagios plug-in called check_by_ssh - which is a standard plug-in designed to run a command on a remote machine and collect the output. The configuration of Nagios is done entirely via text-based configuration files. Hosts and other resources are defined inblocks, which can also inherit information from other pre-defined blocks, making complex configurations possible and more manageable. There are several third-party applications available that provide a web or other GUI interface to assist one with configuring Nagios, but these were not tested for this project. The following configuration block defines a generic host template called “linux-server”. Many of the configuration values such as “24x7” and “workhours” are actually defined in other configuration blocks within the Nagios configuration. This allows administrators to define custom names to a specific time period, such as “workhours”, and use that definition in other parts of the configuration.

define host {

   name                       linux-server
   use                        generic-host
   check_period               24x7
   max_check_attempts         10
   check_command              check-host-alive
   notification_period        workhours
   notification_interval      120
   notification_options       d,u,r
   contact_groups             admins
   register                   0
}
Nagios Host Definition Template

Individual hosts are defined in configuration blocks. Below is a sample configuration for an individual host called management. Notice the use statement is inheriting other definitions from the previously defined generic template mentioned above called “linux-server”.

define host {
   use            linux-server ;Name of host template to use.
   host_name      management
   alias          Management Server
   address        10.197.173.100
}

Finally, hosts may be organized into logical groups for easier management. The following is a hostgroup that defines a group that includes five hosts.

define hostgroup {
   hostgroup_name test
   alias          Test Servers
   members        localhost,management,www,rhel4-production2,network
}

Nagios is distributed with a wide assortment of plug-ins that can be used to obtain data or check a particular service. Plug-ins are distributed as a separate package which must be installed with both the server and the agent if an agent is to be used. The Nagios plug-ins are simply stand-alone executable programs, each of which can perform a particular task and return a result code for each service or subsystem being tested. Since plug-ins are individual scripts or binary programs, they often will accept different arguments to change their behavior and what information they return. The command usage of each plug-in must be defined individually within the configuration files using the define command syntax. Some plug-ins can accept multiple options which can be customized when writing the configuration for a particular system. The define command definition provides a sort of usage template so that Nagios will know how to run the command later. Luckily for new users, the default sample configuration files already provide accurate definitions for the default plug-ins. Once one is familiar with how commands are defined, however, new commands or custom scripts can also be defined here as well.

NRPE: is the Nagios Remote Plugin Executor that is installed on a remote host. It is designed simply to execute Nagios plugins on behalf of the Nagios server and return the results. The same plugins that are installed on the server must then be installed on the remote host for NRPE to utilize. A new plug-in called check_nrpe is also distributed with the NRPE agent and is used to query the NRPE daemon from the Nagios server. NRPE utilizes a rudimentary access control system to assure that only particular Nagios hosts will be allowed to contact the NRPE client. A configuration directive such as the following within NRPE’s configuration file will only allow communication with a particular host:

allows_hosts=10.197.173.100

It is possible to configure NRPE run nearly any command with any arguments, although one is warned against doing this in the documentation. By default, NRPE will only run specific commands and their arguments as specified in its own configuration file (located on the host itself). Meaning that the Nagios server can tell NRPE to execute only specific commands specified in the remote host’s /etc/nrpe.cfg file, but the server may not pass arbitrary commands or plug-in arguments for the agent to execute. Below is a sample NRPE configuration. The specific commands (plug-ins) and arguments must be specified here. The Nagios server can then request NRPE to execute one or more of these commands and return the results:

command[check_users]=/usr/local/nagios/libexec/check_users –w 5 –c 10
command[check_load]=/usr/local/nagios/libexec/check_load –w 15,10,5 –c 30,25,20
command[check_disk_root]=/usr/local/nagios/libexec/check_disk –w 20 –c 10 –p /dev/sda1
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs –w 5 –c 10 –s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs –w 150 –c 200

By default NRPE utilizes SSL communications between itself and the Nagios server. The SSL parameters are generated at compile time and stored in the C header file called dh.h within the NRPE source tree. This header file is then used to compile the NRPE daemon and the check_nrpe plugin. This means that both the NRPE daemon and the check_nrpe plugin must be compiled using the same parameters (typically from the same source tree) if one wishes to utilize SSL communications.

III. Hyperic: Hyperic HQ is a Java-based monitoring application consisting of a central monitoring server and one or more remote agents to report node status information to the server. Hyperic HQ is supported on a wide array of platforms, including Linux, Solaris, Windows, HP-UX, AIX, Mac OS X and FreeBSD. Hyperic distributes two versions of its software;

An open source version licensed under the GNU GPL
and a commercial version called HQ Enterprise which includes additional components and functionality.

HQ Open Source and HQ Enterprise Feature Set Comparison
Note: As of HQ 3.0 thefeature-set distribution between the Open-Source and Enterprise versions has changed. Please see http://www.hyperic.com/products/hq_for_ent.html for more details.

Hyperic Installation and Configuration: Hyperic HQ aims to be quick to install and relatively easy to configure. The installation is performed via the command-line, and will prompt the administrator for all the information (administrator password, database information, etc) it will need to successfully run. Upgrading can also be done relatively easily by simply running the installer with the –upgrade option. Hyperic HQ provides a web interface to deliver monitoring alerts and status information to the end-user. However, unlike other monitoring applications the web-interface is also used as the primary configuration interface for the application. All node and agent details, metric options and alerts may be configured directly over the web interface. The monitoring agent is installed in a similar manner as the server. Because all agent configuration is done via the web interface on the server, the only information the agent installation script needs is login information for the server, the preferred path on the node to which it should install the agent files and various other pieces of information such as the port numbers on which the server and agent will be running. Once the agent successfully registers itself with the server, the administrator can then log in to the web interface and import the new system into its list of monitored hosts. The Hyperic HQ server utilizes the open-source PostgreSQL database application to store configuration and monitoring data. PostgreSQL comes prepackaged with the Hyperic HQ software, and can be installed and configured automatically by the installation system. One may also choose to use an existing PostgreSQL or Oracle database server if one exists. The installation system would then prompt the administrator for information about the database so that Hyperic HQ may log in and store its data. By default, Hyperic HQ stores its authentication information within this database as well, but may also be configured to utilize and external LDAP server if one is available.

Auto-Discovery: A unique feature of the Hyperic HQ monitoring solution is its ability to automatically locate and monitor services and daemons running on the remote node. Once the agent is installed on the remote node it can then scan for a variety of known services and add it to the hosts inventory. Once added to the inventory, metrics and alerts can be configured to monitor that particular service. Hyperic HQ supports two scanning options, auto-scan and file-scan. Agents run an auto-scan periodically by default which scans the process list for known server types. A more comprehensive scan called a file-scan can actually search through the file system on the remote node and locate known applications. Because it requires more time to run and is more resource intensive, this type of scan must be scheduled and configured manually by the administrator.

Alerts and Notifications: Hyperic HQ supports the configuration of alerts based on any metric for any particular resource (such as the host itself) or service running on the host. For example, an alert can be triggered when the Availability metric for a host falls changes at all, or falls below a predefined value. When an alert is triggered an email can be sent to a predefined email address. Depending on the priority of the alert, a message will also be posted to the Dashboard, the Hyperic HQ administration front page. The HQ Open Source version lacks many of the more advanced notification options that are available in the Enterprise version. HQ Enterprise also supports the concept of Recovery Alerts, which are alerts that can be configured to cancel and reset triggered alerts. When an alert is triggered in the Open Source version, the alert will continue to be triggered until the problem is fixed or the alert is disabled. Recovery Alerts allow an administrator to automate the process of disabling an active alert, and then re-enabling the alert when the problem is corrected. HQ Enterprise also supports the option of sending SNMP traps as a notification option.

Hyperic HQ Plugins: Hyperic HQ plugins are distributed as .jar or .xml files that are deployed on the server and the agent. Plugins can be developed to enhance the collection of metrics from certain applications or services, locate and inventory new services and control actions to control specific resources. The Hyperic website provides comprehensive documentation on plugin development. Developing and adding a new plugin tends to be a more complex process compared to Nagios or other monitoring applications. The framework provided by Hyperic HQ, however, provides advanced APIs from which the plugins can query information on multiple platforms. On Windows, for example, Hyperic HQ includes classes which a plugin may use to access Windows specific data and functions. These functions can provide access to performance information, registry data, event log information and the Service Control Manager (SCM). Hyperic HQ also provides support for simple script-based plugins to gather particular metrics. Even individual scripts or Nagios plugins may be imported and configured for use by the Hyperic HQ server and agents.

SIGAR – System Information Gatherer And Reporter: SIGAR is the primary data collection component of the Hyperic HQ agent. The software is designed to collect system and process information from a number of platforms - including Linux, Windows, Solaris, AIX, HP-UX, FreeBSD and Mac OSX. SIGAR is written in C, but Hyperic provides C, C#, Java and Perl APIs which one may use to to integrate SIGAR into their applications. The SIGAR component is licensed under the GNU GPL, and is distributed separately from the Hyperic monitoring agent for potential use in third-party applications. The Sigar API provides a portable interface for gathering system information such as:

System memory, swap, cpu, load average, uptime, logins
Per-process memory, cpu, credential info, state, arguments, environment, open files
File system detection and metrics
Network interface detection, configuration info and metrics
Network route and connection tables

user@linux:~/hyperic-sigar-1.3.0.0> java -jar sigar-bin/lib/sigar.jar
Loaded rc file: /home/user/hyperic-sigar-1.3.0.0/sigar-bin/lib/.sigar_shellrc
sigar> help
Available commands:
        alias          - Create alias command
        cpuinfo        - Display cpu information
        df             - Report filesystem disk space usage
        du             - Display usage for a directory recursively
        free           - Display information about free and used memory
        get            - Get system properties
        help           - Gives help on shell commands
        ifconfig       - Network interface information
        iostat         - Report filesystem disk i/o
        kill           - Send signal to a process
        mps            - Show multi process status
        netinfo        - Display network info
        netstat        - Display network connections
        pargs          - Show process command line arguments
        penv           - Show process environment
        pfile          - Display process file info
        pinfo          - Display all process info
        pmodules       - Display process module info
        ps             - Show process status
        ptql           - Run process table query
        quit           - Terminate the shell
        route          - Kernel IP routing table
        set            - Set system properties
        sleep          - Delay execution for the a number of seconds
        source         - Read a file, executing the contents
        sysinfo        - Display system information
        test           - Run sigar tests
        time           - Time command
        ulimit         - Display system resource limits
        uptime         - Display how long the system has been running
        version        - Display sigar and system version info
        who            - Show who is logged on
sigar>
Example SIGAR usage from the command-line.

And that does it for the “Monitoring” section. There are so many other tools we got a chance to play with like Monit, Argus, OProfile etc. but am running out of space …… As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability Part 4: Systems Configuration

kishi — Fri, 25 May 2007 18:18:00 GMT

Background: This is Part 4, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the second part of the “ontology” which is “Systems Configuration”

Level-Set: System Configuration and Management encompasses all tasks related to the configuration of a host in a standardized and (when possible) centralized way. Many projects in this category provide a common configuration interface, either command-line or GUI-based, designed to ease typical administrative tasks. Other projects, specifically Cfengine, provide a higher level policy-based system to provide consistent configuration and state management for a set of systems. Again, in this case there’s lots of different tools out there that can be used but we have focused on the most popular ones such as Webmin, YaST, SSH, VNC and Cfengine. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I. WEBMIN: "Webmin is a web-based interface for system administration for Unix. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on." Webmin is very modular in design, allowing third-party developers to add support for a particular service or task relatively easily. Many of the tasks involve easing or automating system administration tasks, or editing a configuration file using a specific syntax. Webmin is currently supported by OpenCountry, a company that sells Linux management solutions. The OpenCountry website includes information about Webmin, including two variations of the system that they support.

Webmin Plus: Webmin Plus, is a version of Webmin supported by OpenCountry. According to the website, the Webmin Plus version has been tested by the company and includes new features, specifically support for the back-up and restore application called Bacula (http://www.bacula.org/). Webmin Plus is still freely distributed under a “Mozilla-type” license, and is downloadable via Sourceforge.
Webmin Pro: Last I checked, Webmin Pro had not yet been released, but the following is an excerpt about the product from the OpenCountry website: “...a commercially supported comprehensive product for the entire data center enabling centralized systems administration of mixed distributions, Linux and Windows!”

II. YAST: YaST (Yet another Setup Tool) is an OS installation and configuration utility used primarily in SUSE-based systems. YaST typically serves as the primary control panel interface in, and can be used for a number of configuration tasks – such as adding and removing software, patch management, user management, device configuration and for configuring individual services and daemons. Other common administration tasks such as obtaining system information and reading server logs is also possible via the YaST interface. All of the aforementioned YaST features are implemented as modules, each of which provide a specific functionality or perform certain tasks. These tasks typically involve editing one or more text configuration files on the system in a specific format to configure a specific service or daemon. On other Linux or UNIX-like systems, these tasks are typically performed manually via the command-line.

The YaST utility is very modular in its design, allowing Novell or other third-party providers to add modules into the YaST interface to configure a particular device or service. Many of these modules work independently of each other, and as such are often packaged as individual RPM packages that may be added or removed depending on the software and devices that are installed on a system. YaST modules are written using a scripting language specific to YaST called YCP. Other scripts, such as Perl or shell scripts can also be utilized via a YaST module to perform a particular task. A CIM module for YaST is also distributed with SLES10, which provides a client interface for CIMOM (Common Information Object Manager) to other YaST2 modules. It seems the most common administration task for which YaST is used involves setting up individual package repositories (discussed further in the Patch Management and Maintenance), adding or removing software packages and configuring or initiating online updates. YaST is capable of searching for and locating software on remote repositories, retrieving the software packages, resolving package dependencies, checking the cryptographic signature of the package (if available) and then installing the software on the system. Multiple repositories can be configured. Repositories can be located on a hard disk or CD/DVD, or on a remote system obtainable via HTTP(s), FTP, NFS or CIFS. Once a repository is configured it can then be indexed for later searching. The software search functionality is very powerful, allowing one to search for appropriate software packages using many of the attributes available in the RPM package header – such as the description or contents of the package. Besides software management, the quality and completeness of many YaST modules varies. Many modules (such as the log viewing modules) offer minimal functionality, and only work well enough to provide a few basic configuration options. Complex server configurations will therefore still require one to edit text-based configuration files by hand, or use another configuration engine for the task, such as Webmin. However, many other common tasks, such as configuring display settings or a printer, can be done entirely via YaST.

III. SSH/SCP/SFTP: SSH (Secure SHell) is likely the most widely used remote administration tool for Linux and UNIX-based systems. The typical SSH toolset includes the SSH client and server, as well as the SCP and SFTP client applications for copying files, both of which simply utilize the ssh binary on the backend. The following excerpt is from the OpenSSH project home page: "OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions. Since most any task can be performed via the command-line, the OpenSSH utilities are likely the most critical component for a Linux administrator to have available. The remote copy and command execution options allow one to build, deploy and run a script on a number of machines relatively quickly and securely. OpenSSH is typically installed by default in most Linux-based distributions, although in some distributions the server may by default be disabled or blocked by the firewall. "Most common uses of SSH are:

Remote Command Execution: One of the most common uses of the ssh utility is to run commands in remote machines
Authentication: In order to log into, copy files or run commands on a remote host via ssh, one must first authenticate to the remote machine. Local password authentication may be used without any particular configuration on the server. Linux systems that support PAM (Pluggable Authentication Modules) may also use password authentication to authenticate using a number of mechanisms, including NIS, LDAP, Kerberos, or AD (Active Directory).Rather than requiring a user to manually type in a password, a more common authentication option is to use public key authentication. A user’s public key must be inserted into a file on the remote host called authorized_keys before they are allowed to authenticate to that server. This can often be done via the post installation procedures built into kickstart or autoyast.
Tunneling other Applications: Another common application is to use SSH for tunneling other protocols. This is often used to tunnel protocols that are not typically encrypted such as NFS or X11. By using the –Y or –X switches with the ssh client application, one can “turn on” X11 tunneling, allowing graphical applications run on the remote machine to display locally on the administrators workstation. For example, if one were to SSH into a SUSE-based system and run “yast2” via the command line, the YaST display would be tunneled via SSH and displayed on the local system – even though the actual application is running on the remote system.
Cluster Management: When managing a number of Linux servers, such as a HPC cluster, it is typical to have a large number of systems with identical configurations. To ease administration of these systems there have been a number of SSH-based utilities that allow one to run commands on multiple systems, or copy a file to multiple systems, in parallel. The following utilities are all licensed under the GNU GPL.

ClusterSSH: ClusterSSH allows an administrator run to open a remote SSH session to a number of systems at once via a single terminal window, and run commands or alter configuration files on all the systems simultaneously.
PCP: PCP is a tool designed to copy files in parallel to multiple nodes in a cluster or server farm.
GEXEC: GEXEC is a tool that is somewhat complimentary to PCP that provides a parallel remote command execution system for large clusters. The system includes a client and server, as well as a library that allows integration into third-party applications.
PSSH: PSSH is distributed as a suite of utilities that perform many of same functions as ClusterSSH, PCP and GEXEC.
pssh – Parallel SSH, similar to ClusterSSH or GEXEC.
pscp – Parallel SCP, allows one to copy files to multiples nodes in manner a similar to PCP.
prsync – The prsync utility automates running rsync on multiple nodes in parallel, essentially another method of copying files or entire directory trees to a number of remote nodes.
pnuke – The pnuke command can be used to kill a number of processes running on multiple nodes.
pslurp – The pslurp utility is similar to the pscp utility, except that it is designed to copy files from a set of hosts. This allows one to copy, for example, a log file that exists on multiple nodes, to a local directory tree.

IV. Cfengine: “Cfengine, or the configuration engine is an autonomous agent and a middle to high level policy language and agent for building expert systems to administrate and configure large computer networks. Cfengine is designed to be a part of a computer immune system. It is ideal for cluster management and has been adopted for use all over the world in small and huge organizations alike.” Cfengine consists of a userspace application called cfagent and a host of other utilities that reads and parses a series of text configuration files and performs tasks on the host system based on the configuration. The configuration syntax of Cfengine is actually a high-level policy language that allows cfagent to test the system’s configuration and perform corrective actions based on those tests. For example, cfagent may test to assure that a certain line of text exists within a configuration file, and if not it will add the text and restart the associated service. The cfagent utility is typically run on an hourly (or so) basis via cron, a task-scheduling application. This assures that mis-configurations will be found and corrected within a reasonable time frame.

The policy simply tests to make sure an entry for user root exists within the /etc/shadow file, and also checks to make sure the password matches. This assures that all systems have the same password for the root user. The configuration of Cfengine can become very complex, which would likely not surprise those who have had experience with the tool. The structure of the policy language eases this dilemma a bit, as platform definitions can be made and inherited by other blocks to help determine the appropriate action to take. The configuration is essentially a high-level policy language, and thus the various tests must be built and scripted manually. The toolset is, however, enormously powerful when implemented correctly. But as with many open-source technologies, the learning curve can be quite steep, and one must study the complexities of the tool before it can be competently used in a production environment. A version of Cfengine has been ported to the Windows platform to run under Cygwin.

editfiles:
                    # We have different passwords for lab systems and workstations.
       linux.shadowpasswords.md5passwords.(!workstations)::
{ /etc/shadow
    SetLine "root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::"
    AppendIfNoLineMatching '^root:.*'
    LocateLineMatching '^root:.*'
    ReplaceLineWith “root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::”
}
       linux.shadowpasswords.md5passwords.workstations::
{ /etc/shadow
    SetLine “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
    AppendIfNoLineMatching ‘^root:.*’
    LocateLineMatching ‘^root:.*’
    ReplaceLineWith “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
}
Example Cfengine policy to check the password for the root user.

The following example Cfengine policy checks for the existence and the contents of the /etc/cron.d/yast2-online-update file for SUSE systems. If necessary it creates the file, and writes a cron entry into the file to schedule a daily check for updates and patches. Upon completion, it then runs the command “/etc/init.d/cron restart” as defined in the suse.restartcrond definition.

editfiles:
        suse::
                { /etc/cron.d/yast2-online-update
                  DefineClasses "restartcrond"
                  Umask 077
                  AutoCreate
                  BeginGroupIfNoLineMatching "^.*[\s\t]+root[\s\t]+online_update"
                      AppendIfNoSuchLine "30 3 * * * root online_update"
                  EndGroup
                }
shellcommands:
        suse.restartcrond::
                "/etc/init.d/cron restart"

Example Cfengine policy to assure that SUSE systems check for updates daily.

And that does it for the “Systems Configuration” section As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability Part 3 - Provisioning and Deployment

kishi — Thu, 03 May 2007 17:29:00 GMT

I want to start this blog with a note of Thanks to Ajay Mungara, the Manageability Developer Community Manager from Intel and “einhverfr”, both of whom gave some very constructive feedback on the previous blog. In the next six blogs to follow, including this one, I will do a “deep-dive” into the six specific areas we covered under the “Systems Manageability” ontology.

Let’s start this blog with the first of the six categories from the ontology - “Deployment and Provisioning”

Level-Set: Deployment and Provisioning as we understand it, encompasses all tasks related to the initial installation of an operating system on remote system, as well as post-installation of software on a remote system. Much of these toolsets are geared toward automated system provisioning and cloning. There’s lots of different tools out there that can be used but we have focused on the most popular ones, namely Kickstart, Autoyast, “Bare-Metal” provisioning and RedHat Network. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I.KICKSTART: is an automated installation utility for RedHat-based systems, including Fedora Core and RHEL based systems. Kickstart software requires the creation of a configuration file (similar to an “answer file” in Windows lingo) which contains all the information the installation program will require to install the operating system. The configuration file and all the RPM software packages are typically kept on a remote server such as a HTTP or FTP. The location of the Kickstart configuration file is typically passed to the kernel at boot time. For example, once the bootloader (GRUB, LILO) loads, the user is often presented with a “boot:” prompt that allows the user to pass arguments to the kernel. To load a kickstart configuration file from a remote server one would type the following:

boot: linux ks=http://<server>/location/of/kickstart.cfg

The administrator can then create multiple Kickstart configuration files for different configurations. With the addition of a PXE-based server, much of the provisioning process can be automated. A Kickstart configuration file is simply a text file, which can be created and edited manually with any text-editor. A GUI-based Kickstart file creation program called system-config-kickstart is also available from the Fedora and RHEL distributions. Both kickstart and autoYaST provide the ability to run arbitrary commands during the post-installation phase. This allows administrators to run individual commands, or entire shell scripts to automate any post installation tasks that may be required. RedHat provides a GUI-based tool to assist one in building a kickstart configuration , however in practice it is very easy to customize the configuration file by hand. The ability to run shell commands via the post-configuration phase is simple yet extremely powerful.

II. AutoYaST: is another automated installation utility, similar to RedHat’s Kickstart utility, but used primarily with SUSE-based systems. Novell provides a YaST2 module for autoYaST, which is a GUI tool that can be used to create an autoYaST configuration file – also similar to RedHat’s system-config-kickstart utility. Many of the same rules and procedures used with the kickstart utility also apply to autoYaST. Administrators that need to deploy SUSE clients or servers can create any number of autoYaST configuration files to fit a particular system profile. These can then be used to automate the installation of a SUSE system over a network. When combined with PXE/DHCP setup administrators can deploy and start a SUSE install without using any physical media (i.e. an installation CD/DVD).

III. Bare-Metal Provisioning: Automated deployment tools such as Kickstart and Autoyast support system provisioning via HTTP, (T)FTP and NFS. Completely automated installations can also be configured using the Pre-Execution Environment (PXE), DHCP, tftp and kickstart or autoyast. By automating the boot process, it no longer becomes necessary to manually initialize the installation process via a CDROM or other bootable medium.

Many cluster deployment solutions utilize these very same technologies to deploy large numbers of nodes in a very short time. ROCKS, for example, automates the booting and (re)deployment cluster nodes on the network using PXE, DHCP and kickstart, a process that can sometimes require less than 10-minutes for a single node. Proprietary tools such as CSM, IBM’s cluster management software, also utilizes PXE, DHCP and kickstart or autoYaST to (re)provision cluster nodes as needed.

Configuring a PXE Server for Automated Installations: There are two common utilities one may use to configure a PXE server on Linux. Testing for this scenario was done using a RedHat based system. Therefore some aspects of the following descriptions, such as locations of configuration files, will be RedHat-centric. The general necessity of the configuration and the components, however, are not distribution specific.

pxeos – This utility can be used to configure operating system descriptions within the PXE boot files. Operating system descriptions include the OS name, the protocol used to obtain the OS files (HTTP, FTP, NFS) and the full URI and path to the installation files.
system-config-netboot – The system-config-netboot utility is a graphical application that can perform many of the same tasks as the pxeos utility.

Configuring DHCP: The DHCP daemon can be configured via the /etc/dhcpd.conf file. Those options that are specific to allowing PXE boot clients are listed below.

allow bootp;
class "pxeclients" {
match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
filename "linux-install/pxelinux.0";
}

Configuring TFTP: The TFTP daemon must first be enabled via xinetd super daemon. The following configuration is added to /etc/xinetd.conf, or sometimes /etc/xinetd.d/tftp, depending on the distribution:

service tftp
{
        disable                 = no
        port                   = 69
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}
TFTP Configuration in xinetd.conf
The directory /tftpboot/linux-install is the default used by the system-config-netboot configuration tool. The directory contains kernels and the necessary configuration files required to boot a system and begin a kickstart installation. The configuration file /tftpboot/linux-install/pxelinux.cfg/pxeos.xml contains specific definitions about which network install profiles are available to PXE boot clients.

That’s it for the Provisioning and Deployment section. As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability – Part 2: Scope, Methodology and Ontology

kishi — Fri, 06 Apr 2007 18:14:00 GMT

In my last blog called “Why Manageability Matters” I talked about why we chose to work on “Systems Manageability” as a whole and get a grassroots understanding of it within the context of Linux and Open Source space. In this blog, I’m going to address the Methodology and Ontology of the Systems Manageability project. This will shed immediate light on how we approach, design and implement projects in the OSSL. Let’s start with the main goals and purpose behind the project.

I. Systems Manageability Project Goals:

Survey the scope, depth and breadth of Linux manageability solutions, primarily utilizing open source software
Use the knowledge from the research to develop an understanding of Manageability landscape in the Linux and Open Source
Build and simulate common Linux management solutions in the OSSL (SMORG scale – small and medium sized organizations )
Understand if a cohesive Linux manageability stack exists today--Understand manageability focus from a Linux Sysadmin’s perspective (SMORG scale)

Once we defined what we needed to get into, yet another realization dawned on us, which was the sheer size and volume of the data and information that was staring at us in the face. Let’s just say “overwhelmed” was a mild word compared to what we were looking at. My colleague, Steve Zarkos and I immediately realized that it was time to trim the scope of what we were doing and limit ourselves to what’s achievable in three months and with two people J. This called for drawing up what we considered to be “out-of-scope”, which was:

II. Out-of-scope:

Enterprise-level Management Stacks such as HP Openview, Tivoli etc.
Security and user management, which is a very large area in itself, and would include policies, permissions, authentication, identity management etc.
Asset management frameworks that allow for inventory management, asset tracking etc.
Disaster recovery and planning which would include backup technologies and data protection management

III. Systems Manageability Project Methodology:

The approach taken for the project was simple and scientific. The project was divided into three stages:

Stage 1 – Discovery and Definition: We would explore, uncover and scour the Linux Manageability landscape for toolsets, apps and frameworks prevalent in the SMORG scale workspace today
Stage 2 – Lab Implementation: We would implement the results of our research which would be actual Linux and Open Source products in the Lab
Stage 3 – Analysis and Conclusion: After discovering, defining and implementing Linux and Open Source based manageability toolsets, apps and frameworks in the lab, we would spend time understanding their form, function and experience to draw conclusions

IV. Systems Manageability Project Ontology (classification):

The hardest and most challenging aspect of the project was to develop some sort ontology, characterization or classification of the manageability technologies prevalent in the IT environments today. The diagram below represents the overall "buckets" defined as part of this exercise. Each section of the diagram is broken down to provide a detailed breakdown of each of these Systems Manageability classifications represented:

In the next blog to follow, I will break down the first segment of ontology i.e. “Provisioning and Deployment” and discuss our research with all of you. Meanwhile, we always look forward to hearing from you, our audience and urge you for any feedback you may have about the topic. Thank You for tuning into Port25.

Cheers!

Systems Manageability - Part 1: Why Manageability Matters

kishi — Wed, 21 Mar 2007 20:09:00 GMT

Impetus: This is the 19th year I have spent in the Information Technology business, out of which more than 15 were spent designing and implementing IT environments of various scopes, platforms and sizes. Among several similarities and differences between each implementation, a few constants always emerged, and my favorite of all: Systems Manageability. This question always got asked during every high-profile design review meeting. You know the meeting I’m talking about, the one with the CIO and the IT Director are sitting across the table and asking how we’re planning on managing the environment. The main concern you could see in everyone’s expression was “what sort of manageability needs to be built around for operations and support ?”. So why does manageability matter ? Let’s start with what people imagine, when they think of Systems Manageability. It means different things to people in different roles: the Infrastructure folks imagine uptime and redundancy, the Developers imagine reliability, the Business Managers imagine efficiency. But the answer is so overwhelmingly obvious. Had it not been for systems management toolsets, apps and frameworks, ITPro’s and Admins would be worried sick about everything from uptime to reliability to scalability. So yes, manageability matters, a lot, because it’s the knowledge that “all’s well and running smoothly” that matters to everyone from a CIO to a Developer to a IT Admin.

Importance: Now that we have established the impetus behind why Systems Manageability matters, we should now address the importance tied to it, such as:

Increasingly complex and heterogeneous environments need increased attention: Pick any environment of your choice, whether it’s where you work or the one you hear about the most. It comes with its own share of challenges and oddities. What differentiates one environment from the other is the manner in which it is “run”. The more complex you hardware, software or network stack, the greater is the manageability tied to it. This means that with every component you add to your existing system, you’re only increasing the complexity even further. So is that a bad thing – NO. Because it’s the environment that should be designed with “adaptation” in mind and not the other way round. Thus, having a complex environment simply translates to keeping an eye on more things. Systems Manageability plays a key role in this scenario
Infrastructure sits at the “core” of IT: Here’s an exercise for any of you that may be interested – the next time you see or hear a technology professional being interviewed about a certain “application” or tool that they’ve developed – try to imagine an entire infrastructure that needs to support and run that “tool” or “App”. The exercise will make you think about what’s going on in the mind of an IT Admin who is responsible for running your environment implementation and why Infrastructure is a big deal. Systems Manageability plays a key role in this scenario
End-user productivity has a directly proportional relationship to Performance Tuning and Optimization: Growth comes in spurts and bursts and never an even pace as we all know. This means, various pieces of your hardware, software, toolsets are implemented in various growths and phases. Simply put, growth does not follow a pre-chartered course and timeline. This makes ongoing performance tuning and optimization a necessity. And it has its benefits - it allows you to see the app or toolkit from the eyes of the people who made it. It also gives you the “know how” to make the specific changes in thresholds and values that could mean a difference of night and day, in terms of efficiency. Systems Manageability plays a key role in this scenario
Striking a balance between Manageability and Flexibility is tough: Identity Management and Security issues are now a mainstream topic of discussion and more and more attention is being paid to system security, access and authentication framework. The single reason for putting these in place to ensure that “you are who you say you are”. On the flip side, overly managed and restrictive environments can limit the flexibility of what an end-user can do. That’s why it is always tough to strike a balance between a system that is secure / well managed and one which seems more “seamless”. Systems Manageability plays a key role in this scenario

In the next seven blogs or so to follow, you will get a detailed breakdown on the Systems Manageability project that we have completed in the lab. We will be covering the Project Methodology and Project Ontology in my next blog. As always, send us your comments and feedback and THANK YOU for tuning into Port25.

It's not all technology...

MichaelF — Sat, 03 Feb 2007 02:00:00 GMT

If you live on the eastside in the greater Seattle area, you may be among the million-plus people who were out of power during the massive windstorm last month. As Bill Hilf pointed out in his previous blog “what matters” (see blog), these are rare intermissions which allow us to step back and understand what sets us apart as “intelligent” beings. After reading Bill’s blog, I started pondering over the role that technology plays in our daily life and my reaction is that this role is a bit too exaggerated. Why you may ask - well, let’s see: it took the million plus people spread across all over the eastside neighborhoods, close to nine days before power was fully restored all over. A lot of us took the time and caught up on chores we had been putting off for a while, some of us shared childhood stories around a fireplace and talked about things that sat underneath the surface, suppressed by constant engagement with a laptop or an MP3 player. Unfortunately, there were also a handful of others who didn’t make it through this trying time and died of carbon monoxide poisoning.

We talk a lot about “technology” here on Port25 and being involved with Port25 gave me a perfect opportunity to draw parallels in what I was experiencing and comparing it to the “hype” about the “advancement” of technology and how it has improved the quality of life. Making a radical shift here, let’s at least agree on a simple fact that without an “evolved” attitude, these advancements will and cannot replace the “self introspection” we all need to do from time to time. The power outage was a perfect time for that, a time that made us all realize that there is a larger reality out there which we should never be oblivious of. We need to acknowledge that to fully understand and take-in the benefits that technology has already given us rather putting our mind to think about “Bigger, Better, Faster, More”.

So is technology bad – NO ..HELL NO!! Just because we can text messages to others, while listening to our collection of 10,000 songs on MP3 and driving an electronically controlled, fuel injected vehicle, let’s NOT make the assumption that playing around with a few gadgets has made us a better person. Let’s examine this more closely, shall we, and try to truly comprehend the role technology played when we were all sitting without power:

1. Abstinence: No matter how juiced up my MP3 player was or how many extra battery packs I had for my laptop, by Day 3, I was probably staring at a cozy fireplace wondering what my life was like before High-Definition football broadcasts and movies streaming through extenders. For many of the folks who put their comments on local newspaper and blogging sites later, this was a “revelation”. My question at this stage was “Gee…there’s very little technology in this present moment, isn't there ?”

2. Chaos: I remember, it was our second day without power and we were driving through a major traffic intersection in Downtown Bellevue. As we drove past a gas station, we saw police cruisers on both sides of the entrance to the gas station with troopers standing guard w/ guns. Later someone told us that fights had broken out at the gas station as people started to cut each other off and next thing you know, it was chaos. I was also told that in a grocery store, they ran out of bottled water and tempers flared between those who had bought crates of water and those who were left without any to buy. So my question again “would any type of technology or device have truly helped the situations here in some way”……NO, not without us taking a more civil, educated and an more evolved approach

3. Awareness: As hundreds of thousands of people tried to cope with loss of power, in the middle of December, all sorts of stories were emerging on how people survived. My boss, for instance, was left with no choice but to use construction grade wood and cut it into 2x4’s for his fireplace to keep his kids warm. Others resorted to picking up downed trees and gathering them for burning wood. A lot of folks flew or drove out to other towns to their friends and relatives who did have power. All through this a thought kept gnawing at me repeatedly, and does so even today. I thought “We live in one of the most technologically aware, literate and advanced places in the US and it takes DAYS to restore power ???”. With every passing hour I kept reflecting back to my days in IT Operations where outages were measured up to three decimal points. I got a chuckle out of thinking back and wondering what “metrics” could be applied to this situation, I guess the curve is really messed up now…. ***. And guess what “there’s no technology anywhere in sight unless you lived in the Seattle downtown area, which wasn’t affected”

4. Fragile Infrastructure: I was also compelled to ask myself and those who had been living here for longer than me, if this is how fragile our infrastructure really is. After using my daughter’s stash of quarters from her piggy bank (no joke), I drove by a pay-phone not too far from my house twice a day and checked the status w/ the power company to get an estimate on when we could see the lights coming back on. This is no exaggeration, no matter how many times I called, I ALWAYS got a different answer. This led me to think about technology in a the context of the power company. A few random thoughts that followed were:

What type of grid-management technology are they using to keep tabs on what grids are working and which ones are down ?
What type of communications technology are they using internally since no one had one consistent answer about power restoration ?
What type of technology is helping them to bring the power lines back up ?
And MOST importantly, what type of technology will they use to avert or be prepare for such a disaster in the future ?

So did I get the answers I was looking for, nope. Did I learn a lot of lessons on how to be prepared for an emergency – you betcha. And as far as my curiosity on the role of technology in power restoration process, no one would answer my questions and I can understand to some degree. But I’m still curious as to what role technology adoption would have played in this scenario. Whether a specific technology, if adopted pre-emptively or post-incident, could have made this less cumbersome and trying for everyone. Are they using the right technology for the right job ? Are they fully aware of the potential of technology and the possibilities ?

In the end, I came to a simple realization which was…it's not all technology…there’s MUCH more to life and to our existence...

What Lies Beneath: Setting up underlying HPC tools

kishi — Thu, 21 Dec 2006 22:34:00 GMT

This blog continues what I started writing about w/ Thinking About HPC Infrastructure and what Frank wrote in about in Overloading Clusters.

After reading thru the previous blogs on HPC, someone might ask “What are some of the core components of HPC ?”. After all, once you’ve seen the outside of a Maserati or a Pantera DeTomaso, you’re not going to be satisfied just by ogling at it. Even after a test drive, the engineer in you will want to pop the hood and see what’s inside. Taking a similar approach let’s uncover some underlying HPC technologies by looking at any basic HPC setup. Once all the provisioning has been completed, the HPC system will be physically deployed with an OS and relevant drivers, utilities etc. Yet, before the actual HPC application can get installed across, there remains a critical step in the process, i.e. configuration of cluster and file system along with any tools and interfaces such as MPI (Message Passing Interface) etc. After peeling through the HPC application layer, its worthwhile to do a “deep-dive” into what really runs the HPC clusters. A broad category of these tools are:

Cluster Management tools e.g. CSM
Job Scheduling tools e.g. SCALI, Maui
Resource Management tools e.g. Torque

If you’re trying to understand the “WHY” behind the existence of these tools and their importance, take a look at Cluster Management for example. Cluster configuration, installation and management can be difficult and requires intimate familiarity with the HPC hardware, OS, underlying architecture etc. Without specific tools that attend to and manage specific underlying HPC sub-components, HPC just won’t be what it is. So, it is worthwhile to understand the unique installation experience of the tools, such as the ones listed above to understand the complexity of HPC systems. Ready – let’s dive in to the installation and function of these tools:

1. SCALI: The SCALI management and MPI software packages provide deployment, monitoring and job scheduling services for a cluster. After you deploy this software, you will be able see all the compute nodes that may have been preconfigured or are configured on your system. Scali will enable you to monitor the systems and run jobs using the SCALI graphical interface. In order to license the SCALI software, you must utilize the scainstall command to produce a license request file. This file can then be sent to SCALI to receive a permanent key. For those that need some hand-holding through this, luckily SCALI provides very comprehensive documentation on their website. A large portion of the SCALI Manage User’s Guide is dedicated to pre-setup planning and configuration of the cluster and the network. The documentation provides detailed recommendations about how you can set up their Ethernet-based network environment and out-of-band management network. The documentation also provides a general overview about how to install and configure higher performance interconnects, including bonded Ethernet, Infiniband, Myrinet and SCI. The SCALI Manage interface provides simple tools to assist in configuring and testing DET, Infiniband, and Myrinet devices for use with the SCALI MPI implementation. The SCALI MPI software supports multiple Infiniband stacks including Mellanox, Topspin, Voltaire and Infinicon.

2. HP-MPI: HP-MPI is Hewlett-Packard’s Linux-based implementation of the Message Passing Interface (MPI). Many of the utilities distributed with HP-MPI are similar to other common MPI utilities such as MPICH - e.g. mpicc, mpirun, etc. In order to utilize the HP-MPI software, a license is required for each CPU core in the cluster. To obtain a license file you are required to obtain the MAC address from each node (typically eth0) and input that information into a form at licensing.hp.com. The resulting file can then be copied to the compute node. The HP-MPI software is non-functional until licensing files are generated for the nodes

3. CSM (Cluster Systems Management): The CSM software suite is designed to automate the deployment and management of cluster nodes. Nodes can be remotely installed with an operating system as well as the CSM software for later monitoring. The CSM software supports RedHat and Novell on multiple platforms. In order to obtain and install the CSM software one must register with IBM’s website and download the required RPMs. In order to configure CSM, it can remotely install the operating system and/or the CSM software on the compute nodes. Much like Platform ROCKS, CSM makes use of PXE functionality and RedHat’s kickstart or the autoyast software to remotely install the operating system. The CSM software provides multiple methods for defining the nodes that should be deployed and managed:

a. The first method involves creating a hostname mapping (hostmap) file, which is a colon-delimited file that defines a number of attributes of each node
b. The second method also involves manually creating and editing a “node definition” (nodedef) file. This is the method suggested by the documentation for use with small clusters

Proper remote power and remote console capabilities greatly ease the administration and deployment of the compute nodes, however according to the CSM FAQ remote power management is not absolutely required. All the compute nodes must be rebooted (remotely or manually). They are then PXE booted and installed with RHEL4 using the kickstart installation system.

4. Maui and Torque: Both Torque and Maui are free software which must be compiled from the source distribution on the head node. Maui is an open-source job scheduler for compute clusters. It supports a number of task management features not found in other parallel batch processing software including policy-based scheduling and prioritization of tasks. Torque is an open-source resource manager for managing compute nodes and scheduled jobs. It can integrate with Maui to provide additional features for scheduling and managing scheduled tasks. Installation of Torque can be done using the guidance available in the Torque 2.0 Admin Manual .

5. Platform Rocks: Platform Rocks is a cluster deployment software that facilitates the deployment of various software stacks (“rolls”) onto the compute nodes. The software is capable of deploying the base operating system and utilities required for cluster administration, management and scheduling. The software can also manage configuration and updates to ensure consistency throughout the cluster. Platform Rocks is a suite of utilities that are packaged together as separate installable rolls. One of the main goals of the software is to allow for easy installation and integration of third-party rolls and applications. One unique aspect to the Platform Rocks installation approach is that the software installs an operating system on the head node, and also installs all the required rolls at the same time. The software can also automatically set up the subsystem required to install an operating system and other packages on the compute nodes (such as management agents, etc).

That about does it for a quick “deep-dive”. Let me insert a gentle reminder that these are not the only cluster or resource management technologies out there in the HPC space but rather the ones most prevalent. If you have additional tools that you have worked with, we’d like to hear from you and thank you for tuning in to Port 25. HAPPY HOLIDAYS!

Thinking about HPC Infrastructure

kishi — Fri, 01 Dec 2006 19:21:00 GMT

I started the first HPC blog (See “previous blog“) with an understanding that HPC is an area where there has been a surge of activity from a development/investment standpoint. This segment of Information Technology has experienced a heightened level of engagement from OEM’s and partners, all trying to meet the growing computing needs of their customers. So after getting a basic understanding behind the importance of why HPC matters, the next logical step that needed uncovering was “How to think” about HPC Infrastructure and tap into the “wisdom” behind managing it. You might ask why this is relevant. For starters, setting up HPC Infrastructure is an experience that, just like any other infrastructure, be it Network or Storage, requires intricate planning and intimate familiarity with its individual contributing components. In case of HPC, let’s just say you really need to know your nodes J. Let’s talk more about what’s involved in setting up an HPC Infrastructure and how to think about it as a whole:

1. Investment Impetus: To successfully plan and design an HPC Infrastructure, the first and foremost step should be to “look beneath the surface” . This simply means to understand, the primary reason for investing in HPC. The demand for HPC equipment, linked to a set of business objectives should have clear purpose around the outcome and expectation. This is specially true today than at any other moment in time because the consumption of HPC cycles, specifically in the research and development areas across all verticals has seen a steady 70% growth over the past four years (Source: primeur ). Despite this tremendous growth in the proliferation of HPC technology, the growth pattern itself is sporadic. One of the reasons for it may be the complexity, not only in terms of design but also in terms of consumption as well. Take the case of SHARCNET in Southern Ontario that developed a long range plan around adoption and implementation of HPC technology. According to the report, some of the elementary challenges around planning for HPC emerge from the fact that “it is an enabling technology for an extremely diverse set of researchers”. This embodies the essence of the sentiment behind the complexity and diversity predominant in the HPC space.

2. Planning and Designing Hardware: While thinking about planning and designing an HPC infrastructure implementation, I spoke to several folks in this area, drew from a decade and a half of my experience as an Infrastructure Architect and thought of some key areas that I would consider. These include:

a. Facility considerations (Rackspace, Power and Cooling): Talk to any enterprise level Datacenter manager what his/her top 10 pain-points are and you are bound to hear the words “rackspace, power and cooling” in what follows. Dig deeper and you’ll realize that in any datacenter, there’s a fixed number of colo’s (Colocation) you can populate based on the HVAC designs. This means that rackspace is what’s at a premium in each of these colo’s with every “u” accounted for. Packing in dense chipsets in small form-factor server add to existing power and cooling challenges
Translation – you need more outlets and more airflow per rack than what you did a decade ago with a handful of 4 and 5u servers taking up the entire rack
b. Physical Plant planning: Quoting the resident HPC Guru Frank Chism who says “I cannot over emphasize the importance to planning for physical plant in HPC deployments. Things like room and raceways for well managed and planned cabling. HPC uses more cable than anything except maybe SAN. Also, pay attention to floor loads, air flow, clean and redundant power. Finally, never never forget out-of-band management. Deep subfloor really helps with all that cabling”.

Translation – Effective HPC performance calls for an effective HPC design, which includes tweaking hard as well as soft components. These components can be as covert as chip-design or as overt as subfloor depth.
c. Hardware and Processing Power: Pushing the envelope on hardware and processor architectures today translates to increased performance (the heart and soul of HPC). Adding energy efficient hardware on top of the architecture amounts to greater investment in raw computing power, which in turn translates to building a sound HPC infrastructure. The key advantages one needs to look for in this scenario are faster data access and increased instructions. The word “performance” is repeated throughout the theme of this topic because it IS what HPC is all about, the ability to reduce the number of cycles to process data. Addressing the hardware and processing specs as part of core requirements ensures a smoother build-out.

3. Implementing HPC Tools and Software: Like any other piece of hardware, a HPC cluster is just that until software and tools exploit the underlying architecture to drive results and performance to do what it does best – compute. When thinking of some core elements of HPC tools and software, here’s how I thought to break them up:

a. Setup and deployment systems: Setting up HPC clusters goes back to what I said earlier in Section 1 – what do you want to do with it? Although there are various ways and methods that allow you to drive the software and installation experience of an HPC system, the bottom line is that this depends to a great extent of what components make up the genetic composition of the HPC cluster you ordered. Taking a look at some HPC software setup and deployment tools out there, a few mainstream ones are SCALI and HP-MPI (HP’s message passing interface). These packages provide deployment, monitoring and job scheduling services for managing and administering an HPC cluster just like IBM’s CSM (Cluster Systems Manager). In the Open Source space, there’s Maui and Torque, that work as job scheduler and resource managers for managing compute nodes and clusters. Platform Rocks is another suite of utilities that allow installation and integration of third party apps

b. Parallel FS: This is truly what I think is going to be the frontier for some intense activity over the next few years. Using Wikipedia’s description, “Distributed parallel file systems stripe data over multiple servers for high performance. Some of the distributed parallel file systems use object storage device (OSD) (In Lustre called OST) for chunks of data together with centralized metadata servers such as Ceph Scalable, Distributed File System from University of California, Santa Cruz. (Fault-tolerance in their roadmap.), Lustre from Cluster File Systems. (Lustre has failover, but multi-server RAID1 or RAID5 is still in their roadmap for future versions.) and Parallel Virtual File System (PVFS, PVFS2)”.

Deep-Dive: At Base, parallel file systems are global namespaces for files that achieve high bandwidth via parallelism. That bandwidth comes in three dimensions, high aggregate bandwidth, high single stream bandwidth, and high metadata operations per second. No one seems to have achieved high performance in all of these dimensions. Don’t forget that the volumes of data are so large that backup is a major undertaking and thus, reliability is required as well. Further, nobody seems to be able to make a parallel file system that performance well for high-speed data for short I/Os, like say you do when compiling a major application

c. Multiple Networks: A final comment on implementation of HPC is that HPC often has multiple networks. For example, it does little good to have a parallel file system that delivers gigabytes per second of data to single nodes if the network can’t handle that much bandwidth!

So in conclusion, here’s a recap on the learning behind setting up HPC Infrastructure:

Comprehensive understanding beneath WHY you’re investing in HPC and what you expect as an outcome
Deep familiarity with the core HPC Hardware and design components
Facility and Physical plant considerations to ensure adequate cabling and subfloor space
Visibility into prominent HPC based software and toolsets
Understanding the three dimensions of bandwidth
And finally accommodating the concept of “Multiple Networks” into node design to accommodate the required bandwidth

Look forward to getting back to you with more on HPC over the new few weeks again. Until then “Happy Computing”!!

HPC - The way all computing will look...

MichaelF — Wed, 01 Nov 2006 22:37:00 GMT

I have been itching to write on this subject ever since I first met w/ Doug Lora and Frank Chism. High-Performance computing – Wow! The first time someone explained the concept to me, I couldn’t help but visualize a scene from the movie “Blade Runner” and the futuristic feel of how Supercomputers actually work. My interest took me deeper into the heart of HPC to try to get my head around what HPC really is all about. High-performance computing systems, also referred to sometimes as “Supercomputers” are more prevalent today across prominent verticals such as Oil and Gas, Bioinformatics, Finance and Entertainment than ever before. Wikipedia described HPC as “ Supercomputers and Computer Clusters i.e. computing systems comprised of multiple (usually mass-produced) processors linked together in a single system with commercially available interconnects. Usually, computer systems in or above the teraflop-region are counted as HPC-computers” . An HPC Cluster is usually implemented to provide increased performance by splitting a computational task across many different nodes in the cluster.

Applications that run on HPC systems are prevalent in heavy-duty research and experimentation to engineering scenarios including transactional processing, data warehousing, computational fluid dynamics, virtual prototype testing etc. The evolution behind clustering technology has a lot to do with the growing adoption of this technology as well. The additional element that has made this technology very attractive is price. An HPC cluster can be implemented at a fraction of the cost today as compared to 10-15 years ago. Take a Cray Y-MP c916 supercomputer that cost close to $40 million 15 years ago. Today, you can get computing power very close to that for almost $4,000. The proof of this adoption is in the fact that every industry vertical is deploying HPC. From a “mainframe” approach that existed decades ago, the implementation trend of this technology is gravitating towards decentralized grids and clusters.

So why do I say that this is the way all computing will look - HPC is already a $9 billion growing market (source HPCwire). Evolution in this sphere is occurring at a blazing speed and the demand for HPC systems across various verticals is expected to multiply. Bottom-line – HPC will play a very key role in how computing power is used, stacked and scaled. Not only that, the development of HPC propagated file-systems will be an area that we all should watch very closely over the next few years. Let’s also fully realize the “impact” of this technology in the business space. If done right, HPC clusters hold the key to superior systems performance, while maintaining reasonable economies of scale. Delving into the benefits of these clusters, which until recently was a domain of the scientific community, is literally like lighting a fuse to an explosive. I say this with a strong ethos because we have yet to recognize all possible uses of HPC clusters with their underlying potential. According to some researchers at INIST-CNRS, “Analytic methods, statistical modeling, and pattern searching algorithms that are common in scientific computing can now be applied to the vast amounts of operational and historical data generated by business transactions to extract knowledge that can be used for competitive advantage”.

A nagging question still remained in my head as to the WHY behind the importance of HPC? I kept looking for the single reason behind the heavy investment in this area and why it’s such a critical component of highly-complex computational analysis being done. The biggest advantage or theme that emerged from wherever I looked was that HPC is one of the few tangible technologies out there, whose sheer computing power helps solve highly complex computational workloads and problems. This is not to mention the solitary advantage of using this technology – time. Time that it takes to resolve highly complex workloads is greatly reduced with a faster outcome. And anyone reading this blog knows the value of time and how it is THE most valuable element of all. And everyone knows, no matter how evolved and fast hardware can get, there will always be bleeding-edge problems that will demand processing power beyond what the best clusters can provide.

My endeavor here at OSSL, is to understand this topic from the ground up, have an open discussion on the subject matter as well as educate the audience along the way. And how do I plan to do that – well, we have started venturing into doing more with HPC and understanding the various HPC platforms and technologies out there. Over the course of the next few months I’ll be sharing more on this subject with all of you including market trends, evolution of HPC, Grid Computing Scenarios, “chip” supercomputing etc.

-Kishi

Centeris was Here

MichaelF — Thu, 12 Oct 2006 16:30:00 GMT

Capturing the sentiment from my colleagues, Anandeep, Hank and Sam – I LOVE THIS JOB !!! Last week I had a chance to meet w/ Mozilla, watch Sam interview Steve Wozniak and the wonder of it never ceases to amaze me. This week, we had a chance to have lunch w/ Barry Crist, the CEO of Centeris, Krishna Ganugapati their VP of Development and Chuck Mount, the VP of Marketing. Centeris is a company based out of Bellevue, WA that makes the Likewise product which allows Linux Servers to be manages within a Windows centric environment. We all got off to a great start in our discussion because one of the core and common goal that ties us together to the charter of partners like Centeris is “Interop”. Yes, Interop, and finding more and better ways for Microsoft and non-Microsoft platforms or products to co-exist and thrive. This is a really important charter for not only the Microsoft Open Source Software Lab but for also all of Microsoft. After brief introductions to the Program Managers of the Team and our beloved Penguins, we got down to discussing what Centeris as a company was all about and more importantly, what is it that “Likewise” did. Barry and Chuck gave us a very good insight into what the overall focus of Centeris was about and why there is a prominent need for providing this functionality in a heterogeneous environment.

If you’re an ITPro managing a small, medium or an enterprise-wide shop, you know how diverse and today’s implementations are and/or can be. This translates to greater complexity while managing your environment, which as the market data will tell you, is rarely single-platform centric. Thus, making accommodations for manageability of the diverse platform-portfolio is a skill that we all much acquire sooner than later. This is where Centeris fits in perfectly for several reasons because it is extending Windows-based manageability and windows-based tools towards day-to-day management of Linux servers and improve interop. This also means that organizations that have tight budgets can continue to manage their environment with existing skill-sets under tight budgets.

The way Likwise works is where the console is installed on the Admin’s machine, the agent (which is an open source product) is installed on the Linux Server/s and using the console, these servers are managed using the Microsoft Management Console (MMC). Likewise Open Agent includes server-side components (that work w/ Samba) and client-side components (that work w/ MMC) The functionality that is extended to the Linux systems is possible through RPC’s and SOAP (Simple Object Access Protocol). Likewise open agent is available on sourceforge.net and has been released under the CDDL (common development and distribution license). We found the approach that Centeris took towards Linux manageability to be very simple and ITPro centric.

The highlight of our discussion yesterday was getting to know more about Krishna Ganugapati. Krishna spent 10 years at Microsoft from 1993 to 2003 most of it in Windows development team. After we got into deep discussions, we found out that Krishna was the inventor of ADSI (Active Directory Services Interfaces), the preferred means for accessing Active Directory. Krishna also led the development teams for Windows IPSec and Window wireless security through the Windows 2000 and XP releases. The interaction that followed between all of us, penguins, PM’s and Krishna was very rewarding. Krishna got into the guts of how manageability is being approached as a concept by Centeris. The big takeway after we saw the Centeris demo, for me was that there doesn’t always have to be a steep learning curve every time new technology is introduced into the environment. Sometimes, its easier to manage new technology with familiar tools and that was a very novel concept that I walked away with yesterday. It also affirmed my faith as to why “Interop” is as prominent, as important and as critical as it is to us and to the success of Microsoft.

Thanks Centeris !!

Managing Global Infrastructure Services: Kishi interviews Arne Josefsberg

admin — Wed, 30 Aug 2006 18:17:00 GMT

Kishi Interviews Arne Josefsberg General Manager of Infrastructure Services in Windows Live Operations. He is responsible for the strategy, design and operation of the online infrastructure that forms the foundation for Microsoft’s online businesses. His areas of responsibility include: global data centers, networks, hardware and operating systems standards; and foundational shared system level services such as caching, load balancing, DNS, Active Directory, back-up/restore, remote management access, content replication etc.

During Arne’s 20 years with Microsoft, he has held technical leadership roles in a number of areas. His Microsoft career started as a Technical Manager on the team that built the Nordic subsidiary region business. Since then he has held management roles in Product Support Services leading OEM and ISV customer teams and OEM hardware compatibility certification; as well as leadership roles in Information Technology managing customer systems applications and IT infrastructure. As a member of the MSN 1.0 launch team, Arne gained early knowledge of the online services business building the dial-up access network as well as data center operations.

Prior to joining Microsoft, Arne worked in systems engineering at Intel Corporation specializing in CPU, operating system and software consulting for Nordic customers. Arne holds a Masters degree in Physics from the Lund Institute of Technology, Sweden.Outside of work he enjoys spending time with his wife and two children, as well as photography, reading and various forms of exercise including soccer, weight training and cycling. Arne shares some insight into the challenges faced when managing a global infrastructure and provides some tips for success.

Video: Managing Global Infrastructure Services: Kishi interviews Arne Josefsberg

Alternate Video Format
Download MPEG4 Video

Infrastructure Management and Strategic Design: Part 4 – Service Management Frameworks

jcannon — Tue, 01 Aug 2006 16:51:00 GMT

Today, the IT departments offering and managing various IT Services might find themselves in what I would call a “pressure-cooker”. They are faced with a multitude of tasks and added pressure to maintain daily operations while driving efficacy, managing the growing complexity of Service Offerings and most importantly, doing so while keeping pace with the industry best practices. This has been one of the most explosive areas of growth and re-examination for the past few years. Back in my Ops days, I trained under ITIL i.e. IT Infrastructure Library and MOF i.e. Microsoft Operations Fundamentals to get a first hand look at some of the best Service Management practices in the industry. No matter how good I thought our Service Management practices might have been, I could not help but to think in terms of the maturity level of the Services that can be achieved by applying these principles. When you get down to it, you realize that the heart and soul of effective Service Management lies in how mature the offering and support model is. I have learnt a lot from the ITIL Service Management Essentials course, which I attribute to research and practices that have gone into developing these models. I’d like to share w/ you what made sense to me:

Bridge Concept: As described in various ITIL formats, Service Management can be referred to as the “bridge layer” between Business and Technology. It is through the conduit of Service Management that core business needs as well as core technologies find their match with each other. This is attained by aligning the Business Needs and Goals of the organization with the various technologies and IT functions that can map to these overall goals
Think Framework: Once you have scoped out, what may be a void in the service hierarchy of your organization, make a commitment to implementing a Service Management and Delivery framework that is suitable for your org. One size DOES NOT fit all and although I am only talking about ITIL concepts here, there are several methodologies you can explore before making the plunge.
Benefits of implementing a Service Management Framework: In just a few minutes after sitting in the ITIL Essentials Training class I was able to get a very crisp idea of what the benefits of implementing a Service Management Framework are. To name a few –

Raising the bar on Service Delivery Quality
More accurate alignment w/ Business Needs
Enhanced relationship between service provider and consumer
Deeper visibility into service complexity
Driving efficiencies with optimal resource utilization

Goals for implementing a Service Management Framework: if and when you do make the commitment to implementing a Service Management Framework, here’s a quick set of goals you can set when you proceed with the implementation of Service Management:

Put an SLA ( Service Level Agreement) or an SLO (Service Level Objective) around the critical and non-critical services you offer, respectively
Manage and monitor the implementation and practice of these SLA’s and SLO’s
Create and publish a service catalog describing the services offered by your IT department/division
Monetize the service management offerings, even if your customers are internal. This will help quantify the service effectiveness and bring measurability across the board

Additional Resources: the itSMF or IT Service Management Forum (www.itsmf.com ) is an independent, non-profit, user group distributed all across the world that has dedicated itself to exploring and promoting IT Service Management concepts and practices.

I am very eager to hear back from those of you that are an integral part of the Service Management Lifecycle. Please share your experiences, challenges and learning with us.

Kindest Regards and have a great week ahead!