Port 25: The Open Source Community at Microsoft : Technical Analysis, Kishi Malhotra

Systems Manageability Part 7 - Log Management and Analysis

kishi — Tue, 07 Aug 2007 15:57:00 GMT

Level-Set – Log Management: This section includes open-source technology directed primarily on host-based logging, log file rotation and log file analysis. Many of these tools are very common free and open-source software tools that are distributed and preconfigured with most of the major Linux systems, including major vendors such as RedHat and Novell.

I. Logrotate

Logrotate is a very popular application utilized in a number of Linux systems, including all RedHat and SUSE based systems. The logrotate utility typically runs periodically via cron, a task scheduling application. The utility will read a configuration file (/etc/logrotate.conf), and archive and compress log files according to the configuration. Administrators can configure when log files should be rotated based on age and size, and how long backlogs should be maintained. Older archived log files can then be swapped out and replaced with newer archives.

II. Syslogd and klogd

Typical Linux systems utilize a syslog daemon to capture log messages from userspace applications and write them to text-based log files or send them to a logging host over the network. The syslogd daemon is often accompanied by a klogd application which is designed to capture and log kernel messages.

The behavior of the syslog daemon can be configured via the /etc/syslog.conf configuration file. All messages captured by syslog are categorized by facility and priority. Messages can then be sent to particular log files or logging hosts, or dropped completely based on their facility and priority attributes.

Facilities

Priorities

- auth or security

- authpriv

- cron

- daemon

- kern

- lpr

- mail

- mark

- news

- syslog

- user

- uucp

- local0 through local7

- debug

- info

- notice

- warning or warn

- err or error

- error

- crit

- alert

- emerg or panic

List of syslog facilities and priorities.

III. Syslog-ng

The syslog-ng application aims to be an enhanced drop-in replacement for the traditional syslog daemon. It provides many of the same features of the standard syslog daemon, but includes additional features such as advanced message filtering based on content, remote logging via UDP or TCP, and the ability to write log files to a database such as MySQL or PostgreSQL. More recent SUSE-based systems such as SLES10 have switched to syslog-ng as the default syslog server.

IV. Viewing Logs

Most log files on a Linux system are stored in plain-text, which means they can be viewed and parsed using a number of different command-line tools. Typical utilities such as tail, head, grep, cat, less, more, sed and awk can be used to view and filter log messages via the command line.

There are also a myriad of utilities designed to parse and view log files via a GUI or web browser. Some utilities are even designed to handle specific log formats, such as those generated by Linux’s Netfilter firewall subsystem.

GNOME System Log Viewer

The GNOME system includes a GTK-based system log viewing application that displays system logs via the GUI.

YaST System Log Module

SUSE-based systems using YaST typically include a module called View System Log (called internally as view_anymsg). Similar to the GNOME System Log viewer, the YaST module allows an administrator to view many of the various system logs without using the command-line.

V. Log Analysis

LogWatch

The logwatch utility is designed to parse system logs and located any entries that might indicate security threat or system failure and send an email report to a designated address. Logwatch is distributed with RedHat Enterprise Linux systems. The following is an excerpt from the RPM description:

“LogWatch is a customizable log analysis system. LogWatch parses through your system's logs for a given period of time and creates a report analyzing areas that you specify, in as much detail as you require. LogWatch is easy to use and claims that it will work right out of the package on almost all systems. Note that LogWatch now analyzes Samba logs.”

LogWatch is typically executed periodically via cron, a task scheduling application.

LogCheck

The logcheck utility is a part of the Sentry Tools project that also includes portsentry, a utility designed to detect port scans. Similar to the LogWatch utility, the software is designed to parse system log files, find log entries that may indicate security problems and send an email to a preconfigured address. Also similar to the LogWatch utility, logcheck relies on the standard cron utility to be periodically executed.

That does it for Log Management and Analysis section. We have one last blog to go and certainly hope that you found the information we have captured for you useful. If you’re running any special toolsets or customizable scripts for log management and analysis and would like to share your experience with us, please send us your feedback and as always, THANK YOU for tuning into Port25.

Systems Manageability Part 3 - Provisioning and Deployment

kishi — Thu, 03 May 2007 17:29:00 GMT

I want to start this blog with a note of Thanks to Ajay Mungara, the Manageability Developer Community Manager from Intel and “einhverfr”, both of whom gave some very constructive feedback on the previous blog. In the next six blogs to follow, including this one, I will do a “deep-dive” into the six specific areas we covered under the “Systems Manageability” ontology.

Let’s start this blog with the first of the six categories from the ontology - “Deployment and Provisioning”

Level-Set: Deployment and Provisioning as we understand it, encompasses all tasks related to the initial installation of an operating system on remote system, as well as post-installation of software on a remote system. Much of these toolsets are geared toward automated system provisioning and cloning. There’s lots of different tools out there that can be used but we have focused on the most popular ones, namely Kickstart, Autoyast, “Bare-Metal” provisioning and RedHat Network. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I.KICKSTART: is an automated installation utility for RedHat-based systems, including Fedora Core and RHEL based systems. Kickstart software requires the creation of a configuration file (similar to an “answer file” in Windows lingo) which contains all the information the installation program will require to install the operating system. The configuration file and all the RPM software packages are typically kept on a remote server such as a HTTP or FTP. The location of the Kickstart configuration file is typically passed to the kernel at boot time. For example, once the bootloader (GRUB, LILO) loads, the user is often presented with a “boot:” prompt that allows the user to pass arguments to the kernel. To load a kickstart configuration file from a remote server one would type the following:

boot: linux ks=http://<server>/location/of/kickstart.cfg

The administrator can then create multiple Kickstart configuration files for different configurations. With the addition of a PXE-based server, much of the provisioning process can be automated. A Kickstart configuration file is simply a text file, which can be created and edited manually with any text-editor. A GUI-based Kickstart file creation program called system-config-kickstart is also available from the Fedora and RHEL distributions. Both kickstart and autoYaST provide the ability to run arbitrary commands during the post-installation phase. This allows administrators to run individual commands, or entire shell scripts to automate any post installation tasks that may be required. RedHat provides a GUI-based tool to assist one in building a kickstart configuration , however in practice it is very easy to customize the configuration file by hand. The ability to run shell commands via the post-configuration phase is simple yet extremely powerful.

II. AutoYaST: is another automated installation utility, similar to RedHat’s Kickstart utility, but used primarily with SUSE-based systems. Novell provides a YaST2 module for autoYaST, which is a GUI tool that can be used to create an autoYaST configuration file – also similar to RedHat’s system-config-kickstart utility. Many of the same rules and procedures used with the kickstart utility also apply to autoYaST. Administrators that need to deploy SUSE clients or servers can create any number of autoYaST configuration files to fit a particular system profile. These can then be used to automate the installation of a SUSE system over a network. When combined with PXE/DHCP setup administrators can deploy and start a SUSE install without using any physical media (i.e. an installation CD/DVD).

III. Bare-Metal Provisioning: Automated deployment tools such as Kickstart and Autoyast support system provisioning via HTTP, (T)FTP and NFS. Completely automated installations can also be configured using the Pre-Execution Environment (PXE), DHCP, tftp and kickstart or autoyast. By automating the boot process, it no longer becomes necessary to manually initialize the installation process via a CDROM or other bootable medium.

Many cluster deployment solutions utilize these very same technologies to deploy large numbers of nodes in a very short time. ROCKS, for example, automates the booting and (re)deployment cluster nodes on the network using PXE, DHCP and kickstart, a process that can sometimes require less than 10-minutes for a single node. Proprietary tools such as CSM, IBM’s cluster management software, also utilizes PXE, DHCP and kickstart or autoYaST to (re)provision cluster nodes as needed.

Configuring a PXE Server for Automated Installations: There are two common utilities one may use to configure a PXE server on Linux. Testing for this scenario was done using a RedHat based system. Therefore some aspects of the following descriptions, such as locations of configuration files, will be RedHat-centric. The general necessity of the configuration and the components, however, are not distribution specific.

pxeos – This utility can be used to configure operating system descriptions within the PXE boot files. Operating system descriptions include the OS name, the protocol used to obtain the OS files (HTTP, FTP, NFS) and the full URI and path to the installation files.
system-config-netboot – The system-config-netboot utility is a graphical application that can perform many of the same tasks as the pxeos utility.

Configuring DHCP: The DHCP daemon can be configured via the /etc/dhcpd.conf file. Those options that are specific to allowing PXE boot clients are listed below.

allow bootp;
class "pxeclients" {
match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
filename "linux-install/pxelinux.0";
}

Configuring TFTP: The TFTP daemon must first be enabled via xinetd super daemon. The following configuration is added to /etc/xinetd.conf, or sometimes /etc/xinetd.d/tftp, depending on the distribution:

service tftp
{
        disable                 = no
        port                   = 69
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}
TFTP Configuration in xinetd.conf
The directory /tftpboot/linux-install is the default used by the system-config-netboot configuration tool. The directory contains kernels and the necessary configuration files required to boot a system and begin a kickstart installation. The configuration file /tftpboot/linux-install/pxelinux.cfg/pxeos.xml contains specific definitions about which network install profiles are available to PXE boot clients.

That’s it for the Provisioning and Deployment section. As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability – Part 2: Scope, Methodology and Ontology

kishi — Fri, 06 Apr 2007 18:14:00 GMT

In my last blog called “Why Manageability Matters” I talked about why we chose to work on “Systems Manageability” as a whole and get a grassroots understanding of it within the context of Linux and Open Source space. In this blog, I’m going to address the Methodology and Ontology of the Systems Manageability project. This will shed immediate light on how we approach, design and implement projects in the OSSL. Let’s start with the main goals and purpose behind the project.

I. Systems Manageability Project Goals:

Survey the scope, depth and breadth of Linux manageability solutions, primarily utilizing open source software
Use the knowledge from the research to develop an understanding of Manageability landscape in the Linux and Open Source
Build and simulate common Linux management solutions in the OSSL (SMORG scale – small and medium sized organizations )
Understand if a cohesive Linux manageability stack exists today--Understand manageability focus from a Linux Sysadmin’s perspective (SMORG scale)

Once we defined what we needed to get into, yet another realization dawned on us, which was the sheer size and volume of the data and information that was staring at us in the face. Let’s just say “overwhelmed” was a mild word compared to what we were looking at. My colleague, Steve Zarkos and I immediately realized that it was time to trim the scope of what we were doing and limit ourselves to what’s achievable in three months and with two people J. This called for drawing up what we considered to be “out-of-scope”, which was:

II. Out-of-scope:

Enterprise-level Management Stacks such as HP Openview, Tivoli etc.
Security and user management, which is a very large area in itself, and would include policies, permissions, authentication, identity management etc.
Asset management frameworks that allow for inventory management, asset tracking etc.
Disaster recovery and planning which would include backup technologies and data protection management

III. Systems Manageability Project Methodology:

The approach taken for the project was simple and scientific. The project was divided into three stages:

Stage 1 – Discovery and Definition: We would explore, uncover and scour the Linux Manageability landscape for toolsets, apps and frameworks prevalent in the SMORG scale workspace today
Stage 2 – Lab Implementation: We would implement the results of our research which would be actual Linux and Open Source products in the Lab
Stage 3 – Analysis and Conclusion: After discovering, defining and implementing Linux and Open Source based manageability toolsets, apps and frameworks in the lab, we would spend time understanding their form, function and experience to draw conclusions

IV. Systems Manageability Project Ontology (classification):

The hardest and most challenging aspect of the project was to develop some sort ontology, characterization or classification of the manageability technologies prevalent in the IT environments today. The diagram below represents the overall "buckets" defined as part of this exercise. Each section of the diagram is broken down to provide a detailed breakdown of each of these Systems Manageability classifications represented:

In the next blog to follow, I will break down the first segment of ontology i.e. “Provisioning and Deployment” and discuss our research with all of you. Meanwhile, we always look forward to hearing from you, our audience and urge you for any feedback you may have about the topic. Thank You for tuning into Port25.

Cheers!

What Lies Beneath: Setting up underlying HPC tools

kishi — Thu, 21 Dec 2006 22:34:00 GMT

This blog continues what I started writing about w/ Thinking About HPC Infrastructure and what Frank wrote in about in Overloading Clusters.

After reading thru the previous blogs on HPC, someone might ask “What are some of the core components of HPC ?”. After all, once you’ve seen the outside of a Maserati or a Pantera DeTomaso, you’re not going to be satisfied just by ogling at it. Even after a test drive, the engineer in you will want to pop the hood and see what’s inside. Taking a similar approach let’s uncover some underlying HPC technologies by looking at any basic HPC setup. Once all the provisioning has been completed, the HPC system will be physically deployed with an OS and relevant drivers, utilities etc. Yet, before the actual HPC application can get installed across, there remains a critical step in the process, i.e. configuration of cluster and file system along with any tools and interfaces such as MPI (Message Passing Interface) etc. After peeling through the HPC application layer, its worthwhile to do a “deep-dive” into what really runs the HPC clusters. A broad category of these tools are:

Cluster Management tools e.g. CSM
Job Scheduling tools e.g. SCALI, Maui
Resource Management tools e.g. Torque

If you’re trying to understand the “WHY” behind the existence of these tools and their importance, take a look at Cluster Management for example. Cluster configuration, installation and management can be difficult and requires intimate familiarity with the HPC hardware, OS, underlying architecture etc. Without specific tools that attend to and manage specific underlying HPC sub-components, HPC just won’t be what it is. So, it is worthwhile to understand the unique installation experience of the tools, such as the ones listed above to understand the complexity of HPC systems. Ready – let’s dive in to the installation and function of these tools:

1. SCALI: The SCALI management and MPI software packages provide deployment, monitoring and job scheduling services for a cluster. After you deploy this software, you will be able see all the compute nodes that may have been preconfigured or are configured on your system. Scali will enable you to monitor the systems and run jobs using the SCALI graphical interface. In order to license the SCALI software, you must utilize the scainstall command to produce a license request file. This file can then be sent to SCALI to receive a permanent key. For those that need some hand-holding through this, luckily SCALI provides very comprehensive documentation on their website. A large portion of the SCALI Manage User’s Guide is dedicated to pre-setup planning and configuration of the cluster and the network. The documentation provides detailed recommendations about how you can set up their Ethernet-based network environment and out-of-band management network. The documentation also provides a general overview about how to install and configure higher performance interconnects, including bonded Ethernet, Infiniband, Myrinet and SCI. The SCALI Manage interface provides simple tools to assist in configuring and testing DET, Infiniband, and Myrinet devices for use with the SCALI MPI implementation. The SCALI MPI software supports multiple Infiniband stacks including Mellanox, Topspin, Voltaire and Infinicon.

2. HP-MPI: HP-MPI is Hewlett-Packard’s Linux-based implementation of the Message Passing Interface (MPI). Many of the utilities distributed with HP-MPI are similar to other common MPI utilities such as MPICH - e.g. mpicc, mpirun, etc. In order to utilize the HP-MPI software, a license is required for each CPU core in the cluster. To obtain a license file you are required to obtain the MAC address from each node (typically eth0) and input that information into a form at licensing.hp.com. The resulting file can then be copied to the compute node. The HP-MPI software is non-functional until licensing files are generated for the nodes

3. CSM (Cluster Systems Management): The CSM software suite is designed to automate the deployment and management of cluster nodes. Nodes can be remotely installed with an operating system as well as the CSM software for later monitoring. The CSM software supports RedHat and Novell on multiple platforms. In order to obtain and install the CSM software one must register with IBM’s website and download the required RPMs. In order to configure CSM, it can remotely install the operating system and/or the CSM software on the compute nodes. Much like Platform ROCKS, CSM makes use of PXE functionality and RedHat’s kickstart or the autoyast software to remotely install the operating system. The CSM software provides multiple methods for defining the nodes that should be deployed and managed:

a. The first method involves creating a hostname mapping (hostmap) file, which is a colon-delimited file that defines a number of attributes of each node
b. The second method also involves manually creating and editing a “node definition” (nodedef) file. This is the method suggested by the documentation for use with small clusters

Proper remote power and remote console capabilities greatly ease the administration and deployment of the compute nodes, however according to the CSM FAQ remote power management is not absolutely required. All the compute nodes must be rebooted (remotely or manually). They are then PXE booted and installed with RHEL4 using the kickstart installation system.

4. Maui and Torque: Both Torque and Maui are free software which must be compiled from the source distribution on the head node. Maui is an open-source job scheduler for compute clusters. It supports a number of task management features not found in other parallel batch processing software including policy-based scheduling and prioritization of tasks. Torque is an open-source resource manager for managing compute nodes and scheduled jobs. It can integrate with Maui to provide additional features for scheduling and managing scheduled tasks. Installation of Torque can be done using the guidance available in the Torque 2.0 Admin Manual .

5. Platform Rocks: Platform Rocks is a cluster deployment software that facilitates the deployment of various software stacks (“rolls”) onto the compute nodes. The software is capable of deploying the base operating system and utilities required for cluster administration, management and scheduling. The software can also manage configuration and updates to ensure consistency throughout the cluster. Platform Rocks is a suite of utilities that are packaged together as separate installable rolls. One of the main goals of the software is to allow for easy installation and integration of third-party rolls and applications. One unique aspect to the Platform Rocks installation approach is that the software installs an operating system on the head node, and also installs all the required rolls at the same time. The software can also automatically set up the subsystem required to install an operating system and other packages on the compute nodes (such as management agents, etc).

That about does it for a quick “deep-dive”. Let me insert a gentle reminder that these are not the only cluster or resource management technologies out there in the HPC space but rather the ones most prevalent. If you have additional tools that you have worked with, we’d like to hear from you and thank you for tuning in to Port 25. HAPPY HOLIDAYS!