Port 25: The Open Source Community at Microsoft : Kishi Malhotra, Software Testing

Systems Manageability Part 4: Systems Configuration

kishi — Fri, 25 May 2007 18:18:00 GMT

Background: This is Part 4, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the second part of the “ontology” which is “Systems Configuration”

Level-Set: System Configuration and Management encompasses all tasks related to the configuration of a host in a standardized and (when possible) centralized way. Many projects in this category provide a common configuration interface, either command-line or GUI-based, designed to ease typical administrative tasks. Other projects, specifically Cfengine, provide a higher level policy-based system to provide consistent configuration and state management for a set of systems. Again, in this case there’s lots of different tools out there that can be used but we have focused on the most popular ones such as Webmin, YaST, SSH, VNC and Cfengine. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I. WEBMIN: "Webmin is a web-based interface for system administration for Unix. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on." Webmin is very modular in design, allowing third-party developers to add support for a particular service or task relatively easily. Many of the tasks involve easing or automating system administration tasks, or editing a configuration file using a specific syntax. Webmin is currently supported by OpenCountry, a company that sells Linux management solutions. The OpenCountry website includes information about Webmin, including two variations of the system that they support.

Webmin Plus: Webmin Plus, is a version of Webmin supported by OpenCountry. According to the website, the Webmin Plus version has been tested by the company and includes new features, specifically support for the back-up and restore application called Bacula (http://www.bacula.org/). Webmin Plus is still freely distributed under a “Mozilla-type” license, and is downloadable via Sourceforge.
Webmin Pro: Last I checked, Webmin Pro had not yet been released, but the following is an excerpt about the product from the OpenCountry website: “...a commercially supported comprehensive product for the entire data center enabling centralized systems administration of mixed distributions, Linux and Windows!”

II. YAST: YaST (Yet another Setup Tool) is an OS installation and configuration utility used primarily in SUSE-based systems. YaST typically serves as the primary control panel interface in, and can be used for a number of configuration tasks – such as adding and removing software, patch management, user management, device configuration and for configuring individual services and daemons. Other common administration tasks such as obtaining system information and reading server logs is also possible via the YaST interface. All of the aforementioned YaST features are implemented as modules, each of which provide a specific functionality or perform certain tasks. These tasks typically involve editing one or more text configuration files on the system in a specific format to configure a specific service or daemon. On other Linux or UNIX-like systems, these tasks are typically performed manually via the command-line.

The YaST utility is very modular in its design, allowing Novell or other third-party providers to add modules into the YaST interface to configure a particular device or service. Many of these modules work independently of each other, and as such are often packaged as individual RPM packages that may be added or removed depending on the software and devices that are installed on a system. YaST modules are written using a scripting language specific to YaST called YCP. Other scripts, such as Perl or shell scripts can also be utilized via a YaST module to perform a particular task. A CIM module for YaST is also distributed with SLES10, which provides a client interface for CIMOM (Common Information Object Manager) to other YaST2 modules. It seems the most common administration task for which YaST is used involves setting up individual package repositories (discussed further in the Patch Management and Maintenance), adding or removing software packages and configuring or initiating online updates. YaST is capable of searching for and locating software on remote repositories, retrieving the software packages, resolving package dependencies, checking the cryptographic signature of the package (if available) and then installing the software on the system. Multiple repositories can be configured. Repositories can be located on a hard disk or CD/DVD, or on a remote system obtainable via HTTP(s), FTP, NFS or CIFS. Once a repository is configured it can then be indexed for later searching. The software search functionality is very powerful, allowing one to search for appropriate software packages using many of the attributes available in the RPM package header – such as the description or contents of the package. Besides software management, the quality and completeness of many YaST modules varies. Many modules (such as the log viewing modules) offer minimal functionality, and only work well enough to provide a few basic configuration options. Complex server configurations will therefore still require one to edit text-based configuration files by hand, or use another configuration engine for the task, such as Webmin. However, many other common tasks, such as configuring display settings or a printer, can be done entirely via YaST.

III. SSH/SCP/SFTP: SSH (Secure SHell) is likely the most widely used remote administration tool for Linux and UNIX-based systems. The typical SSH toolset includes the SSH client and server, as well as the SCP and SFTP client applications for copying files, both of which simply utilize the ssh binary on the backend. The following excerpt is from the OpenSSH project home page: "OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions. Since most any task can be performed via the command-line, the OpenSSH utilities are likely the most critical component for a Linux administrator to have available. The remote copy and command execution options allow one to build, deploy and run a script on a number of machines relatively quickly and securely. OpenSSH is typically installed by default in most Linux-based distributions, although in some distributions the server may by default be disabled or blocked by the firewall. "Most common uses of SSH are:

Remote Command Execution: One of the most common uses of the ssh utility is to run commands in remote machines
Authentication: In order to log into, copy files or run commands on a remote host via ssh, one must first authenticate to the remote machine. Local password authentication may be used without any particular configuration on the server. Linux systems that support PAM (Pluggable Authentication Modules) may also use password authentication to authenticate using a number of mechanisms, including NIS, LDAP, Kerberos, or AD (Active Directory).Rather than requiring a user to manually type in a password, a more common authentication option is to use public key authentication. A user’s public key must be inserted into a file on the remote host called authorized_keys before they are allowed to authenticate to that server. This can often be done via the post installation procedures built into kickstart or autoyast.
Tunneling other Applications: Another common application is to use SSH for tunneling other protocols. This is often used to tunnel protocols that are not typically encrypted such as NFS or X11. By using the –Y or –X switches with the ssh client application, one can “turn on” X11 tunneling, allowing graphical applications run on the remote machine to display locally on the administrators workstation. For example, if one were to SSH into a SUSE-based system and run “yast2” via the command line, the YaST display would be tunneled via SSH and displayed on the local system – even though the actual application is running on the remote system.
Cluster Management: When managing a number of Linux servers, such as a HPC cluster, it is typical to have a large number of systems with identical configurations. To ease administration of these systems there have been a number of SSH-based utilities that allow one to run commands on multiple systems, or copy a file to multiple systems, in parallel. The following utilities are all licensed under the GNU GPL.

ClusterSSH: ClusterSSH allows an administrator run to open a remote SSH session to a number of systems at once via a single terminal window, and run commands or alter configuration files on all the systems simultaneously.
PCP: PCP is a tool designed to copy files in parallel to multiple nodes in a cluster or server farm.
GEXEC: GEXEC is a tool that is somewhat complimentary to PCP that provides a parallel remote command execution system for large clusters. The system includes a client and server, as well as a library that allows integration into third-party applications.
PSSH: PSSH is distributed as a suite of utilities that perform many of same functions as ClusterSSH, PCP and GEXEC.
pssh – Parallel SSH, similar to ClusterSSH or GEXEC.
pscp – Parallel SCP, allows one to copy files to multiples nodes in manner a similar to PCP.
prsync – The prsync utility automates running rsync on multiple nodes in parallel, essentially another method of copying files or entire directory trees to a number of remote nodes.
pnuke – The pnuke command can be used to kill a number of processes running on multiple nodes.
pslurp – The pslurp utility is similar to the pscp utility, except that it is designed to copy files from a set of hosts. This allows one to copy, for example, a log file that exists on multiple nodes, to a local directory tree.

IV. Cfengine: “Cfengine, or the configuration engine is an autonomous agent and a middle to high level policy language and agent for building expert systems to administrate and configure large computer networks. Cfengine is designed to be a part of a computer immune system. It is ideal for cluster management and has been adopted for use all over the world in small and huge organizations alike.” Cfengine consists of a userspace application called cfagent and a host of other utilities that reads and parses a series of text configuration files and performs tasks on the host system based on the configuration. The configuration syntax of Cfengine is actually a high-level policy language that allows cfagent to test the system’s configuration and perform corrective actions based on those tests. For example, cfagent may test to assure that a certain line of text exists within a configuration file, and if not it will add the text and restart the associated service. The cfagent utility is typically run on an hourly (or so) basis via cron, a task-scheduling application. This assures that mis-configurations will be found and corrected within a reasonable time frame.

The policy simply tests to make sure an entry for user root exists within the /etc/shadow file, and also checks to make sure the password matches. This assures that all systems have the same password for the root user. The configuration of Cfengine can become very complex, which would likely not surprise those who have had experience with the tool. The structure of the policy language eases this dilemma a bit, as platform definitions can be made and inherited by other blocks to help determine the appropriate action to take. The configuration is essentially a high-level policy language, and thus the various tests must be built and scripted manually. The toolset is, however, enormously powerful when implemented correctly. But as with many open-source technologies, the learning curve can be quite steep, and one must study the complexities of the tool before it can be competently used in a production environment. A version of Cfengine has been ported to the Windows platform to run under Cygwin.

editfiles:
                    # We have different passwords for lab systems and workstations.
       linux.shadowpasswords.md5passwords.(!workstations)::
{ /etc/shadow
    SetLine "root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::"
    AppendIfNoLineMatching '^root:.*'
    LocateLineMatching '^root:.*'
    ReplaceLineWith “root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::”
}
       linux.shadowpasswords.md5passwords.workstations::
{ /etc/shadow
    SetLine “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
    AppendIfNoLineMatching ‘^root:.*’
    LocateLineMatching ‘^root:.*’
    ReplaceLineWith “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
}
Example Cfengine policy to check the password for the root user.

The following example Cfengine policy checks for the existence and the contents of the /etc/cron.d/yast2-online-update file for SUSE systems. If necessary it creates the file, and writes a cron entry into the file to schedule a daily check for updates and patches. Upon completion, it then runs the command “/etc/init.d/cron restart” as defined in the suse.restartcrond definition.

editfiles:
        suse::
                { /etc/cron.d/yast2-online-update
                  DefineClasses "restartcrond"
                  Umask 077
                  AutoCreate
                  BeginGroupIfNoLineMatching "^.*[\s\t]+root[\s\t]+online_update"
                      AppendIfNoSuchLine "30 3 * * * root online_update"
                  EndGroup
                }
shellcommands:
        suse.restartcrond::
                "/etc/init.d/cron restart"

Example Cfengine policy to assure that SUSE systems check for updates daily.

And that does it for the “Systems Configuration” section As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability - Part 1: Why Manageability Matters

kishi — Wed, 21 Mar 2007 20:09:00 GMT

Impetus: This is the 19th year I have spent in the Information Technology business, out of which more than 15 were spent designing and implementing IT environments of various scopes, platforms and sizes. Among several similarities and differences between each implementation, a few constants always emerged, and my favorite of all: Systems Manageability. This question always got asked during every high-profile design review meeting. You know the meeting I’m talking about, the one with the CIO and the IT Director are sitting across the table and asking how we’re planning on managing the environment. The main concern you could see in everyone’s expression was “what sort of manageability needs to be built around for operations and support ?”. So why does manageability matter ? Let’s start with what people imagine, when they think of Systems Manageability. It means different things to people in different roles: the Infrastructure folks imagine uptime and redundancy, the Developers imagine reliability, the Business Managers imagine efficiency. But the answer is so overwhelmingly obvious. Had it not been for systems management toolsets, apps and frameworks, ITPro’s and Admins would be worried sick about everything from uptime to reliability to scalability. So yes, manageability matters, a lot, because it’s the knowledge that “all’s well and running smoothly” that matters to everyone from a CIO to a Developer to a IT Admin.

Importance: Now that we have established the impetus behind why Systems Manageability matters, we should now address the importance tied to it, such as:

Increasingly complex and heterogeneous environments need increased attention: Pick any environment of your choice, whether it’s where you work or the one you hear about the most. It comes with its own share of challenges and oddities. What differentiates one environment from the other is the manner in which it is “run”. The more complex you hardware, software or network stack, the greater is the manageability tied to it. This means that with every component you add to your existing system, you’re only increasing the complexity even further. So is that a bad thing – NO. Because it’s the environment that should be designed with “adaptation” in mind and not the other way round. Thus, having a complex environment simply translates to keeping an eye on more things. Systems Manageability plays a key role in this scenario
Infrastructure sits at the “core” of IT: Here’s an exercise for any of you that may be interested – the next time you see or hear a technology professional being interviewed about a certain “application” or tool that they’ve developed – try to imagine an entire infrastructure that needs to support and run that “tool” or “App”. The exercise will make you think about what’s going on in the mind of an IT Admin who is responsible for running your environment implementation and why Infrastructure is a big deal. Systems Manageability plays a key role in this scenario
End-user productivity has a directly proportional relationship to Performance Tuning and Optimization: Growth comes in spurts and bursts and never an even pace as we all know. This means, various pieces of your hardware, software, toolsets are implemented in various growths and phases. Simply put, growth does not follow a pre-chartered course and timeline. This makes ongoing performance tuning and optimization a necessity. And it has its benefits - it allows you to see the app or toolkit from the eyes of the people who made it. It also gives you the “know how” to make the specific changes in thresholds and values that could mean a difference of night and day, in terms of efficiency. Systems Manageability plays a key role in this scenario
Striking a balance between Manageability and Flexibility is tough: Identity Management and Security issues are now a mainstream topic of discussion and more and more attention is being paid to system security, access and authentication framework. The single reason for putting these in place to ensure that “you are who you say you are”. On the flip side, overly managed and restrictive environments can limit the flexibility of what an end-user can do. That’s why it is always tough to strike a balance between a system that is secure / well managed and one which seems more “seamless”. Systems Manageability plays a key role in this scenario

In the next seven blogs or so to follow, you will get a detailed breakdown on the Systems Manageability project that we have completed in the lab. We will be covering the Project Methodology and Project Ontology in my next blog. As always, send us your comments and feedback and THANK YOU for tuning into Port25.