Port 25: The Open Source Community at Microsoft : Management

Part 1: Lessons I Learned as a Project Manager Converting to Agile

saraford — Mon, 19 Oct 2009 19:16:00 GMT

Before I became the Program Manager for CodePlex.com, Microsoft's open source project hosting site, I worked on the Visual Studio team on four different product cycles. Since Visual Studio uses traditional Microsoft product lifecycle releases, I had to learn about Agile development alongside learning about open source development when I joined the CodePlex.com team. Making the switch from releasing every three years to every three weeks didn't happen overnight!

One of the things I discovered is that Agile is used a lot in open source communities. In fact, many of the people who I've worked with personally on learning Agile have strong roots in OSS. Also, the fact that Microsoft is starting to adopt Agile philosophies shows how the company is changing, becoming more transparent, finding more ways to connect with the community, and embracing other schools of thought. And this is why I am here, to be on the inside to push for these cultural changes within Microsoft.

Although Agile is the single greatest thing a team could do to significantly improve the user experience and quality of their website, I believe it can be quite challenging for anyone not in a developer role to get accustomed to. I hope that by sharing my experiences, I can help others in non-developer disciplines on an Agile team.

In this series of blog posts, you'll discover how I learned to program manage an Agile team after six years of waterfall (the traditional method of software development).

Three Major Takeaways

If I could go back in time, here are the three things I would tell myself about Agile.

1. Design and plan for the very next step.

It's not about reaching the moon, but getting out of your front door. Sure, you can design the perfect feature, but if it is going to take you six months to get there, it is useless to an Agile team. It's about designing the journey towards the perfect feature that matters.

2. Break down work into the smallest possible functional sets.

Adding work is fun and rewarding, but removing incomplete work due to a lack of development time is painful and risky. But, you can't deploy a half-written feature either. First, break down the work into the smallest pieces. Then, put together the smallest functional sets that have to be deployed together for the feature to make sense. Your development team will tell you how many sets they can do per release.

3. Design and plan only 80% of the way.

Not designing the full 100% is a true blessing in disguise. Since you have another release right around the corner, you have the time to collect user feedback and incorporate it into the next design. Not only does this solve the remaining 20% (getting you closer to the perfect design with less cost), but also allows your customers to be virtual members of your team.

The Program Manager Release Cycle

To begin, here's an Agile release cycle from the point of view of the Program Manager. For simplicity, this illustration only depicts a single release cycle, without any overlap of previous or future cycles.

Ratings and Reviews: An Example

To further illustrate, consider the ratings and reviews feature on CodePlex.com as an example. Users can rate a release and write a review for projects on CodePlex.

One quick aside about ratings and reviews: CodePlex users rate an individual release instead the entire project. For example, consider Stephen King as an author. What does it mean for me to rate Stephen King 4 out of 5 stars? I find some of his books to be awful, like the Tommyknockers. I want those 7 hours of my life back. 1 out of 5 stars. Yet, for me, some of his other books are incredible, like The Dark Tower. 5 out of 5 stars. Hence, we allow users to rate an individual release to provide more relevant information to potential downloaders of the project.

Let's explore the ratings and reviews feature step-by-step in the Program Manager shoes.

1. Design Phase Part 1: Limit the scope to designing the minimum to make the feature useful and meaningful. For ratings and reviews, the feature must have the following:

a. User can rate a release. User can view the rating.

b. User can write a review. User can read the review.

c. User can sort by highest rated releases in project directory.

2. Design Phase Part 2: Bucket into smallest deployable functional sets. For me, personally, I use sticky notes to illustrate the "must have" pieces for each set.

a. Sticky Note #1: Rate releases / View rating

b. Sticky Note #2: Write a review / read review

c. Sticky Note #2: Sort by highest rated releases

3. Iteration Planning Meeting: To start the development cycle, meet with the development team to discuss costing.

a. In the case of ratings and reviews, my devs said they could do Sticky Notes #1 and #2, but the project directory sorting feature would have to wait for the next release.

4. Dev Cycle: Because the designs are closer to 1-page specifications rather than fully-documented implementations, questions will come up from the dev team. This is where you, the Program Manager, will:

a. Answer any questions about the missing 20% of the specifications / wireframes

b. "Course correct" (more on that later)

c. Add more feature work if time allows

5. Deployment: The release goes live. Now you can collect user feedback and incorporate it into the next development cycle.

a. The very first tweet I saw regarding the ratings and reviews feature was "Sara, is there a way to sort by highest rated?" Here, I was able to ask the user questions about how this feature should work to confirm our designs. Most of the time we don't inform users what's coming up next (we like surprises.) But in this case, it was pretty obvious.

Here's the visual representation of the ratings and reviews feature in the Program Management agile release cycle:

Conclusion of Part 1: The Agile Program Management Cycle

This concludes my first post on Program Managing an Agile team. Since I could go on endlessly writing about topics I'm passionate about (and those who know me will confirm this is not an exaggeration), I'm going to pause here to conclude this initial train of thought.

If you like what you see, let me know! And if you don't like what you see, please don't hesitate to let me know. Seriously, I love discussing my Love / Hate Relationship with Agile development, as depicted on my personal blog.

Up next: The concept of Course Correction.

GroundWork Open Source Joins Microsoft's System Center Alliance

Peter Galli — Thu, 24 Sep 2009 13:31:00 GMT

GroundWork Open Source, Inc., a commercial open source company that produces network management software, last week announced the availability of the GroundWork Connector for Microsoft System Center Operations Manager.

The company has also become a member of the System Center Alliance.

GroundWork Monitor, which already has more than 1,500 plugins available, integrates with System Center Operations Manager and extends monitoring and management coverage to non-Windows systems, applications and devices.

The new GroundWork Connector pulls information from System Center Operations Manager and displays it within GroundWork Monitor Enterprise, giving customers a deeper visibility into the availability and performance of all critical infrastructures on a single console. The connector gives insight into applications, databases, virtual machines and network devices that may be running on Linux, Unix, Windows or embedded operating systems.

I talked to David Dennis, the company's senior director of marketing and business development this week about the move, which he feels is a great follow-up to the release of the System Center Cross Platform extensions earlier this year.

That release broke new ground for using System Center in heterogeneous environments. "In the field, we have more and more users asking about how they can integrate the management of Windows with open source tools for managing network infrastructure, Unix, Linux, and the applications that run on top of them," he told me.

The dialog also no longer seems to be about choice between Windows or Open Source but rather "I want both - now how do I make them work together," he says. Even though GroundWork Open Source is an open source company, about half of the operating systems managed by GroundWork Monitor are running Windows.

"The combination of System Center Operations Manager and GroundWork Monitor provides a full-featured alternative to traditional systems management frameworks, but with greater openness and at a much lower price point," Dennis says.

Tuxera Signs File System IP Agreement with Microsoft

Peter Galli — Wed, 26 Aug 2009 12:21:00 GMT

Tuxera, a Finnish company that develops NTFS drivers, has entered into a file system IP agreement with Microsoft in the area of data portability for devices

The agreement gives Tuxera access to the exFAT specifications and source code, as well as testing and verification tools, and the company will develop exFAT drivers for host device manufacturers. The company will also join the Interoperability Vendor Alliance.

While this is also the first-ever file system IP agreement that Microsoft has signed with an independent software vendor, Tuxera joins a growing number of companies entering into the exFAT Program.

The Extended File Allocation Table (exFAT) is a new file system that is better adapted to the growing needs of mobile personal storage. It allows a common file system to be used across all platform and devices that implement exFAT, which allows for data portability.

It also handles not only large files, such as those used for media storage, but also enables seamless interoperability between desktop PCs and devices like portable media devices, cameras or even kitchen devices so that files can easily be copied between desktop and device.

"Adding exFAT into our product portfolio is the logical step to help our customers to solve their interoperable file systems needs. Microsoft supports exFAT as the first-choice for many Windows interoperability needs including flash memories in consumer devices. exFAT is for example an integral part of SDXC formats, and the driver is good for flash drives, including devices that use SDXC cards," says Tuxera CTO Szabolcs Szakacsits

For his part, Mikko Välimäki, the company's CEO, believes that his company's ongoing collaboration with Microsoft ensures seamless file system interoperability into the future.

"We were the first to offer exFAT drivers based on our first-hand access to specifications and source code.We are also licensing exFAT drivers to device manufacturers, we are joining Microsoft's partner program, and have also announced a new product: Tuxera exFAT for Embedded Systems, which will be first available for Linux, but we can also port the product to proprietary platforms," he says.

The IP agreement the company has signed with Microsoft is consistent with Tuxera's business model, which is to integrate and license commercial, high-performance versions of its file system drivers to customers, as well as to provide support, maintenance, and product updates, Välimäki says.

Project Quant

Peter Galli — Wed, 15 Apr 2009 22:54:00 GMT

I noticed today that my colleague Jeff Jones in the security group is launching a metric project that appears to be leveraging some of the good bits of open techniques.

I touched base with him briefly and he gave me a little more information about Project Quant, which is being undertaken along with Securosis, an independent security research firm.

Project Quant will be working on the metrics of patch management and is as much an experiment of a new research process as it is one of security metrics, said Securosis founder Rich Mogull in a blog post.

"For this project Jeff wanted to be involved, but also asked for an open, unbiased model that will be useful to community-at-large (in other words, he didn't ask for a sales tool). Rather than us developing something back at the metrics lab, Jeff asked us to lead an open community project with as much involvement from the different corners of the industry as possible," Mogull said.

While he also acknowledged that it is risky for Securosis to allow direct involvement of the sponsor, the company is hoping that the process works the way it thinks it will and which also happens to match Microsoft's project goals.

So, this is what's expected to happen: a project landing site has been set up at Securosis that will contain all material and research as it is developed; every piece of research will be posted for public comment and no comments will be filtered unless they are spam, totally off topic, or personal insults.

All significant contributors will also be acknowledged in the final report, although there will be no financial compensation for contributors and the project itself will retain ownership rights. All material will also be released under a Creative Commons license, with spreadsheets released in both Excel and open formats.

"In short, we are developing all research out in the open, soliciting community involvement at every stage, making all the materials public, acknowledging contributors, and eventually releasing the final results for free and public use. The end goal of the project is to deliver a metrics model for patch management response to help organizations assess their costs, optimize their process, and achieve their business goals. Let us know what you think, even if you think we're just full of it," Mogull said.

For his part, Jones told me that while he has been zealous in past reports about using repeatable methodologies, pointing to his source of public data, and outlining his assumptions step-by-step, he would like to take transparency one step further by developing models and methodologies first, in an open and transparent manner, so that everyone can agree on the pros and cons before the methodologies are applied.

"I think being completely open and transparent will help credibility since, similar to open source, everyone can scrutinize every step of the analysis ... creating open models and potentially getting community involvement just seems to be the right process," he says.

I plan to interview him at greater length in the next few weeks, so look for a follow-up blog then.

SMB2: a Complete Redesign of the Main Remote File Protocol for Windows

Peter Galli — Mon, 08 Dec 2008 16:24:00 GMT

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's.

Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time.

Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

In this blog Jose Barreto, a senior technical evangelist in Microsoft's Storage Solutions Division, explains some of the history behind the protocol and outlines important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Introduction

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's. Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time. Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

A History of SMB and CIFS

When it was first introduced to the public, the remote file protocol was called SMB (Server Message Block). SMB was used, for instance, by Microsoft LAN Manager in 1987 and by Windows for Workgroups in 1992. Later, a draft specification was submitted to the IETF under the name Common Internet File System (CIFS). The CIFS specification is a description of the protocol as it was implemented in 1996 as part of Microsoft Windows NT 4.0. A preliminary draft of the IETF CIFS 1.0 specification was published in 1997. Later, extensions were made to address other scenarios like domains, Kerberos, shadow copy, server to server copy and SMB signing. Windows 2000 (released in 2000) included those extensions. At that time, some people went back to calling the protocol SMB once again. CIFS/SMB has also been implemented on Unix, Linux and many other operating systems (either as part of the OS or as a server suite like Samba). A few times, those communities also extended the CIFS/SMB protocol to address their own specific requirements.

One important limitation of SMB was its "chattiness" and lack of concern for network latency. It would take a series of synchronous round trips to accomplish many of the most common tasks. The protocol was not created with WAN or high-latency networks in mind and there was limited use of compounding (combining multiple commands in a single network packet) or pipelining (sending additional commands before the answer to a previous command arrives). This even led to products created to address the specific issues around SMB WAN acceleration. There were also limitations regarding the number of open files, shares and users. Due to the large number of commands and subcommands, the protocol was also difficult to extend, maintain and secure.

Introducing SMB2

The first major redesign of SMB happened with the release of SMB2 by Microsoft. SMB2 was introduced with Windows Vista in 2007 and updated with the release of Windows Server 2008 and Windows Vista SP1 in 2008.

SMB2 brought a number of improvements, including but not limited to:

Reduced complexity, going from over 100 commands and subcommands to just 19 (see details below)
General mechanisms for data pipelining and credit-based flow control (see details below)
Request compounding, which allows multiple SMB requests to be sent as a single network request(see details below)
Larger reads and writes make better use of faster networks, even with high latency
Caching of folder and file properties, where clients keeps local copy of information on folders and files
Durable handles allow an SMB2 connection to transparently reconnect to the server if there is a temporary loss of network connectivity
Message signing improved (HMAC SHA-256 replaces MD5 as hashing algorithm) and configuration/interoperability issues simplified
Improved scalability for file sharing (number of users, shares and open files per server greatly increased)
Protocol works well with Network Address Translation (VC count is gone)
Extension mechanism (for instance, create context or variable offsets)
Support for symbolic links

It is important to highlight that, to ensure interoperability, SMB2 uses the existing SMB1 connection setup mechanisms, and then advertises that it is capable of a new version of the protocol. Because of that, if the opposite end does not support SMB2, SMB1 will be used.

The SMB2 protocol specification was published publicly by Microsoft and you can find the link at the end of this post.

Reduced Complexity

One of the ways to showcase the reduced complexity in SMB2 is to make a comparison to the commands and subcommands in the old version.

Here is the complete list of the 19 opcodes (or commands) used by SMB2 in the message exchanges between the client and the server, grouped in three categories:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP, LOGOFF, TREE_CONNECT, TREE_DISCONNECT)
File, directory and volume access (CANCEL, CHANGE_NOTIFY, CLOSE, CREATE, FLUSH, IOCTL, LOCK, QUERY_DIRECTORY, QUERY_INFO, READ, SET_INFO, WRITE)
Other (ECHO, OPLOCK_BREAK)

When you try to get a similar list for the old SMB, things get a little more complex. I tried to make a list of all commands and subcommands using only the documents linked below and came up with over 100:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP_ANDX, TRANS2_SESSION_SETUP, LOGOFF_ANDX, PROCESS_EXIT, TREE_CONNECT, TREE_CONNECT_ANDX, TREE_DISCONNECT)
File, directory and volume access (CHECK_DIRECTORY, CLOSE, CLOSE_PRINT_FILE, COPY, CREATE, CREATE_DIRECTORY, CREATE_NEW, CREATE_TEMPORARY, DELETE, DELETE_DIRECTORY, FIND_CLOSE, FIND_CLOSE2, FIND_UNIQUE, FLUSH, GET_PRINT_QUEUE, IOCTL, IOCTL_SECONDARY, LOCK_AND_READ, LOCK_BYTE_RANGE, LOCKING_ANDX, MOVE, NT_CANCEL, NT_CREATE_ANDX, NT_RENAME, NT_TRANSACT, NT_TRANSACT_CREATE, NT_TRANSACT_IOCTL, NT_TRANSACT_NOTIFY_CHANGE, NT_TRANSACT_QUERY_QUOTA, NT_TRANSACT_QUERY_SECURITY_DESC, NT_TRANSACT_RENAME, NT_TRANSACT_SECONDARY, NT_TRANSACT_SET_QUOTA, NT_TRANSACT_SET_SECURITY_DESC, OPEN, OPEN_ANDX, OPEN_PRINT_FILE, QUERY_INFORMATION, QUERY_INFORMATION_DISK, QUERY_INFORMATION2, READ, READ_ANDX, READ_BULK, READ_MPX, READ_RAW, RENAME, SEARCH, SEEK, SET_INFORMATION, SET_INFORMATION2, TRANS2_CREATE_DIRECTORY, TRANS2_FIND_FIRST2, TRANS2_FIND_NEXT2, TRANS2_FIND_NOTIFY_FIRST, TRANS2_FIND_NOTIFY_NEXT, TRANS2_FSCTL , TRANS2_GET_DFS_REFERRAL, TRANS2_IOCTL2, TRANS2_OPEN2, TRANS2_QUERY_FILE_INFORMATION, TRANS2_QUERY_FS_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_REPORT_DFS_INCONSISTENCY, TRANS2_SET_FILE_INFORMATION, TRANS2_SET_FS_INFORMATION, TRANS2_SET_PATH_INFORMATION, TRANSACTION, TRANSACTION_SECONDARY, TRANSACTION2, TRANSACTION2_SECONDARY, UNLOCK_BYTE_RANGE, WRITE, WRITE_AND_CLOSE, WRITE_AND_UNLOCK, WRITE_ANDX, WRITE_BULK, WRITE_BULK_DATA, WRITE_COMPLETE, WRITE_MPX, WRITE_MPX_SECONDARY, WRITE_PRINT_FILE, WRITE_RAW)
Other (ECHO, TRANS_CALL_NMPIPE, TRANS_MAILSLOT_WRITE, TRANS_PEEK_NMPIPE, TRANS_QUERY_NMPIPE_INFO, TRANS_QUERY_NMPIPE_STATE, TRANS_RAW_READ_NMPIPE, TRANS_RAW_WRITE_NMPIPE, TRANS_READ_NMPIPE, TRANS_SET_NMPIPE_STATE, TRANS_TRANSACT_NMPIPE, TRANS_WAIT_NMPIPE, TRANS_WRITE_NMPIPE)

I make no claim that the list above for SMB is exact or complete, but it does make a point. As an interesting exercise, check the lists above to verify that, while SMB2 has a single WRITE operation, there are 14 distinct WRITE operations in the old protocol.

SMB2 also requires TCP as a transport. SMB2 no longer supports NetBIOS over IPX, NetBIOS over UDP or NetBEUI (as SMB version 1 did).

Pipelining

A key improvement in SMB2 is the way it makes it easy for clients to send a number of outstanding requests to a server. This allows the client to build a pipeline of requests instead of waiting for a response before sending the next request. This is especially relevant when using a high latency network.

SMB2 uses a credit based flow control, which allows the server to control a client's behavior. The server will start with a small number of credits and automatically scale up as needed. With this, the protocol can keep more data "in flight" and better utilize the available bandwidth.

This is key to make a large transfer go from hours (in SMB) to minutes (in SMB2) in a "long and fat pipe" (high bandwidth, high latency network).

For an example of how pipelining in SMB2 can improve performance, check out this blog post.

Compounding

When you look at the command set for the new SMB2 protocol, you notice that they are all simple operations. The old SMB1 protocol had some complex commands and subcommands that combined a set of simple operations as required in specific scenarios.

One of the important changes in SMB2 is the ability to send an arbitrary set of commands in a single request (single network round trip). This is called compounding and it can be use to mimic the old complex operations in SMB1 without the added complexity of a larger command set.

For instance, an old SMB1 RENAME command can be replaced by a single request in SMB2 that combines three commands: CREATE (which can create a new file or open an existing file), SET_INFO and CLOSE. The same can be done for many other complex SMB1 commands and subcommands like LOCK_AND_READ and WRITE_AND_UNLOCK.

This compounding ability in SMB2 is very flexible and the chain of commands can be unrelated (executed separately, potentially in parallel) or related (executed in sequence, with the output of one command available to the next). The responses can also be compounded or sent separately.

This new compounding feature in SMB2 can be used to perform a specific task in less time due to the reduced number of network round trips.

Conclusion

I hope this post has helped you understand some of the important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Reference

Below is a list of important links that document SMB2, SMB and CIFS, including the latest protocol specifications published by Microsoft:

Two Years and Counting....

Peter Galli — Wed, 19 Nov 2008 04:16:00 GMT

It is two years this month since Microsoft and Novell struck their ground-breaking technical collaboration agreement, a move that has effectively ensured greater interoperability between Windows Server and SUSE Linux Enterprise Server.

This technical collaboration has already resulted in a number of milestones, including two new offerings announced today: the availability in the first half of 2009 of an Advanced Management Pack for SUSE Linux Enterprise for Microsoft System Center Operations Manager 2007 R2, and a free beta download of Novell's Moonlight, a rich media application.

Some analysts, vendors and enterprises have said the company that develops effective cross-platform management tools will have an advantage and strategic differentiator over its competitors who do not. Microsoft is already doing that.

The Microsoft Operations Manger 2007 Cross Platform Extensions enable the assessment and management of Windows and Linux servers from a single, unified console, eliminating the costs and complexities of having multiple management consoles. The Advanced Management Pack extends this Linux monitoring capability.

Also, given the current tough economic environment, this solution helps reduce training costs since staff only need to be trained on one management tool for both Windows and Linux environments.

Attendees at the Microsoft TechEd EMEA conference in Barcelona earlier this month got to see a technical preview of the Advanced Management Pack, whose release will coincide with that of Microsoft System Center Operations Manager 2007 R2.

A beta of Novell's Moonlight, an open source implementation of Microsoft Silverlight, will also be released going forward as an open source plug-in for the Firefox web browser. Moonlight brings Linux-based users the same high-definition media capabilities currently available for the Windows and Apple environments.

So, expect to see a lot more solutions in the next year that promote interoperability and help ease customer pain-points across their heterogeneous environments.

ApacheCon and the Stonehenge Proposal

Kamaljit Bath — Mon, 10 Nov 2008 21:24:00 GMT

This is Kamaljit Bath, and I am in the Big Easy to experience my first Apache Conference! I am a Principal Program Manager in the Interoperability Technical Strategy Team at Microsoft. We have been doing a lot of great interoperability work and have done quite a few Open Source projects to build bridging solutions etc., but this is the first time I am attending ApacheCon. This is a learning experience for me.

Microsoft is certainly interested in expanding interoperability between Open Source solutions and Microsoft technologies, and is working with individuals and communities for that purpose. I think this is great because it will enable choice of solutions and create strong partnerships to promote growth for everyone in the industry.

ApacheCon has been quite an experience for me so far. I have seen the energy and high quality decision making. It is amazing how people from many different backgrounds can come together and accomplish so much in so little time.

I have also met some very interesting people and some that I have wanted to meet for a long time. Meeting motivated and driven people is what I like the most about conferences. I have learnt a lot from this experience and I will take back great memories from this trip.

In his keynote today Sam Ramji, the Senior Director for Platform Strategy at Microsoft, gave an update on the many interoperability and Open Source projects that Microsoft is engaged in. I am sure that some of this was news to many of the attendees, but hopefully it gave them an idea of the breadth of work that Microsoft is doing in this area.

Sam covered a lot of things, including our participation in Apache QPID project; the release of the 'Oslo-M' language under the Open Source Promise; participation in the Apache HBase project; and support for the new Stonehenge proposal by WSO2.

Sam also covered many other open source projects that Microsoft has used to build bridging technologies, while my manager, Jean Paoli, has covered these in detail in his blog.

Sam is a well known figure in these avenues and needs no introduction - he has been representing Microsoft at many of these conferences.

But it is also important to have a more grass-root level developer and architecture presence from Microsoft, and we are now moving in that direction. Hopefully, we will see an increased Microsoft presence at such events.

Microsoft is also supportive of the new Apache incubation proposal - Stonehenge - that was proposed by WSO2. It will focus on building a set of sample applications based on approved W3C and OASIS standard protocols with goal of proving interoperability between different implementations on various platforms.

I think these sample applications will provide developers a great starting place for their tasks by providing best practice guidelines and reference implementations on various platforms. They will also help find potential interoperability problems and hopefully develop into a great community to discuss the architecture of multi-tier SOA apps. We look forward to working with WS02 on the scope of this project, and having discussions with the community.

These are exciting times for the software industry and we are seeing the co-existence of commercial and open source software and coming together of various forces to create solutions for the new heterogeneous IT environment.

Onwards, with great faith and hope!

Technical Analysis: Remote Administration of Windows Systems with SSH

jcannon — Wed, 11 Jun 2008 13:22:00 GMT

Abstract: SSH has largely replaced Telnet for remote administration of UNIX and Linux systems, but has not yet been used much on Windows. SSH is generally considered to be more secure than Telnet and the Berkeley remote commands (rlogin, etc). This paper uses SSHWindows, a minimal package of Cygwin and OpenSSH. It is available from http://sshwindows.sourceforge.net. The paper is written such that an average Windows system administrator can get an SSH server up while understanding how to make use of security features.

Download Remote Administration of Windows Systems with SSH (PDF)

Note: This paper represents testing and documentation in a lab environment. User Account Control (UAC) is an essential security component to Windows and Microsoft does not recommend turning off UAC in production environments.

Managing Towards Open

Sam Ramji — Tue, 29 Apr 2008 19:59:00 GMT

I have the privilege of interacting almost every day with technical and business experts who are creating the future of software—including both core engineering teams at Microsoft and thought leaders across a broad spectrum of open source communities. Especially in the last few months, I’ve been able to take more time to articulate where I think this is going – such as writing how open source has influenced Windows Server 2008 and participating in Infoworld’s roundtable on the state of open source.

I think that many people are seeing that the interrelationship between Microsoft and open source is being changed fundamentally (and for mutual benefit).

Today, Bob Muglia and Brad Anderson announced that System Center will have the ability to deliver automated management across heterogeneous IT environments, such as UNIX and Linux. What I see as a best practice for commercial and community engagement with open source technology plays a big part in this.

Specifically, Microsoft will deliver an agent infrastructure and management packs (MPs) for monitoring Linux and UNIX platforms for System Center Operations Manager 2007. Early partners like Xandros and Quest are delivering cross-platform MPs for MySQL and Apache, and Oracle, respectively. Microsoft and Novell are collaborating on the SUSE Linux Enterprise MP.

The agent infrastructure Microsoft is building to interoperate with UNIX and Linux is built leveraging industry standards and open source such as WS-Management and OpenPegasus. Pegasus is an open-source implementation of the DMTF CIM and WBEM standards coded in C++, designed to be portable, and licensed under an MIT license, and work is underway to integrate with the newly DMTF ratified WS-Management standard. Pegasus already ships as part of major Linux and UNIX distros.

It simply makes great technical and business sense to cooperate with the OpenPegasus community to build upon an industry-standards based, cross-platform technology. Just as important, however, is preserving the virtuous cycle of contribution, benefit, and subsequent contribution: Microsoft is joining the OpenPegasus Steering Committee. The agent technology—being built will be contributed back to the community under the Microsoft Public License (MS-PL), an OSI approved open source license.

I greatly appreciate Allen Brown's positive comments (Allen is the President and CEO for The Open Group) and the support and education we’ve received from the sponsors and maintainers of Pegasus. He said:

“We are pleased to have Microsoft join the OpenPegasus Steering Committee and welcome their commitment as a positive step for the global open source development community. Since The Open Group initiated the OpenPegasus project seven years ago, it has been deployed across a wide range of IT platforms worldwide. We look forward to Microsoft’s active participation in the continuing development of the project.”

Today’s announcement and the business and technical decisions made by the System Center team are a great example of the fact that commercial innovation, industry partnerships, and open source participation can all work together to make the whole greater than the sum of its parts. My enthusiasm and excitement—and my applause for the System Center team, partners like Xandros, Quest, and Novell, and the OpenPegasus community—is tempered solely by my conviction this is not the only or last example of the best of Microsoft, partners, and open source growing together. This is a great day – and there are more great days to come.

How open source has influenced Windows Server 2008

Sam Ramji — Wed, 27 Feb 2008 12:00:00 GMT

When I think about what works really well in open source development and technology, the following things stand out:

Modular architectures
You can find these wherever you see participation at scale – and often a rearchitecture to a more modular system precedes expanded participation. Great examples of this are Firefox, OpenOffice, and X11 – from both the historical rearchitecture and the increased participation that resulted. The Apache HTTP server and APR are good examples that have been modular for as long as I can recall.
Programming language agnostic
A given project uses a consistent language, but there are no rules on what languages are in scope or out of scope. Being open to more languages means opportunity to attract more developers – the diversity of PHP/Perl/Python/Java has been a core driver in the success of a number of projects including Linux.
Feedback-driven development
The “power user” as product manager is a powerful shift in how to build and tune software – and this class of users includes developers who are not committing code back, but instead submitting CRs and defects – resulting in a product that better fits its end users.
Built-for-purpose systems
Most frequently seen in applications of Linux, the ability to build a system that has just what is needed to fulfill its role and nothing else (think of highly customizable distributions like Gentoo or BusyBox, as well as fully custom deployments).
Sysadmins who write code
The ability of a skilled system administrator to write the “last mile” code means that they can make a technology work in their particular environment efficiently and often provide good feedback to developers. This is so fundamental to Unix and Linux environments that most sysadmins are competent programmers.
Standards-based communication
Whether the standard is something from the IETF or W3C, or simply the implementation code itself, where these are used projects are more successful (think of Asterisk and IAX2) and attract a larger ecosystem of software around them.

So where did we apply these ideas to the development of Windows Server 2008?

Modular architectures was applied in multiple areas, but the one that stands out most to me is Internet Information Server 7 (IIS7). IIS7 has been rearchitected for flexibility as 40 individual modules, enable more to be written by community developers or delivered as out-of-band releases. This has already enabled performance improvements and independent evolution, and I expect to see further enhancements.
Programming language agnostic is something we’ve delivered on with support for PHP on IIS7 and the enhancements to FastCGI (which can be used by any of the P* languages). We set a goal of having PHP certified on Windows Server 2008, and we’ve achieved that. We’ll continue to improve runtime, security, and manageability support for non-.NET languages and the applications that are built on them, as well as testing the full stacks of PHP-based applications running on Windows Server, IIS, and SQL Server.
Feedback-driven development based on developer and customer trials (RDPs, TAPs, and Betas in our process) led to a range of “feature completion” developments that connected different components – like connecting Windows Firewall with Active Directory central policy, and the end-to-end improvements in SMB 2.0. Features like the RODC (Read-Only Domain Controller) have become more and more solid through experience with early alpha and beta customer deployments, and requests to enforce things like BitLocker encryption of user disks from a central authority have achieved full support.
Built-for-purpose systems such as DNS, DHCP, file and web serving can be created through wizard-driven configuration thanks to Windows Server Core. The goal of having a minimum attack surface and a small hardware footprint, inspired by the capabilities mentioned above, yet achievable by a broad base of admins has been achieved. Additionally, this has created an opportunity for Windows admins to become much more knowledgeable about the low-level structure of the operating system.
Sysadmins who write code are first-class citizens in the PowerShell-driven infrastructure. We’ve increased Windows administrators’ opportunity to master the full surface area of WMI and demonstrate that mastery in reusable, low-level scripts. As we evolve this to support multiple language bindings and bash aliasing, this should become a comfortable home for highly skilled sysadmins.
Standards-based communication such as in CardSpace (with support for X.509, SAML, Kerberos tokens, and more) and the Web Services stack (not only are all 38 Web Services standard under the Open Specification promise, but our implementations have achieved a high level of interop with Apache’s Axis web services stack), and beta support for emerging standards like Xen virtualization represent a small subset of the standards built into Windows Server 2008.

Overall, we’ve learned and continue to learn from open source development principles. These are making their way into the mindset, development practices, and ultimately into the products we bring to market.

I’ve focused here on “what Microsoft has learned from Open Source” – and ironically, I’ve agreed to do a panel at OSBC on 3/25 with Jim Zemlin of the Linux Foundation on “what Open Source can learn from Microsoft”. As all of the different organizations in IT continue to evolve, we’ll learn from each others’ best practices and make increasingly better software. As in science, this incremental improvement will move all of us forward.

Systems Manageability Part 7 - Log Management and Analysis

kishi — Tue, 07 Aug 2007 15:57:00 GMT

Level-Set – Log Management: This section includes open-source technology directed primarily on host-based logging, log file rotation and log file analysis. Many of these tools are very common free and open-source software tools that are distributed and preconfigured with most of the major Linux systems, including major vendors such as RedHat and Novell.

I. Logrotate

Logrotate is a very popular application utilized in a number of Linux systems, including all RedHat and SUSE based systems. The logrotate utility typically runs periodically via cron, a task scheduling application. The utility will read a configuration file (/etc/logrotate.conf), and archive and compress log files according to the configuration. Administrators can configure when log files should be rotated based on age and size, and how long backlogs should be maintained. Older archived log files can then be swapped out and replaced with newer archives.

II. Syslogd and klogd

Typical Linux systems utilize a syslog daemon to capture log messages from userspace applications and write them to text-based log files or send them to a logging host over the network. The syslogd daemon is often accompanied by a klogd application which is designed to capture and log kernel messages.

The behavior of the syslog daemon can be configured via the /etc/syslog.conf configuration file. All messages captured by syslog are categorized by facility and priority. Messages can then be sent to particular log files or logging hosts, or dropped completely based on their facility and priority attributes.

Facilities

Priorities

- auth or security

- authpriv

- cron

- daemon

- kern

- lpr

- mail

- mark

- news

- syslog

- user

- uucp

- local0 through local7

- debug

- info

- notice

- warning or warn

- err or error

- error

- crit

- alert

- emerg or panic

List of syslog facilities and priorities.

III. Syslog-ng

The syslog-ng application aims to be an enhanced drop-in replacement for the traditional syslog daemon. It provides many of the same features of the standard syslog daemon, but includes additional features such as advanced message filtering based on content, remote logging via UDP or TCP, and the ability to write log files to a database such as MySQL or PostgreSQL. More recent SUSE-based systems such as SLES10 have switched to syslog-ng as the default syslog server.

IV. Viewing Logs

Most log files on a Linux system are stored in plain-text, which means they can be viewed and parsed using a number of different command-line tools. Typical utilities such as tail, head, grep, cat, less, more, sed and awk can be used to view and filter log messages via the command line.

There are also a myriad of utilities designed to parse and view log files via a GUI or web browser. Some utilities are even designed to handle specific log formats, such as those generated by Linux’s Netfilter firewall subsystem.

GNOME System Log Viewer

The GNOME system includes a GTK-based system log viewing application that displays system logs via the GUI.

YaST System Log Module

SUSE-based systems using YaST typically include a module called View System Log (called internally as view_anymsg). Similar to the GNOME System Log viewer, the YaST module allows an administrator to view many of the various system logs without using the command-line.

V. Log Analysis

LogWatch

The logwatch utility is designed to parse system logs and located any entries that might indicate security threat or system failure and send an email report to a designated address. Logwatch is distributed with RedHat Enterprise Linux systems. The following is an excerpt from the RPM description:

“LogWatch is a customizable log analysis system. LogWatch parses through your system's logs for a given period of time and creates a report analyzing areas that you specify, in as much detail as you require. LogWatch is easy to use and claims that it will work right out of the package on almost all systems. Note that LogWatch now analyzes Samba logs.”

LogWatch is typically executed periodically via cron, a task scheduling application.

LogCheck

The logcheck utility is a part of the Sentry Tools project that also includes portsentry, a utility designed to detect port scans. Similar to the LogWatch utility, the software is designed to parse system log files, find log entries that may indicate security problems and send an email to a preconfigured address. Also similar to the LogWatch utility, logcheck relies on the standard cron utility to be periodically executed.

That does it for Log Management and Analysis section. We have one last blog to go and certainly hope that you found the information we have captured for you useful. If you’re running any special toolsets or customizable scripts for log management and analysis and would like to share your experience with us, please send us your feedback and as always, THANK YOU for tuning into Port25.

Systems Manageability Part 6: Patch Management and Online Updates

kishi — Fri, 29 Jun 2007 17:44:00 GMT

Level-Set - Patch Management: Patch Management and Maintenance focuses on those solutions available to deploy and install software update on Linux systems, with a primary focus on Novell based Linux systems. This is going to be a very short blog because the only open source tool that I could find, which is used in a widespread manner, is YaST. I know there are tons of solutions out there, some proprietary like RHN and some custom built. YaST was the only common thread we could recognize. A deeper look at YaST and its online update abilities follows:

YAST Online Update Utility

Probably the most common and important modules in YaST are those related to software management (adding and removing software) and patch management. Software and updates for a typical SUSE system are obtained from software repositories, which can be local or remote software inventories from which new software or updates may be obtained. At a deeper level, the SLES9 package management system utilizes the common rpm utility to install, remove, and update packages and manage the package and dependency database. Although this subsystem is similar to RedHat’s, Novell has chosen a very different approach to distributing its patches, choosing to utilize what are called patch RPMs. With many RPM-based distributions, when a package needs to be updated for one reason or another the distributor will modify or patch the original source tree and recompile/repackage the software to produce a new RPM for that particular package. Therefore in these cases the new RPM will simply be an updated version of the original RPM.

Novell has taken a slightly different approach with patching via RPMs. Instead of updating and repacking the entire package, Novell updates the original source tree, recompiles, and then produces a delta (or a diff) between the original binaries in the package and the newly patched/recompiled binaries. The delta is a binary file that contains information about the differences between two binary files. The deltas will then be packaged within an RPM and distributed to clients. The patch RPM can then be manually or automatically installed in the same way a standard RPM would be installed. An advantage to this technique is that patches are often smaller in size – typically anywhere between 5KB and 8MB depending on the size of the package and the changes being applied. This often allows the update process to progress far faster than it would otherwise when using full RPMs – especially for large applications.

Major updates to the stable SLES9 branch are released as an installable “service pack”. Novell typically recommends installing the service pack files via YaST2, from either a CDROM or network location that contains the service pack files. One may also simply utilize the Online Update module of YaST2 to update the system manually or automatically. In this case, the service pack will be distributed as a large number of individual packages, similar to how RedHat distributes major updates (i.e. RHEL4 U4). Aside from a log file, SLES9 does not currently have an email mechanism to inform the administrator when a patch is automatically downloaded and installed (as RedHat does). However, a log file that contains information about each automatic update is maintained in /var/lib/YaST2/you/youlog. This log is generally very easy for an administrator to read and discover when, or if, a patch RPM was downloaded and installed.

There are other ways to find information about installed patches, however. By default, SLES9 archives each patch RPM that is downloaded and installed. Full RPMs will also be archived if they were installed via YaST2 after the original system installation. This functionality can be disabled with YaST2, of course, although it can sometimes be useful to maintain the archive if a patch ever needs to be reinstalled.

1. YaST Software/Update Repositories

Software repositories are typically added manually via the Installation Source module in YaST or can be scanned using SLP (Service Location Protocol). From this module, one may add references to locations from which to receive updates. These references typically take the form of a URI or a directory path. YaST supports the following software repository references:
FTPHTTP(S)
SMB/CIFS
NFS
CD or DVD
Local Directory
Using this methodology it is also quite common for an administrator to install a centralized repository for software and updates. Updates may then be obtained from Novell by a single server, and other servers on the LAN may then pull patches from the central patch server using one of the above protocols.

2. YaST Security

Although software repositories for SLES and SLED distributions are typically operated by Novell, it is quite possible to add third-party repositories to obtain software not offered by Novell, or even different versions of the same software packages. Novell warns against this, however, since adding repositories not controlled by Novell can result in the installation of untested or possibly malicious software, which ultimately could compromise security, but more likely may result in software instability and RPM package conflicts.
All official software and patches obtained by Novell are cryptographically signed, which can be verified with Novell’s public key. The public keys used to verify these signatures are typically obtained via the official SLES/SLED CDs or DVDs, but may also be obtained via Novell’s website. Once these public keys are accepted and imported, any software package or update obtained with an invalid signature will produce a warning and may not install without user intervention.

3. YaST Automatic Updates

Automatic updates can be configured via YaST’s Online Update Setup module, which allows a user to schedule updates to occur at a particular time either daily or weekly. On the backend, this module simply installs a new cron entry, a task scheduling application, which periodically runs another program to check for and install updates pushed out by Novell.
In earlier SUSE-based systems, YOU (YaST Online Update) had been used to automate the installation of updates packages. The cron utility would execute a shell script called /usr/bin/online_update which would automate the patch installation process. Newer versions of SUSE, including SLED10, utilize a similar process but instead of a shell script a utility called rug is used. The rug utility is the command-line interface to the ZENworks management agent that is present on new SUSE systems.

If you are running any open source based tools or applications in your environment to push patches and manage online update scenarios, we would REALLY like to hear what you have to say. As always THANK YOU for tuning into Port25

Systems Manageability Part Five: Monitoring

kishi — Thu, 21 Jun 2007 14:16:00 GMT

Background: This is Part 5, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the third part of the “ontology” which is “Monitoring”

Level-Set – Monitoring: Monitoring and other data collection tools are an essential component of any management strategy. The proper collection and organization of host data allows for manual and sometimes automated reactive corrective measures. This section outlines many of the open source and free software monitoring tools available on the Linux platform. Much of the analysis in this section is focused on the inner workings of these tools as data collection systems, rather than feature comparisons between the various monitoring applications. The WBEM/CIM overview has been placed in this section due to its basis as a data collection and management system, even though its use is not limited the confines of this category.

I.WBEM/CIM: The following section includes an overview of the WBEM initiative and the open-source CIM implementations that exist today. The Distributed Management Task Force (DMTF) classifies WBEM (Web Based Enterprise Management) as the following:

“[WBEM is] a set of management and Internet standard technologies developed to unify the management of distributed computing environments. WBEM provides the ability for the industry to deliver a well-integrated set of standard-based management tools, facilitating the exchange of data across otherwise disparate technologies and platforms.”

Core components and industry standards used in WBEM include CIM, CIM-XML, CIM Query Language, SLP (Service LocationProtocol, for WBEM Discovery) and WBEM URI (Universal Resource Identifier) mapping. The DMTF has also developed a WBEM Management profile template for the purpose of systems manageability. WBEM has been designed to be compatible with all the major existing management protocols, including SNMP, DMI, and CMIP. There are several open source implementations of WBEM including OpenWBEM, WBEM Services, OpenPegasus and SBLIM. These are discussed in more detail below. Additionally, there are both client and server implementations available for the WBEM standard:

WBEM clients include PyWBEM, an open-source WBEM library written in Python, and the Purgos open-source management client for Windows written in C++.
WBEM server implementations include OpenPegasus and OpenWBEM, an open-source client and server written in C++ (Novell has adopted this and added it to SLES9/10).

The following is an explanation of CIM, or Common Information Model, from the DMTF documentation:

“provides a common definition of management information for systems, networks, applications and services, and allows for vendor extensions. CIM’s common definitions enable vendors to exchange semantically rich management information between systems throughout the network. It is a conceptual information model for describing management that is not bound to a particular implementation. This allows for the interchange of management information between management systems and applications. This can be either "agent to manager" or "manager to manager" communications that provides for Distributed System Management.”

CIM includes two components; a specification and a Schema.

CIM Specification: This describes the language, naming, Meta Schema and mapping techniques to other management models such as SNMP MIBs, and DMTF MIFs etc. The Meta Schema is a formal definition of the model. It defines the terms used to express the model and their usage and semantics. The elements of the Meta Schema are Classes, Properties, and Methods. The Meta Schema also supports Indications and Associations as types of Classes and References as types of Properties. Essentially, the CIM specification
CIM Schema: This provides the actual model descriptions. The CIM Schema supplies a set of classes with properties and associations that provide a well-understood conceptual framework within which it is possible to organize the available information about the managed environment. The CIM Schema itself is structured into three distinct layers:

The Core Schema is an information model that captures notions that are applicable to all areas of management.
Common Schemas are information models that capture notions that are common to particular management areas, but independent of a particular technology or implementation. The common areas are systems, devices, networks, applications, metrics, databases, the physical environment, event definition and handling, management of a CIM infrastructure (the Interoperability Model), users and security, policy and trouble ticketing/ knowledge exchange (the Support Model). These models define classes addressing each of the management areas in a vendor-neutral manner.
Extension Schemas represent organizational or vendor-specific extensions of the Common Schema. These schemas can be specific to environments, such as operating systems (for example, UNIX® or Microsoft Windows®). Extension Schema fall into two categories, Technology-Specific areas such UNIX98 or Product-Specific areas that are unique to a particular product such as Windows.

WBEM (CIM) Architecture Diagram

OpenPegasus:

OpenPegasus is an open-source implementation of the DMTF CIM and WBEM standards being driven under the auspices of The Open Group. OpenPegasus is open source and is licensed under the MIT open-source license. The distribution is available via CVS, and as snapshot images in tar, zip, and (self-extracting) exe file formats on the OpenPegasus web site. Based on documentation posted on the site, simply put, Pegasus is an open-source CIM Server for DMTF CIM objects. It is written in C++ and includes the Object manager (CIMOM), a set of defined interfaces, an implementation of the CIM Operations over HTTP operations and their cimxml HTTP encodings, and Interface libraries for both clients and providers. It is maintained to be compliant with the DMTF CIM and WBEM specifications with exceptions noted in the documentation. It is designed to be portable and modular. It is coded in C++ and translates the object concepts of the CIM objects into a programming model. Pegasus is designed to be inherently portable and builds and runs today on most versions of UNIX(R), Linux, and Windows. OpenPegasus includes the following components:

A DMTF compliant CIM Server that processes CIM operations, CIM Indications, and includes class and instance repositories and interfaces for creating CIM Providers and CIM Clients.
Provider interfaces so that providers may be build in multiple languages (i.e. C++, C, Java).
A number of CIM Providers.
A MOF compiler.
A number of CIM Clients to provide usage examples, CIM Server test functions, and administrative functions

OpenWBEM On SLES10:

OpenWBEM is included in SUSE Linux Enterprise Server 9 and 10, allowing any WBEM enabled management console to access configuration information on the system. A CIM schema and a MOF compiler are also included as packages in SLES9 and 10, which can be used to create and import the schema.

## Create the namespace called /root/cimv2

SLES10:/etc/openwbem # owcreatenamespace -n /root/cimv2
Creating namespace (/root/cimv2)
## Import the CIM schema.
SLES10:/etc/openwbem # owmofc /usr/share/mof/cimv2.12/cimv212.mof
[ ... Lots of Output ... ]
Compilation finished. 0 errors occurred.
Compiling and Importing the CIM Schema

## Start the OpenWBEM Daemon.
SLES10:~ # /etc/init.d/owcimomd start
Using common server certificate /etc/ssl/servercerts/servercert.pem
Starting the OpenWBEM CIMOM Daemon done
## Check the status of the OpenWBEM service.
SLES10:~ # /etc/init.d/owcimomd status
Checking for service OpenWBEM CIMOM Daemon running
Starting the OpenWBEM Service on SLES10

II. NAGIOS: Nagios is a system monitoring application designed to monitor remote hosts and applications over a network. The application provides a web-based graphical display that allows one to view the status of nodes and particular applications running on the nodes. The following is an excerpt from the Nagios documentation listing some of Nagios’ feature set: Some of the many features of Nagios include:

Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
Monitoring of host resources (processor load, disk usage, etc.)
Simple plugin design that allows users to easily develop their own service checks
Parallelized service checks
Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
Contact notifications when service or host problems occur and get resolved (via email, pager, or user-defined method)
Ability to define event handlers to be run during service or host events for proactive problem resolution
Automatic log file rotation
Support for implementing redundant monitoring hosts
Optional web interface for viewing current network status, notification and problem history, log file, etc.

Nagios can poll servers and obtain data in a number of different ways. The most straight-forward method is to connect to a remote system directly and test to see if the host is available or if a particular service is running. Data internal to the host, such as free memory or processor usage, however, must be gathered using the Nagios agent, SNMP, another custom script or program or a Nagios plug-in called check_by_ssh - which is a standard plug-in designed to run a command on a remote machine and collect the output. The configuration of Nagios is done entirely via text-based configuration files. Hosts and other resources are defined inblocks, which can also inherit information from other pre-defined blocks, making complex configurations possible and more manageable. There are several third-party applications available that provide a web or other GUI interface to assist one with configuring Nagios, but these were not tested for this project. The following configuration block defines a generic host template called “linux-server”. Many of the configuration values such as “24x7” and “workhours” are actually defined in other configuration blocks within the Nagios configuration. This allows administrators to define custom names to a specific time period, such as “workhours”, and use that definition in other parts of the configuration.

define host {

   name                       linux-server
   use                        generic-host
   check_period               24x7
   max_check_attempts         10
   check_command              check-host-alive
   notification_period        workhours
   notification_interval      120
   notification_options       d,u,r
   contact_groups             admins
   register                   0
}
Nagios Host Definition Template

Individual hosts are defined in configuration blocks. Below is a sample configuration for an individual host called management. Notice the use statement is inheriting other definitions from the previously defined generic template mentioned above called “linux-server”.

define host {
   use            linux-server ;Name of host template to use.
   host_name      management
   alias          Management Server
   address        10.197.173.100
}

Finally, hosts may be organized into logical groups for easier management. The following is a hostgroup that defines a group that includes five hosts.

define hostgroup {
   hostgroup_name test
   alias          Test Servers
   members        localhost,management,www,rhel4-production2,network
}

Nagios is distributed with a wide assortment of plug-ins that can be used to obtain data or check a particular service. Plug-ins are distributed as a separate package which must be installed with both the server and the agent if an agent is to be used. The Nagios plug-ins are simply stand-alone executable programs, each of which can perform a particular task and return a result code for each service or subsystem being tested. Since plug-ins are individual scripts or binary programs, they often will accept different arguments to change their behavior and what information they return. The command usage of each plug-in must be defined individually within the configuration files using the define command syntax. Some plug-ins can accept multiple options which can be customized when writing the configuration for a particular system. The define command definition provides a sort of usage template so that Nagios will know how to run the command later. Luckily for new users, the default sample configuration files already provide accurate definitions for the default plug-ins. Once one is familiar with how commands are defined, however, new commands or custom scripts can also be defined here as well.

NRPE: is the Nagios Remote Plugin Executor that is installed on a remote host. It is designed simply to execute Nagios plugins on behalf of the Nagios server and return the results. The same plugins that are installed on the server must then be installed on the remote host for NRPE to utilize. A new plug-in called check_nrpe is also distributed with the NRPE agent and is used to query the NRPE daemon from the Nagios server. NRPE utilizes a rudimentary access control system to assure that only particular Nagios hosts will be allowed to contact the NRPE client. A configuration directive such as the following within NRPE’s configuration file will only allow communication with a particular host:

allows_hosts=10.197.173.100

It is possible to configure NRPE run nearly any command with any arguments, although one is warned against doing this in the documentation. By default, NRPE will only run specific commands and their arguments as specified in its own configuration file (located on the host itself). Meaning that the Nagios server can tell NRPE to execute only specific commands specified in the remote host’s /etc/nrpe.cfg file, but the server may not pass arbitrary commands or plug-in arguments for the agent to execute. Below is a sample NRPE configuration. The specific commands (plug-ins) and arguments must be specified here. The Nagios server can then request NRPE to execute one or more of these commands and return the results:

command[check_users]=/usr/local/nagios/libexec/check_users –w 5 –c 10
command[check_load]=/usr/local/nagios/libexec/check_load –w 15,10,5 –c 30,25,20
command[check_disk_root]=/usr/local/nagios/libexec/check_disk –w 20 –c 10 –p /dev/sda1
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs –w 5 –c 10 –s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs –w 150 –c 200

By default NRPE utilizes SSL communications between itself and the Nagios server. The SSL parameters are generated at compile time and stored in the C header file called dh.h within the NRPE source tree. This header file is then used to compile the NRPE daemon and the check_nrpe plugin. This means that both the NRPE daemon and the check_nrpe plugin must be compiled using the same parameters (typically from the same source tree) if one wishes to utilize SSL communications.

III. Hyperic: Hyperic HQ is a Java-based monitoring application consisting of a central monitoring server and one or more remote agents to report node status information to the server. Hyperic HQ is supported on a wide array of platforms, including Linux, Solaris, Windows, HP-UX, AIX, Mac OS X and FreeBSD. Hyperic distributes two versions of its software;

An open source version licensed under the GNU GPL
and a commercial version called HQ Enterprise which includes additional components and functionality.

HQ Open Source and HQ Enterprise Feature Set Comparison
Note: As of HQ 3.0 thefeature-set distribution between the Open-Source and Enterprise versions has changed. Please see http://www.hyperic.com/products/hq_for_ent.html for more details.

Hyperic Installation and Configuration: Hyperic HQ aims to be quick to install and relatively easy to configure. The installation is performed via the command-line, and will prompt the administrator for all the information (administrator password, database information, etc) it will need to successfully run. Upgrading can also be done relatively easily by simply running the installer with the –upgrade option. Hyperic HQ provides a web interface to deliver monitoring alerts and status information to the end-user. However, unlike other monitoring applications the web-interface is also used as the primary configuration interface for the application. All node and agent details, metric options and alerts may be configured directly over the web interface. The monitoring agent is installed in a similar manner as the server. Because all agent configuration is done via the web interface on the server, the only information the agent installation script needs is login information for the server, the preferred path on the node to which it should install the agent files and various other pieces of information such as the port numbers on which the server and agent will be running. Once the agent successfully registers itself with the server, the administrator can then log in to the web interface and import the new system into its list of monitored hosts. The Hyperic HQ server utilizes the open-source PostgreSQL database application to store configuration and monitoring data. PostgreSQL comes prepackaged with the Hyperic HQ software, and can be installed and configured automatically by the installation system. One may also choose to use an existing PostgreSQL or Oracle database server if one exists. The installation system would then prompt the administrator for information about the database so that Hyperic HQ may log in and store its data. By default, Hyperic HQ stores its authentication information within this database as well, but may also be configured to utilize and external LDAP server if one is available.

Auto-Discovery: A unique feature of the Hyperic HQ monitoring solution is its ability to automatically locate and monitor services and daemons running on the remote node. Once the agent is installed on the remote node it can then scan for a variety of known services and add it to the hosts inventory. Once added to the inventory, metrics and alerts can be configured to monitor that particular service. Hyperic HQ supports two scanning options, auto-scan and file-scan. Agents run an auto-scan periodically by default which scans the process list for known server types. A more comprehensive scan called a file-scan can actually search through the file system on the remote node and locate known applications. Because it requires more time to run and is more resource intensive, this type of scan must be scheduled and configured manually by the administrator.

Alerts and Notifications: Hyperic HQ supports the configuration of alerts based on any metric for any particular resource (such as the host itself) or service running on the host. For example, an alert can be triggered when the Availability metric for a host falls changes at all, or falls below a predefined value. When an alert is triggered an email can be sent to a predefined email address. Depending on the priority of the alert, a message will also be posted to the Dashboard, the Hyperic HQ administration front page. The HQ Open Source version lacks many of the more advanced notification options that are available in the Enterprise version. HQ Enterprise also supports the concept of Recovery Alerts, which are alerts that can be configured to cancel and reset triggered alerts. When an alert is triggered in the Open Source version, the alert will continue to be triggered until the problem is fixed or the alert is disabled. Recovery Alerts allow an administrator to automate the process of disabling an active alert, and then re-enabling the alert when the problem is corrected. HQ Enterprise also supports the option of sending SNMP traps as a notification option.

Hyperic HQ Plugins: Hyperic HQ plugins are distributed as .jar or .xml files that are deployed on the server and the agent. Plugins can be developed to enhance the collection of metrics from certain applications or services, locate and inventory new services and control actions to control specific resources. The Hyperic website provides comprehensive documentation on plugin development. Developing and adding a new plugin tends to be a more complex process compared to Nagios or other monitoring applications. The framework provided by Hyperic HQ, however, provides advanced APIs from which the plugins can query information on multiple platforms. On Windows, for example, Hyperic HQ includes classes which a plugin may use to access Windows specific data and functions. These functions can provide access to performance information, registry data, event log information and the Service Control Manager (SCM). Hyperic HQ also provides support for simple script-based plugins to gather particular metrics. Even individual scripts or Nagios plugins may be imported and configured for use by the Hyperic HQ server and agents.

SIGAR – System Information Gatherer And Reporter: SIGAR is the primary data collection component of the Hyperic HQ agent. The software is designed to collect system and process information from a number of platforms - including Linux, Windows, Solaris, AIX, HP-UX, FreeBSD and Mac OSX. SIGAR is written in C, but Hyperic provides C, C#, Java and Perl APIs which one may use to to integrate SIGAR into their applications. The SIGAR component is licensed under the GNU GPL, and is distributed separately from the Hyperic monitoring agent for potential use in third-party applications. The Sigar API provides a portable interface for gathering system information such as:

System memory, swap, cpu, load average, uptime, logins
Per-process memory, cpu, credential info, state, arguments, environment, open files
File system detection and metrics
Network interface detection, configuration info and metrics
Network route and connection tables

user@linux:~/hyperic-sigar-1.3.0.0> java -jar sigar-bin/lib/sigar.jar
Loaded rc file: /home/user/hyperic-sigar-1.3.0.0/sigar-bin/lib/.sigar_shellrc
sigar> help
Available commands:
        alias          - Create alias command
        cpuinfo        - Display cpu information
        df             - Report filesystem disk space usage
        du             - Display usage for a directory recursively
        free           - Display information about free and used memory
        get            - Get system properties
        help           - Gives help on shell commands
        ifconfig       - Network interface information
        iostat         - Report filesystem disk i/o
        kill           - Send signal to a process
        mps            - Show multi process status
        netinfo        - Display network info
        netstat        - Display network connections
        pargs          - Show process command line arguments
        penv           - Show process environment
        pfile          - Display process file info
        pinfo          - Display all process info
        pmodules       - Display process module info
        ps             - Show process status
        ptql           - Run process table query
        quit           - Terminate the shell
        route          - Kernel IP routing table
        set            - Set system properties
        sleep          - Delay execution for the a number of seconds
        source         - Read a file, executing the contents
        sysinfo        - Display system information
        test           - Run sigar tests
        time           - Time command
        ulimit         - Display system resource limits
        uptime         - Display how long the system has been running
        version        - Display sigar and system version info
        who            - Show who is logged on
sigar>
Example SIGAR usage from the command-line.

And that does it for the “Monitoring” section. There are so many other tools we got a chance to play with like Monit, Argus, OProfile etc. but am running out of space …… As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Hyperic: Java-based Cross-platform Management

Sam Ramji — Fri, 15 Jun 2007 18:38:00 GMT

I had the opportunity to sit down with Javier Soltero, CEO of Hyperic last month in San Francisco at the OSBC. We had a great discussion, which I opened bluntly by saying, “You don’t need to tell me about your software; I’ve seen it, my lab team thinks it’s cool, and we’re impressed.” He was happy to hear it but probably not surprised.

One of the obvious pros of the open source model (like the freeware model of the 90’s) is that you can get what you want without calling anyone or firing off a “please contact me” request to the company’s sales department. Another equally obvious pro is that prospective customers can really walk through the product’s architecture and actual implementation to make sure that the marketing promises (“marketechture”) actually line up with the product being described.

Kishi Malhotra and Stephen Zarkos – the OSSL’s experts on manageability – did a comprehensive teardown of Hyperic and a range of other open source management technologies (such as Nagios and OpenPegasus), which they’ll be posting in the next few days. What they found about Hyperic is that it does a great job of making a low-footprint, easily adaptable management technology and is commercializing it in an open source model. We thought that SIGAR, their agent API, was particularly clever.

Javier and Doug MacEachern (their CTO, and a maintainer for mod_perl among other achievements) spent some time on a podcast with me last week – if you’re interested in hearing their reasons for building Hyperic, how it compares to Nagios, and what they learned in taking their product open source, listen in. They’ll be available to answer questions on this post as well – leave a comment if you’re curious about something they’re doing.

Also, drop us a note and let us know if you interested in more interviews with open source and interoperability technology leaders on Port 25.

Systems Manageability Part 4: Systems Configuration

kishi — Fri, 25 May 2007 18:18:00 GMT

Background: This is Part 4, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the second part of the “ontology” which is “Systems Configuration”

Level-Set: System Configuration and Management encompasses all tasks related to the configuration of a host in a standardized and (when possible) centralized way. Many projects in this category provide a common configuration interface, either command-line or GUI-based, designed to ease typical administrative tasks. Other projects, specifically Cfengine, provide a higher level policy-based system to provide consistent configuration and state management for a set of systems. Again, in this case there’s lots of different tools out there that can be used but we have focused on the most popular ones such as Webmin, YaST, SSH, VNC and Cfengine. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I. WEBMIN: "Webmin is a web-based interface for system administration for Unix. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on." Webmin is very modular in design, allowing third-party developers to add support for a particular service or task relatively easily. Many of the tasks involve easing or automating system administration tasks, or editing a configuration file using a specific syntax. Webmin is currently supported by OpenCountry, a company that sells Linux management solutions. The OpenCountry website includes information about Webmin, including two variations of the system that they support.

Webmin Plus: Webmin Plus, is a version of Webmin supported by OpenCountry. According to the website, the Webmin Plus version has been tested by the company and includes new features, specifically support for the back-up and restore application called Bacula (http://www.bacula.org/). Webmin Plus is still freely distributed under a “Mozilla-type” license, and is downloadable via Sourceforge.
Webmin Pro: Last I checked, Webmin Pro had not yet been released, but the following is an excerpt about the product from the OpenCountry website: “...a commercially supported comprehensive product for the entire data center enabling centralized systems administration of mixed distributions, Linux and Windows!”

II. YAST: YaST (Yet another Setup Tool) is an OS installation and configuration utility used primarily in SUSE-based systems. YaST typically serves as the primary control panel interface in, and can be used for a number of configuration tasks – such as adding and removing software, patch management, user management, device configuration and for configuring individual services and daemons. Other common administration tasks such as obtaining system information and reading server logs is also possible via the YaST interface. All of the aforementioned YaST features are implemented as modules, each of which provide a specific functionality or perform certain tasks. These tasks typically involve editing one or more text configuration files on the system in a specific format to configure a specific service or daemon. On other Linux or UNIX-like systems, these tasks are typically performed manually via the command-line.

The YaST utility is very modular in its design, allowing Novell or other third-party providers to add modules into the YaST interface to configure a particular device or service. Many of these modules work independently of each other, and as such are often packaged as individual RPM packages that may be added or removed depending on the software and devices that are installed on a system. YaST modules are written using a scripting language specific to YaST called YCP. Other scripts, such as Perl or shell scripts can also be utilized via a YaST module to perform a particular task. A CIM module for YaST is also distributed with SLES10, which provides a client interface for CIMOM (Common Information Object Manager) to other YaST2 modules. It seems the most common administration task for which YaST is used involves setting up individual package repositories (discussed further in the Patch Management and Maintenance), adding or removing software packages and configuring or initiating online updates. YaST is capable of searching for and locating software on remote repositories, retrieving the software packages, resolving package dependencies, checking the cryptographic signature of the package (if available) and then installing the software on the system. Multiple repositories can be configured. Repositories can be located on a hard disk or CD/DVD, or on a remote system obtainable via HTTP(s), FTP, NFS or CIFS. Once a repository is configured it can then be indexed for later searching. The software search functionality is very powerful, allowing one to search for appropriate software packages using many of the attributes available in the RPM package header – such as the description or contents of the package. Besides software management, the quality and completeness of many YaST modules varies. Many modules (such as the log viewing modules) offer minimal functionality, and only work well enough to provide a few basic configuration options. Complex server configurations will therefore still require one to edit text-based configuration files by hand, or use another configuration engine for the task, such as Webmin. However, many other common tasks, such as configuring display settings or a printer, can be done entirely via YaST.

III. SSH/SCP/SFTP: SSH (Secure SHell) is likely the most widely used remote administration tool for Linux and UNIX-based systems. The typical SSH toolset includes the SSH client and server, as well as the SCP and SFTP client applications for copying files, both of which simply utilize the ssh binary on the backend. The following excerpt is from the OpenSSH project home page: "OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions. Since most any task can be performed via the command-line, the OpenSSH utilities are likely the most critical component for a Linux administrator to have available. The remote copy and command execution options allow one to build, deploy and run a script on a number of machines relatively quickly and securely. OpenSSH is typically installed by default in most Linux-based distributions, although in some distributions the server may by default be disabled or blocked by the firewall. "Most common uses of SSH are:

Remote Command Execution: One of the most common uses of the ssh utility is to run commands in remote machines
Authentication: In order to log into, copy files or run commands on a remote host via ssh, one must first authenticate to the remote machine. Local password authentication may be used without any particular configuration on the server. Linux systems that support PAM (Pluggable Authentication Modules) may also use password authentication to authenticate using a number of mechanisms, including NIS, LDAP, Kerberos, or AD (Active Directory).Rather than requiring a user to manually type in a password, a more common authentication option is to use public key authentication. A user’s public key must be inserted into a file on the remote host called authorized_keys before they are allowed to authenticate to that server. This can often be done via the post installation procedures built into kickstart or autoyast.
Tunneling other Applications: Another common application is to use SSH for tunneling other protocols. This is often used to tunnel protocols that are not typically encrypted such as NFS or X11. By using the –Y or –X switches with the ssh client application, one can “turn on” X11 tunneling, allowing graphical applications run on the remote machine to display locally on the administrators workstation. For example, if one were to SSH into a SUSE-based system and run “yast2” via the command line, the YaST display would be tunneled via SSH and displayed on the local system – even though the actual application is running on the remote system.
Cluster Management: When managing a number of Linux servers, such as a HPC cluster, it is typical to have a large number of systems with identical configurations. To ease administration of these systems there have been a number of SSH-based utilities that allow one to run commands on multiple systems, or copy a file to multiple systems, in parallel. The following utilities are all licensed under the GNU GPL.

ClusterSSH: ClusterSSH allows an administrator run to open a remote SSH session to a number of systems at once via a single terminal window, and run commands or alter configuration files on all the systems simultaneously.
PCP: PCP is a tool designed to copy files in parallel to multiple nodes in a cluster or server farm.
GEXEC: GEXEC is a tool that is somewhat complimentary to PCP that provides a parallel remote command execution system for large clusters. The system includes a client and server, as well as a library that allows integration into third-party applications.
PSSH: PSSH is distributed as a suite of utilities that perform many of same functions as ClusterSSH, PCP and GEXEC.
pssh – Parallel SSH, similar to ClusterSSH or GEXEC.
pscp – Parallel SCP, allows one to copy files to multiples nodes in manner a similar to PCP.
prsync – The prsync utility automates running rsync on multiple nodes in parallel, essentially another method of copying files or entire directory trees to a number of remote nodes.
pnuke – The pnuke command can be used to kill a number of processes running on multiple nodes.
pslurp – The pslurp utility is similar to the pscp utility, except that it is designed to copy files from a set of hosts. This allows one to copy, for example, a log file that exists on multiple nodes, to a local directory tree.

IV. Cfengine: “Cfengine, or the configuration engine is an autonomous agent and a middle to high level policy language and agent for building expert systems to administrate and configure large computer networks. Cfengine is designed to be a part of a computer immune system. It is ideal for cluster management and has been adopted for use all over the world in small and huge organizations alike.” Cfengine consists of a userspace application called cfagent and a host of other utilities that reads and parses a series of text configuration files and performs tasks on the host system based on the configuration. The configuration syntax of Cfengine is actually a high-level policy language that allows cfagent to test the system’s configuration and perform corrective actions based on those tests. For example, cfagent may test to assure that a certain line of text exists within a configuration file, and if not it will add the text and restart the associated service. The cfagent utility is typically run on an hourly (or so) basis via cron, a task-scheduling application. This assures that mis-configurations will be found and corrected within a reasonable time frame.

The policy simply tests to make sure an entry for user root exists within the /etc/shadow file, and also checks to make sure the password matches. This assures that all systems have the same password for the root user. The configuration of Cfengine can become very complex, which would likely not surprise those who have had experience with the tool. The structure of the policy language eases this dilemma a bit, as platform definitions can be made and inherited by other blocks to help determine the appropriate action to take. The configuration is essentially a high-level policy language, and thus the various tests must be built and scripted manually. The toolset is, however, enormously powerful when implemented correctly. But as with many open-source technologies, the learning curve can be quite steep, and one must study the complexities of the tool before it can be competently used in a production environment. A version of Cfengine has been ported to the Windows platform to run under Cygwin.

editfiles:
                    # We have different passwords for lab systems and workstations.
       linux.shadowpasswords.md5passwords.(!workstations)::
{ /etc/shadow
    SetLine "root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::"
    AppendIfNoLineMatching '^root:.*'
    LocateLineMatching '^root:.*'
    ReplaceLineWith “root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::”
}
       linux.shadowpasswords.md5passwords.workstations::
{ /etc/shadow
    SetLine “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
    AppendIfNoLineMatching ‘^root:.*’
    LocateLineMatching ‘^root:.*’
    ReplaceLineWith “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
}
Example Cfengine policy to check the password for the root user.

The following example Cfengine policy checks for the existence and the contents of the /etc/cron.d/yast2-online-update file for SUSE systems. If necessary it creates the file, and writes a cron entry into the file to schedule a daily check for updates and patches. Upon completion, it then runs the command “/etc/init.d/cron restart” as defined in the suse.restartcrond definition.

editfiles:
        suse::
                { /etc/cron.d/yast2-online-update
                  DefineClasses "restartcrond"
                  Umask 077
                  AutoCreate
                  BeginGroupIfNoLineMatching "^.*[\s\t]+root[\s\t]+online_update"
                      AppendIfNoSuchLine "30 3 * * * root online_update"
                  EndGroup
                }
shellcommands:
        suse.restartcrond::
                "/etc/init.d/cron restart"

Example Cfengine policy to assure that SUSE systems check for updates daily.

And that does it for the “Systems Configuration” section As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.