Port 25: The Open Source Community at Microsoft : Interop, Management

Part 1: Lessons I Learned as a Project Manager Converting to Agile

saraford — Mon, 19 Oct 2009 19:16:00 GMT

Before I became the Program Manager for CodePlex.com, Microsoft's open source project hosting site, I worked on the Visual Studio team on four different product cycles. Since Visual Studio uses traditional Microsoft product lifecycle releases, I had to learn about Agile development alongside learning about open source development when I joined the CodePlex.com team. Making the switch from releasing every three years to every three weeks didn't happen overnight!

One of the things I discovered is that Agile is used a lot in open source communities. In fact, many of the people who I've worked with personally on learning Agile have strong roots in OSS. Also, the fact that Microsoft is starting to adopt Agile philosophies shows how the company is changing, becoming more transparent, finding more ways to connect with the community, and embracing other schools of thought. And this is why I am here, to be on the inside to push for these cultural changes within Microsoft.

Although Agile is the single greatest thing a team could do to significantly improve the user experience and quality of their website, I believe it can be quite challenging for anyone not in a developer role to get accustomed to. I hope that by sharing my experiences, I can help others in non-developer disciplines on an Agile team.

In this series of blog posts, you'll discover how I learned to program manage an Agile team after six years of waterfall (the traditional method of software development).

Three Major Takeaways

If I could go back in time, here are the three things I would tell myself about Agile.

1. Design and plan for the very next step.

It's not about reaching the moon, but getting out of your front door. Sure, you can design the perfect feature, but if it is going to take you six months to get there, it is useless to an Agile team. It's about designing the journey towards the perfect feature that matters.

2. Break down work into the smallest possible functional sets.

Adding work is fun and rewarding, but removing incomplete work due to a lack of development time is painful and risky. But, you can't deploy a half-written feature either. First, break down the work into the smallest pieces. Then, put together the smallest functional sets that have to be deployed together for the feature to make sense. Your development team will tell you how many sets they can do per release.

3. Design and plan only 80% of the way.

Not designing the full 100% is a true blessing in disguise. Since you have another release right around the corner, you have the time to collect user feedback and incorporate it into the next design. Not only does this solve the remaining 20% (getting you closer to the perfect design with less cost), but also allows your customers to be virtual members of your team.

The Program Manager Release Cycle

To begin, here's an Agile release cycle from the point of view of the Program Manager. For simplicity, this illustration only depicts a single release cycle, without any overlap of previous or future cycles.

Ratings and Reviews: An Example

To further illustrate, consider the ratings and reviews feature on CodePlex.com as an example. Users can rate a release and write a review for projects on CodePlex.

One quick aside about ratings and reviews: CodePlex users rate an individual release instead the entire project. For example, consider Stephen King as an author. What does it mean for me to rate Stephen King 4 out of 5 stars? I find some of his books to be awful, like the Tommyknockers. I want those 7 hours of my life back. 1 out of 5 stars. Yet, for me, some of his other books are incredible, like The Dark Tower. 5 out of 5 stars. Hence, we allow users to rate an individual release to provide more relevant information to potential downloaders of the project.

Let's explore the ratings and reviews feature step-by-step in the Program Manager shoes.

1. Design Phase Part 1: Limit the scope to designing the minimum to make the feature useful and meaningful. For ratings and reviews, the feature must have the following:

a. User can rate a release. User can view the rating.

b. User can write a review. User can read the review.

c. User can sort by highest rated releases in project directory.

2. Design Phase Part 2: Bucket into smallest deployable functional sets. For me, personally, I use sticky notes to illustrate the "must have" pieces for each set.

a. Sticky Note #1: Rate releases / View rating

b. Sticky Note #2: Write a review / read review

c. Sticky Note #2: Sort by highest rated releases

3. Iteration Planning Meeting: To start the development cycle, meet with the development team to discuss costing.

a. In the case of ratings and reviews, my devs said they could do Sticky Notes #1 and #2, but the project directory sorting feature would have to wait for the next release.

4. Dev Cycle: Because the designs are closer to 1-page specifications rather than fully-documented implementations, questions will come up from the dev team. This is where you, the Program Manager, will:

a. Answer any questions about the missing 20% of the specifications / wireframes

b. "Course correct" (more on that later)

c. Add more feature work if time allows

5. Deployment: The release goes live. Now you can collect user feedback and incorporate it into the next development cycle.

a. The very first tweet I saw regarding the ratings and reviews feature was "Sara, is there a way to sort by highest rated?" Here, I was able to ask the user questions about how this feature should work to confirm our designs. Most of the time we don't inform users what's coming up next (we like surprises.) But in this case, it was pretty obvious.

Here's the visual representation of the ratings and reviews feature in the Program Management agile release cycle:

Conclusion of Part 1: The Agile Program Management Cycle

This concludes my first post on Program Managing an Agile team. Since I could go on endlessly writing about topics I'm passionate about (and those who know me will confirm this is not an exaggeration), I'm going to pause here to conclude this initial train of thought.

If you like what you see, let me know! And if you don't like what you see, please don't hesitate to let me know. Seriously, I love discussing my Love / Hate Relationship with Agile development, as depicted on my personal blog.

Up next: The concept of Course Correction.

GroundWork Open Source Joins Microsoft's System Center Alliance

Peter Galli — Thu, 24 Sep 2009 13:31:00 GMT

GroundWork Open Source, Inc., a commercial open source company that produces network management software, last week announced the availability of the GroundWork Connector for Microsoft System Center Operations Manager.

The company has also become a member of the System Center Alliance.

GroundWork Monitor, which already has more than 1,500 plugins available, integrates with System Center Operations Manager and extends monitoring and management coverage to non-Windows systems, applications and devices.

The new GroundWork Connector pulls information from System Center Operations Manager and displays it within GroundWork Monitor Enterprise, giving customers a deeper visibility into the availability and performance of all critical infrastructures on a single console. The connector gives insight into applications, databases, virtual machines and network devices that may be running on Linux, Unix, Windows or embedded operating systems.

I talked to David Dennis, the company's senior director of marketing and business development this week about the move, which he feels is a great follow-up to the release of the System Center Cross Platform extensions earlier this year.

That release broke new ground for using System Center in heterogeneous environments. "In the field, we have more and more users asking about how they can integrate the management of Windows with open source tools for managing network infrastructure, Unix, Linux, and the applications that run on top of them," he told me.

The dialog also no longer seems to be about choice between Windows or Open Source but rather "I want both - now how do I make them work together," he says. Even though GroundWork Open Source is an open source company, about half of the operating systems managed by GroundWork Monitor are running Windows.

"The combination of System Center Operations Manager and GroundWork Monitor provides a full-featured alternative to traditional systems management frameworks, but with greater openness and at a much lower price point," Dennis says.

Tuxera Signs File System IP Agreement with Microsoft

Peter Galli — Wed, 26 Aug 2009 12:21:00 GMT

Tuxera, a Finnish company that develops NTFS drivers, has entered into a file system IP agreement with Microsoft in the area of data portability for devices

The agreement gives Tuxera access to the exFAT specifications and source code, as well as testing and verification tools, and the company will develop exFAT drivers for host device manufacturers. The company will also join the Interoperability Vendor Alliance.

While this is also the first-ever file system IP agreement that Microsoft has signed with an independent software vendor, Tuxera joins a growing number of companies entering into the exFAT Program.

The Extended File Allocation Table (exFAT) is a new file system that is better adapted to the growing needs of mobile personal storage. It allows a common file system to be used across all platform and devices that implement exFAT, which allows for data portability.

It also handles not only large files, such as those used for media storage, but also enables seamless interoperability between desktop PCs and devices like portable media devices, cameras or even kitchen devices so that files can easily be copied between desktop and device.

"Adding exFAT into our product portfolio is the logical step to help our customers to solve their interoperable file systems needs. Microsoft supports exFAT as the first-choice for many Windows interoperability needs including flash memories in consumer devices. exFAT is for example an integral part of SDXC formats, and the driver is good for flash drives, including devices that use SDXC cards," says Tuxera CTO Szabolcs Szakacsits

For his part, Mikko Välimäki, the company's CEO, believes that his company's ongoing collaboration with Microsoft ensures seamless file system interoperability into the future.

"We were the first to offer exFAT drivers based on our first-hand access to specifications and source code.We are also licensing exFAT drivers to device manufacturers, we are joining Microsoft's partner program, and have also announced a new product: Tuxera exFAT for Embedded Systems, which will be first available for Linux, but we can also port the product to proprietary platforms," he says.

The IP agreement the company has signed with Microsoft is consistent with Tuxera's business model, which is to integrate and license commercial, high-performance versions of its file system drivers to customers, as well as to provide support, maintenance, and product updates, Välimäki says.

Project Quant

Peter Galli — Wed, 15 Apr 2009 22:54:00 GMT

I noticed today that my colleague Jeff Jones in the security group is launching a metric project that appears to be leveraging some of the good bits of open techniques.

I touched base with him briefly and he gave me a little more information about Project Quant, which is being undertaken along with Securosis, an independent security research firm.

Project Quant will be working on the metrics of patch management and is as much an experiment of a new research process as it is one of security metrics, said Securosis founder Rich Mogull in a blog post.

"For this project Jeff wanted to be involved, but also asked for an open, unbiased model that will be useful to community-at-large (in other words, he didn't ask for a sales tool). Rather than us developing something back at the metrics lab, Jeff asked us to lead an open community project with as much involvement from the different corners of the industry as possible," Mogull said.

While he also acknowledged that it is risky for Securosis to allow direct involvement of the sponsor, the company is hoping that the process works the way it thinks it will and which also happens to match Microsoft's project goals.

So, this is what's expected to happen: a project landing site has been set up at Securosis that will contain all material and research as it is developed; every piece of research will be posted for public comment and no comments will be filtered unless they are spam, totally off topic, or personal insults.

All significant contributors will also be acknowledged in the final report, although there will be no financial compensation for contributors and the project itself will retain ownership rights. All material will also be released under a Creative Commons license, with spreadsheets released in both Excel and open formats.

"In short, we are developing all research out in the open, soliciting community involvement at every stage, making all the materials public, acknowledging contributors, and eventually releasing the final results for free and public use. The end goal of the project is to deliver a metrics model for patch management response to help organizations assess their costs, optimize their process, and achieve their business goals. Let us know what you think, even if you think we're just full of it," Mogull said.

For his part, Jones told me that while he has been zealous in past reports about using repeatable methodologies, pointing to his source of public data, and outlining his assumptions step-by-step, he would like to take transparency one step further by developing models and methodologies first, in an open and transparent manner, so that everyone can agree on the pros and cons before the methodologies are applied.

"I think being completely open and transparent will help credibility since, similar to open source, everyone can scrutinize every step of the analysis ... creating open models and potentially getting community involvement just seems to be the right process," he says.

I plan to interview him at greater length in the next few weeks, so look for a follow-up blog then.

Two Years and Counting....

Peter Galli — Wed, 19 Nov 2008 04:16:00 GMT

It is two years this month since Microsoft and Novell struck their ground-breaking technical collaboration agreement, a move that has effectively ensured greater interoperability between Windows Server and SUSE Linux Enterprise Server.

This technical collaboration has already resulted in a number of milestones, including two new offerings announced today: the availability in the first half of 2009 of an Advanced Management Pack for SUSE Linux Enterprise for Microsoft System Center Operations Manager 2007 R2, and a free beta download of Novell's Moonlight, a rich media application.

Some analysts, vendors and enterprises have said the company that develops effective cross-platform management tools will have an advantage and strategic differentiator over its competitors who do not. Microsoft is already doing that.

The Microsoft Operations Manger 2007 Cross Platform Extensions enable the assessment and management of Windows and Linux servers from a single, unified console, eliminating the costs and complexities of having multiple management consoles. The Advanced Management Pack extends this Linux monitoring capability.

Also, given the current tough economic environment, this solution helps reduce training costs since staff only need to be trained on one management tool for both Windows and Linux environments.

Attendees at the Microsoft TechEd EMEA conference in Barcelona earlier this month got to see a technical preview of the Advanced Management Pack, whose release will coincide with that of Microsoft System Center Operations Manager 2007 R2.

A beta of Novell's Moonlight, an open source implementation of Microsoft Silverlight, will also be released going forward as an open source plug-in for the Firefox web browser. Moonlight brings Linux-based users the same high-definition media capabilities currently available for the Windows and Apple environments.

So, expect to see a lot more solutions in the next year that promote interoperability and help ease customer pain-points across their heterogeneous environments.

ApacheCon and the Stonehenge Proposal

Kamaljit Bath — Mon, 10 Nov 2008 21:24:00 GMT

This is Kamaljit Bath, and I am in the Big Easy to experience my first Apache Conference! I am a Principal Program Manager in the Interoperability Technical Strategy Team at Microsoft. We have been doing a lot of great interoperability work and have done quite a few Open Source projects to build bridging solutions etc., but this is the first time I am attending ApacheCon. This is a learning experience for me.

Microsoft is certainly interested in expanding interoperability between Open Source solutions and Microsoft technologies, and is working with individuals and communities for that purpose. I think this is great because it will enable choice of solutions and create strong partnerships to promote growth for everyone in the industry.

ApacheCon has been quite an experience for me so far. I have seen the energy and high quality decision making. It is amazing how people from many different backgrounds can come together and accomplish so much in so little time.

I have also met some very interesting people and some that I have wanted to meet for a long time. Meeting motivated and driven people is what I like the most about conferences. I have learnt a lot from this experience and I will take back great memories from this trip.

In his keynote today Sam Ramji, the Senior Director for Platform Strategy at Microsoft, gave an update on the many interoperability and Open Source projects that Microsoft is engaged in. I am sure that some of this was news to many of the attendees, but hopefully it gave them an idea of the breadth of work that Microsoft is doing in this area.

Sam covered a lot of things, including our participation in Apache QPID project; the release of the 'Oslo-M' language under the Open Source Promise; participation in the Apache HBase project; and support for the new Stonehenge proposal by WSO2.

Sam also covered many other open source projects that Microsoft has used to build bridging technologies, while my manager, Jean Paoli, has covered these in detail in his blog.

Sam is a well known figure in these avenues and needs no introduction - he has been representing Microsoft at many of these conferences.

But it is also important to have a more grass-root level developer and architecture presence from Microsoft, and we are now moving in that direction. Hopefully, we will see an increased Microsoft presence at such events.

Microsoft is also supportive of the new Apache incubation proposal - Stonehenge - that was proposed by WSO2. It will focus on building a set of sample applications based on approved W3C and OASIS standard protocols with goal of proving interoperability between different implementations on various platforms.

I think these sample applications will provide developers a great starting place for their tasks by providing best practice guidelines and reference implementations on various platforms. They will also help find potential interoperability problems and hopefully develop into a great community to discuss the architecture of multi-tier SOA apps. We look forward to working with WS02 on the scope of this project, and having discussions with the community.

These are exciting times for the software industry and we are seeing the co-existence of commercial and open source software and coming together of various forces to create solutions for the new heterogeneous IT environment.

Onwards, with great faith and hope!

How open source has influenced Windows Server 2008

Sam Ramji — Wed, 27 Feb 2008 12:00:00 GMT

When I think about what works really well in open source development and technology, the following things stand out:

Modular architectures
You can find these wherever you see participation at scale – and often a rearchitecture to a more modular system precedes expanded participation. Great examples of this are Firefox, OpenOffice, and X11 – from both the historical rearchitecture and the increased participation that resulted. The Apache HTTP server and APR are good examples that have been modular for as long as I can recall.
Programming language agnostic
A given project uses a consistent language, but there are no rules on what languages are in scope or out of scope. Being open to more languages means opportunity to attract more developers – the diversity of PHP/Perl/Python/Java has been a core driver in the success of a number of projects including Linux.
Feedback-driven development
The “power user” as product manager is a powerful shift in how to build and tune software – and this class of users includes developers who are not committing code back, but instead submitting CRs and defects – resulting in a product that better fits its end users.
Built-for-purpose systems
Most frequently seen in applications of Linux, the ability to build a system that has just what is needed to fulfill its role and nothing else (think of highly customizable distributions like Gentoo or BusyBox, as well as fully custom deployments).
Sysadmins who write code
The ability of a skilled system administrator to write the “last mile” code means that they can make a technology work in their particular environment efficiently and often provide good feedback to developers. This is so fundamental to Unix and Linux environments that most sysadmins are competent programmers.
Standards-based communication
Whether the standard is something from the IETF or W3C, or simply the implementation code itself, where these are used projects are more successful (think of Asterisk and IAX2) and attract a larger ecosystem of software around them.

So where did we apply these ideas to the development of Windows Server 2008?

Modular architectures was applied in multiple areas, but the one that stands out most to me is Internet Information Server 7 (IIS7). IIS7 has been rearchitected for flexibility as 40 individual modules, enable more to be written by community developers or delivered as out-of-band releases. This has already enabled performance improvements and independent evolution, and I expect to see further enhancements.
Programming language agnostic is something we’ve delivered on with support for PHP on IIS7 and the enhancements to FastCGI (which can be used by any of the P* languages). We set a goal of having PHP certified on Windows Server 2008, and we’ve achieved that. We’ll continue to improve runtime, security, and manageability support for non-.NET languages and the applications that are built on them, as well as testing the full stacks of PHP-based applications running on Windows Server, IIS, and SQL Server.
Feedback-driven development based on developer and customer trials (RDPs, TAPs, and Betas in our process) led to a range of “feature completion” developments that connected different components – like connecting Windows Firewall with Active Directory central policy, and the end-to-end improvements in SMB 2.0. Features like the RODC (Read-Only Domain Controller) have become more and more solid through experience with early alpha and beta customer deployments, and requests to enforce things like BitLocker encryption of user disks from a central authority have achieved full support.
Built-for-purpose systems such as DNS, DHCP, file and web serving can be created through wizard-driven configuration thanks to Windows Server Core. The goal of having a minimum attack surface and a small hardware footprint, inspired by the capabilities mentioned above, yet achievable by a broad base of admins has been achieved. Additionally, this has created an opportunity for Windows admins to become much more knowledgeable about the low-level structure of the operating system.
Sysadmins who write code are first-class citizens in the PowerShell-driven infrastructure. We’ve increased Windows administrators’ opportunity to master the full surface area of WMI and demonstrate that mastery in reusable, low-level scripts. As we evolve this to support multiple language bindings and bash aliasing, this should become a comfortable home for highly skilled sysadmins.
Standards-based communication such as in CardSpace (with support for X.509, SAML, Kerberos tokens, and more) and the Web Services stack (not only are all 38 Web Services standard under the Open Specification promise, but our implementations have achieved a high level of interop with Apache’s Axis web services stack), and beta support for emerging standards like Xen virtualization represent a small subset of the standards built into Windows Server 2008.

Overall, we’ve learned and continue to learn from open source development principles. These are making their way into the mindset, development practices, and ultimately into the products we bring to market.

I’ve focused here on “what Microsoft has learned from Open Source” – and ironically, I’ve agreed to do a panel at OSBC on 3/25 with Jim Zemlin of the Linux Foundation on “what Open Source can learn from Microsoft”. As all of the different organizations in IT continue to evolve, we’ll learn from each others’ best practices and make increasingly better software. As in science, this incremental improvement will move all of us forward.