Port 25: The Open Source Community at Microsoft : Hank Janssen, OSCON

Introduction to the Linux Integration Components

hjanssen — Wed, 22 Jul 2009 19:02:00 GMT

Well, here is blog number two. The initial shock has worn off a bit I hope.

The feedback I have received so far has been pretty positive. This really all started in October of 2008 in a meeting with Mike Neil (GM of Hyper-V) and Tom and myself from the Open Source Technology Center (OSTC) at Microsoft.

In that meeting I proposed to Open Source the Linux Integration Components and contribute them to the Linux Kernel. And, secondly, to have the OSTC continue contributing to these IC's after they made it to the Linux Kernel. Well after some discussion, we all agreed that this was the right thing to do.

And so the whole process started inside of Microsoft. Hey, what can I say, we like to push the envelope a bit here at the OSTC, and we have a reputation to uphold!

Before I go on, I again wanted to thank the Kernel community (specifically Greg Kroah-Hartman) in helping us with explaining and guiding us through community process. It gave us a very nice jumpstart to get all of this going, and provided the groundwork for a good working relationship with the community.

I have also seen a few patches already submitted by community members, which is excellent! (Moritz Muehlenhoff gets major kudos for the first community contributed patch ) I will start submitting patches myself next week once the initial submission has stabilized a bit.

It is my plan to use the kernel as my primary development area, and of course I will continue to provide Greg with my patches. My first step is to clean up the code to make sure it fulfills all Kernel coding standards and requirements.

So, here is blog number two: what are the Linux Integration Components?

1. Overview of Linux VM with ICs

Linux Integration Components(IC) take advantages of the VMBUS and synthetic devices provided in Hyper-V to enhance the performance and usability of Linux guests running on Windows servers.

Figure: Conceptual Architecture overview of Linux guest & Hyper-V. Linux IC modules are painted in yellow color.

Glossary:

VSP: Virtualization Service Provider.

VSC: Virtualization Service Client.

VMBus: Data channel between VSP and VSC.

2. Linux IC modules -- VMBus and VSCs

Communication with parent partition is done through Linux VMBus
VSCs are the Linux drivers for synthetic devices (SCSI, IDE, and Ethernet) provided by Hyper-V.
They translate between Linux I/O requests and Hyper-V VSC commands
Devices are registered with Linux Driver Model (LDM)
Every VSC module contains two portions:
o Driver Interface Mapper (DIM): Released as open source

This portion of the VSC component interacts with the Linux kernel like a regular Linux device driver.

o VSC Core: Released as Open Source

The core portion of the VSC module is implemented based on the protocol of the corresponding VSP at Hyper-V host. The VSC core interacts with VSP via the VMBus interface.

3. Descriptions for each Linux IC module

3.1 VMBus driver (hv_vmbus.c)

The VMBus driver is a Linux kernel module. It provides both a lightweight bus driver and library functionality. As a bus driver, it registers with Linux Driver Model framework (LDM) to provide simple bus and device integration and device tree integration (sysfs). As a library, it implements the VMBus channel protocol and provide an abstraction of channel to its clients (Disk and Network VSCs).

3.2 StorVSC driver (hv_storvsc.c)

The Storage VSC interacts with the Windows Storage VSP. The "wire" protocol defined by the storage VSP determines how a VSC interacts with it. The Linux Storage VSC (LSVSC) basically abstracts the Linux I/O stack from needing to understand the Storage VSP's protocol. At the upper-edge of the LSVSC, it talks to the Linux SCSI subsystem. The Linux SCSI subsystem sees the LSVSC as a SCSI low-level driver (LLD) in Linux parlance. It passes SCSI requests (scsi_cmnd) to LSVSC which in turn converts them into the "wire" format understood by the Windows Storage VSP (VSTOR_PACKET). The bottom-edge of the LSVSC talks to Linux VMBus (LVMBUS) which in turn talks to the Windows VMBus to route the packets to the Storage VSP.

3.3 BlkVSC driver (hv_blkvsc.c)

BlkVSC (BlockVSC) supports "fast boot" and fast access to IDE disks. To enable enlightened IDE support for enhancing the performance of Linux when virtualized on Windows, a separate BlockVSC component is used as a Linux block device driver. Like StorVSC, the BlockVSC component is comprised of an upper edge wrapper that interfaces with the Linux block layer and a lower-edge through the infrastructure modules. The infrastructure modules with Hyper-V through the Linux VMBus.

3.4 NetVSC driver (hv_netvsc.c)

The network VSC send and receive network traffic between a Linux guest and Hyper-V host which has direct connection to physical network. The mechanism that this is used to accomplish is the Remote NDIS (RNDIS) protocol. Thus the communication that flows between the VSP and the VSC primarily happens over the RNDIS protocol which then is packaged and forwarded as payload over to the other side over NetVSP / VMBus protocol.

4. Linux IC's, Location in the Kernel tree

Hopefully you now have a better idea what they are. But where in the kernel tree can you find them?

Well, you can find sources in linux-next tree in /drivers/staging/hv directory.

And the git repository you can find them in right now is:

git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git

Or give this command (assuming your system is set up correctly) to download this repository to your machine:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git <your local name>

Since the IC's are part of the kernel now, we follow the normal community process of getting this all migrated into Linus mainline kernel.

OPEN/SHARED SOURCE AT Microsoft

MichaelF — Tue, 12 Sep 2006 21:25:00 GMT

It has been a while since I posted a blog, and I really have no other excuse than that I have been very busy. I have had a whole bunch of blog ideas percolating in the back of my mind, and I will be writing them down soon.

When we started port25 and the OSSL it was met with great skepticism. But there have been a lot of changes going on around us here at Microsoft. And one of those I wanted to bring to your attention.

A few years ago the mere thought of Open Source at Microsoft was ridiculed both inside and outside of the company. But I am starting to see small and sometimes not so small changes. This blog describes a very positive change.

As you might all know, I went to the 2006 OSCON conference in Portland. And there I met another Microsoft employee, Sara Ford. She works in the Visual Studio and Power Toys area. She has been a very active blogger in the past (unlike myself, working on it though!) And you can find her page here.

We got to talking at the conference and I have worked with her a little since then and found her to be a very energetic person greatly interested in OSS. But why is this interesting??? Well she attended a session at OSCON given by James Howison. (See his OSCON session info here ) And his presentation was on open source communities.

She was so impressed by it that she is currently working on Open Sourcing the Power Toys. I had the pleasure to sit in the training she gave the team, you can see more of the training she gave (unfortunately I was there as well and probably messed up the whole video by opening my mouth. So ignore me!) here.

In any case, who would ever thought Microsoft would open sourcing anything. But it is happening, and in future blogs I will give you all more insight on my first 4 or so months here and the changes I am seeing both internal and external.

Till then!

Hank.

Learning from OSCON 2006

jcannon — Tue, 15 Aug 2006 14:37:00 GMT

Preliminary stuff

Hank Janssen and myself attended the OSCON on the 27^th and 28^th of July. We did not attend the tutorials or the Executive briefing but were there two days of the two and a half days the sessions were in progress. We also attended the keynotes on both days (27^th and 28^th July).

As a strategy, Hank and I discussed the sessions and their subject matter, splitting up to attend different sessions in order to maximize coverage. In general I attended the “business” and “strategy” sessions and Hank attended the more technical sessions.

I’ll cover the sessions I attended. Hank can be responsible for his own thoughts!

This was the first OSCON I have attended, even though I have been to other conferences where there was large open source presence, so it was very exciting for me! I’ll talk about some of the sessions I attended in chronological order.

Some of the sessions I attended are not covered here, because I wasn’t impressed with them. So even in Open Source software there are some, shall we say, “imperfections”! J

The sessions are hyperlinked from the “index” below, that way you can just jump to the one you want without getting meta-carpal tunnel syndrome from blog scrolling!

First impressions

The conference was very well attended. It was clear that there were not only traditional open source “hackers” and startups (though there were many of those), but there were a number of established enterprise vendors (HP, Dell, AMD, VMWare). It also seemed that the startups were a lot more mature than Open Source startups from the past – their message was clear, but not strident.

Surprisingly, IBM pulled out from the conference at the last moment and there was nobody (visible) from IBM at the conference. Google was present and had a number of presenters, but had a booth only for recruiting. (They did announce their portal to Sourceforge at the conference).

There was a dearth of enterprise customers – all the people I met fell into the vendor, academic or Open Source organization ( Mozilla, Apache) category.

The only customer (non-software related company) with an official presence was Ticketmaster, who were recruiting for Linux admins and Open Source developers.

There was a lot of talk about Open Source as a business, a number of keynote speeches and sessions addressed this. There didn't seem to be buzz about a particular technology or company that stood out.

The conference was a good place for me to gauge which products/technologies/companies were gaining momentum and get opinions through face to face contact with both users and principals. It also was a place to make an assessment about what was going well, what was not going so well and concerns of the Open Source community.

A Closed Source Project becomes Open Source – How We Succeeded (Lars Thalmann, mySQL)

Lars Thalmann was part of a team that was acquired into MySQL from Ericsson’s Business Innovation divison. The product that they worked on was a closed source Ericsson product called “Alzato” - which was a clustered database system used mostly by stock markets and telcos.

The product was successful enough that there was a 60 person team in Ericsson developing and maintaining it. mySQL acquired the entire team along with the product and renamed the product mySQL Cluster. mySQL Cluster is open source just like mySQL’s other offerings.

Alzato was meant to be a high availability and performance database with five nines (99.999 %) availability & had a parallel architecture with replication for speed and scalability. In short, this was an advanced technology project.

The architecture was changed so that the clustered storage engine of Alzato was now accessed through mySQL and NDB in mySQL Cluster rather than through SQL and NDB as before.

Lars presented the talk as 10 “shocks” that the closed source team had to go through when they found out what was different between open source and closed source.

Some of the things he said (I will not try to transcribe his entire talk here) were of the nature - “It should be possible to install software in less than 15 minutes" since “The community consists of people with little patience. You surf, you find something, you try it – if it does not work right away you move on!”

The more I listened to Lars, the more I was convinced that open source had forced developers to adopt good practices, just by the nature of development rather than by any coercion. Microsoft also followed the same practices, at least within a large development team, but had come to those processes by painful experience!

The one thing that I learned was to make all interactions explicit - by having bug databases and forums that capture every small piece of information that might be needed. Having community coaches and documentation constantly improved by user review was another highlight.

The thing that really struck me was how strong the motivation of developers could be if they were able to directly interact with the users. Lars said - “Developers work all the time (rather than 9 to 5), being inspired by the feedback and suggestions – which makes people enthusiastic”

So, You want to Build an Open Source community: Learning from Apache, (J Aaron Farr, Member, VP Apache Excalibur)

Apache Incubator is the incubator of projects for Apache. It takes projects and project proposals for open source projects and evaluates them for suitability as Open Source Projects under the Apache umbrella.

The talk was focused on “what makes a project a good candidate to be open sourced through the Apache incubator”?

The thing that I took away from the talk was the danger signs of an Open Source project NOT being a strong project . The list that Aaron Farr presented was:

Some trying to turn an “Orphaned product” into an Open Source one
Inexperience with open source of the team doing the project
Homogenous developers – from the same company, academic institution rather than a diverse group of people from all over
Reliance on salaried developers only and not on a community that is enthusiastic about the project
Fascination with Apache brand, wanting the name, without aspiring to do the work!

An interesting comment was made by the presenter – “Java Enterprise space may be the best analogy to Microsoft OSS activity”. I am not sure I completely followed that – any reader care to comment? The other thing that stayed with me was “One of the upsides to being IN a healthy, thriving OSS project are a renewed enthusiasm for software”. They must have been talking about my job here at the Open Source Software Lab!

Lessons Learned in Taking a Closed Source Product Open (Neelan Choksi, Sr. Director, BEA Systems)

Neelan Choksi was President of Solarmetric that produced a Object Relational database mapping engine called Kodo. Kodo was closed source product, even before it was acquired by BEA.

BEA open sourced Kodo as Open JPA which include the kernel and the J2EE EJB 3 Persistence specification implementation. The decision to make the O/R mapping engine open source was taken in February 2006 and the product was released in July (it took 6 months to open source the project).

Explaining the business reasoning behind the decision, Neelan suggested that the reasons one would want to open source a product were:

Create ubiquity – large distribution of product
As seeding strategy for other products that the company hoped to make revenue from
Due to competitive / commoditization pressures. Kodo was already feeling pressure from its competitors who were open source (Hibernate) even before the merger with BEA.
The Razor / Razor blade strategy. The open source product is free but a recurring critical part that was used by the open source product was not.
Religious fervor – belief in open source “as free as in freedom” philosophy
Financial gain – cost decreases from bug fixes, development, support of product counteract any lost revenue from licenses
For marketing purposes to develop a brand based on a piece of software that can be leveraged later
Strip mining and waste dumping – dumping a product that isn’t valuable
Real world test to understand how open source works.

It was interesting to see how a company that primarily relied on closed source adopted an open source strategy. Hearing it from someone who had been through it himself was refreshing. (Rather than hearing from pundits who theorized on the topic without having the real world experience!). He also said that it was hard to get a business behind an open source strategy, because as soon there was a priority conflict the “old business” people would try to de-prioritize the open source projects.

The one takeaway that Neelan wanted us to have was that open source was not magic dust. To develop a product ,whether you have people with those titles or not, you do need Product Management, QA and (yes!) Marketing.

According to him a business is a business so the success of a product will be determined by overall execution not technical excellence or the closed/open nature of the code

Birds of a Feather Session, “Open Source in Higher Education”, Moderator – Bart Massey, Portland State University.

This was a session with significant participation from Portland State University (PSU) and Oregon State University (OSU), but with other members from as far afoot as Texas and Tennessee. Some of the people were lecturers, some faculty and some were system administrators from academic institutions.

When I asked whether they were aware of the Academic program that offered Windows source code for educational purposes – most seemed to be vaguely aware of it. One of them suggested that Microsoft write a textbook that included programs and instructions on how to use the source code. There are no such textbooks available for academic institutions to use as yet.

Another had anecdotal information of the cost of training students for IT Pro. They said they were teaching Windows because the license for setting up a lab was cheaper from Microsoft – which also had very cheap certified 'train the trainer' programs. Red Hat was almost an order of magnitude more expensive – and their train the trainer program required expensive yearly renewals. Besides, Windows admins were more readily employable in their local regions.

There was some discussion of using open source to teach academic lessons vs. developing open source itself as part of academic training. It seemed like opinions were divided as to the utility of each approach.

Bart Massey talked about the Open Source Education Lab at PSU. Among other things he runs a course (a summer long lab) on Open Source. We are talking to Bart about seeing if any of his students are interested in working at the Open Source Software Lab. Any of you readers out there interested? Drop us an e-mail!

The (Surprising) History of Copyright, and What It Means for Open Source, Karl Fogel, Google, Inc.

Karl had an interesting argument which is interesting not just because he works for Google. His contention is that copyright is not for the benefit of the creator but for the benefit of the distributor. And given that the Internet has made cheap ubiquitous distribution really easy, copyright has lost its utility.

An interesting tidbit that he mentions is that the theory of copyright was advanced to protect their interests by the Royal Stationers guild when censorship, which was implemented by printing only being allowed through the guild, was revoked by the English parliament. He takes the radical position that copyright be abolished and be replaced by some other fair mechanism that doesn’t benefit the distributors but benefits the creators. This puts even the GPL in jeopardy because the basis of the viral nature of GPL is copyright.

Even though his ideas may seem radical, the argument about the nature of distribution changing the landscape for businesses should be taken seriously. His site is at www.questioncopyright.com

Building Succesful Open Source Projects (Jorg Janke, Founder and Principal, Compiere)

Jorg Janke is a founder of Compiere – the premiere vendor of an open source ERP product. According to Jorge, Compiere has seen over 1 million downloads since 1999 and is a top 10 most downloaded product on SourceForge. They have 250+ customers and have concentrated on Product, Process and Distribution. They have 70 partners who play an important part in their ecosystem.

Jorg believes that there is no “one size fits all” open source development model. He said that as a case in point SugarCRM tried Compiere’s model but evolved to other models. He also said that Compiere has been evolving their model constantly since 2002.

He suggested that there were some myths about OSS development

Heinzelmännchen (Good Gnome/Fairy) Theory – put something defective/half-finished/unsuccessful technology out there and some “Good Gnomes” in the open source community will fix it for you!
“Sexy” application theory – according to him, with “sexy” technology, men try something 3-4 times longer before they give up! He said that free had limited sex appeal!

According to Jorg “The Basic Open Source Contract” was

I give you a gadget for free
You can use it

Provided you have skills & time.
If you need help you need to pay for it

3. I hope that in return you help by

Spreading the news
Giving feedback
Offer advice/help
Improve the product

I was impressed by Jorg’s grasp of the software landscape – he was no radical hacker developing in his garage, but a clear thinking businessman with a great grasp of the software business.

He said Compiere first and foremost was a product that solved the major pains of all ERP projects, the installation and implementation. There were no compromises in Compiere because it was open source, it all the features necessary to its users. They also had a clear strategy, Compiere made the enabling product but Compiere’s partners sold it to users.

Another thing that they had thought through was that they only supported one (and only one) version at any point in time. But they didn’t leave their customers high and dry, they had proven tools that migrated between versions which made it easy for their customers to migrate.

Business Models for Open Source Software Companies, (Tony Wasserman, Executive Director, Carnegie Mellon West Center for Open Source Investigation )

According to Tony, some users of open source s/w, especially companies, WANT to pay and EXPECT to pay for use of software. Tony is the person behind OpenBRR, about which organization I had written in my blog “What does business readiness of software really mean?”

According to him the business models that have worked for OSS are (it would take much space to explain them all but you should get a good idea from the companies mentioned). Let me know if you think there are other business models as well out there

Subscriptions - Red Hat
Cross selling (open + commercial products) - Collabnet with Subversion, Borland with Kylix and SugarCRM
Support and Training – a number of small players here
Dual license - mySQL
Hosted service built on open source – Google
Packaging model - SpikeSource , OpenLogic, The 451 group
Commercial Enhancement model – EnterpriseDB, SRA OSS
Consulting model -Gartner (Open Source Summit), IBM Global services, Accenture
Patronage model - IBM, Sun, Microsoft (yes – he put that in himself, it isn’t from me!)
Reseller model
(and suggested by yours truly, drum roll please…) Hardware embedded model –Linksys, Infoblox

An interesting tidbit about paid “volunteers” vs “pure” unpaid volunteers in Open Source was revealed by Tony who said “In the top 100 OSS projects paid volunteers FAR OUTNUMBER the pure volunteers”.

My only regret is that I couldn’t clone myself and be at multiple sessions at the same time! Wish I could have come earlier and stayed longer!