Port 25: The Open Source Community at Microsoft : Hank Janssen, Open Source

Another Great Step Forward for Hyper-V, Red Hat Certification

hjanssen — Wed, 07 Oct 2009 17:06:00 GMT

Hello again! It has been a pretty busy couple of months for us, and I wanted to give you an update on what we've been doing.

We just completed the first step in another major milestone for Hyper-V. As you can read from Mike Neil's Blog, Microsoft and Red Hat just completed certification in each other's virtualization program.

This means that customers now can deploy Microsoft Windows Server and Red Hat Enterprise Linux and a range of select applications, virtualized on Red Hat and Microsoft hypervisor virtualization software, knowing that the solutions will be supported by both companies.

This again demonstrates Microsoft's commitment to its virtualization platform, and the Hyper-V team's continuous investment in interoperability and heterogeneity through the datacenter.

And I am very proud that my team has been working very hard on getting the Red Hat certification completed.

But, for my group, this is just the first step: we worked on and got certified for Hyper-V running Red Hat in emulated mode and, now, the next step for us is to get certification for enlightened mode, the mode where the guest OS is Hyper-V aware,and can thus access Hyper-V functionality directly.

So, what is needed for enlightened mode? Well, to get that step, we needed to get the Linux Integration Drivers submitted to the Linux kernel, and then we needed those drivers officially accepted in a mainline kernel.

This has now happened as well, as we have been accepted into the mainline kernel. We are in Linux Kernel release 2.6.32, and that release is currently going through development and testing. Once that one is final and officially released, we can take the next step, which is to get those official Linux Integration Component drivers certified with Red Hat.

The timeline for that is not completely set in stone right now, and I do not know if there is an official expected release date for 2.6.32 as yet. But we should be able to move forward with the enlightened mode certification soon after the 2.6.32 final release.

And we are, of course, continuing to work on the Linux Integration components, adding new features and all that good stuff. But I will write more about those items in the near future.

Introduction to the Linux Integration Components

hjanssen — Wed, 22 Jul 2009 19:02:00 GMT

Well, here is blog number two. The initial shock has worn off a bit I hope.

The feedback I have received so far has been pretty positive. This really all started in October of 2008 in a meeting with Mike Neil (GM of Hyper-V) and Tom and myself from the Open Source Technology Center (OSTC) at Microsoft.

In that meeting I proposed to Open Source the Linux Integration Components and contribute them to the Linux Kernel. And, secondly, to have the OSTC continue contributing to these IC's after they made it to the Linux Kernel. Well after some discussion, we all agreed that this was the right thing to do.

And so the whole process started inside of Microsoft. Hey, what can I say, we like to push the envelope a bit here at the OSTC, and we have a reputation to uphold!

Before I go on, I again wanted to thank the Kernel community (specifically Greg Kroah-Hartman) in helping us with explaining and guiding us through community process. It gave us a very nice jumpstart to get all of this going, and provided the groundwork for a good working relationship with the community.

I have also seen a few patches already submitted by community members, which is excellent! (Moritz Muehlenhoff gets major kudos for the first community contributed patch ) I will start submitting patches myself next week once the initial submission has stabilized a bit.

It is my plan to use the kernel as my primary development area, and of course I will continue to provide Greg with my patches. My first step is to clean up the code to make sure it fulfills all Kernel coding standards and requirements.

So, here is blog number two: what are the Linux Integration Components?

1. Overview of Linux VM with ICs

Linux Integration Components(IC) take advantages of the VMBUS and synthetic devices provided in Hyper-V to enhance the performance and usability of Linux guests running on Windows servers.

Figure: Conceptual Architecture overview of Linux guest & Hyper-V. Linux IC modules are painted in yellow color.

Glossary:

VSP: Virtualization Service Provider.

VSC: Virtualization Service Client.

VMBus: Data channel between VSP and VSC.

2. Linux IC modules -- VMBus and VSCs

Communication with parent partition is done through Linux VMBus
VSCs are the Linux drivers for synthetic devices (SCSI, IDE, and Ethernet) provided by Hyper-V.
They translate between Linux I/O requests and Hyper-V VSC commands
Devices are registered with Linux Driver Model (LDM)
Every VSC module contains two portions:
o Driver Interface Mapper (DIM): Released as open source

This portion of the VSC component interacts with the Linux kernel like a regular Linux device driver.

o VSC Core: Released as Open Source

The core portion of the VSC module is implemented based on the protocol of the corresponding VSP at Hyper-V host. The VSC core interacts with VSP via the VMBus interface.

3. Descriptions for each Linux IC module

3.1 VMBus driver (hv_vmbus.c)

The VMBus driver is a Linux kernel module. It provides both a lightweight bus driver and library functionality. As a bus driver, it registers with Linux Driver Model framework (LDM) to provide simple bus and device integration and device tree integration (sysfs). As a library, it implements the VMBus channel protocol and provide an abstraction of channel to its clients (Disk and Network VSCs).

3.2 StorVSC driver (hv_storvsc.c)

The Storage VSC interacts with the Windows Storage VSP. The "wire" protocol defined by the storage VSP determines how a VSC interacts with it. The Linux Storage VSC (LSVSC) basically abstracts the Linux I/O stack from needing to understand the Storage VSP's protocol. At the upper-edge of the LSVSC, it talks to the Linux SCSI subsystem. The Linux SCSI subsystem sees the LSVSC as a SCSI low-level driver (LLD) in Linux parlance. It passes SCSI requests (scsi_cmnd) to LSVSC which in turn converts them into the "wire" format understood by the Windows Storage VSP (VSTOR_PACKET). The bottom-edge of the LSVSC talks to Linux VMBus (LVMBUS) which in turn talks to the Windows VMBus to route the packets to the Storage VSP.

3.3 BlkVSC driver (hv_blkvsc.c)

BlkVSC (BlockVSC) supports "fast boot" and fast access to IDE disks. To enable enlightened IDE support for enhancing the performance of Linux when virtualized on Windows, a separate BlockVSC component is used as a Linux block device driver. Like StorVSC, the BlockVSC component is comprised of an upper edge wrapper that interfaces with the Linux block layer and a lower-edge through the infrastructure modules. The infrastructure modules with Hyper-V through the Linux VMBus.

3.4 NetVSC driver (hv_netvsc.c)

The network VSC send and receive network traffic between a Linux guest and Hyper-V host which has direct connection to physical network. The mechanism that this is used to accomplish is the Remote NDIS (RNDIS) protocol. Thus the communication that flows between the VSP and the VSC primarily happens over the RNDIS protocol which then is packaged and forwarded as payload over to the other side over NetVSP / VMBus protocol.

4. Linux IC's, Location in the Kernel tree

Hopefully you now have a better idea what they are. But where in the kernel tree can you find them?

Well, you can find sources in linux-next tree in /drivers/staging/hv directory.

And the git repository you can find them in right now is:

git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git

Or give this command (assuming your system is set up correctly) to download this repository to your machine:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git <your local name>

Since the IC's are part of the kernel now, we follow the normal community process of getting this all migrated into Linus mainline kernel.

More on the Hyper-V Linux Integration Components

hjanssen — Mon, 20 Jul 2009 20:24:00 GMT

Well, there is no easy way to say this, so I am simply going to start this blog with the following line.

Microsoft just submitted source code for the Hyper-V Linux Integration Components to the Linux Kernel Community Under GPL v2.

Well, there's a conversation starter! Are you still all sitting in your chairs???

Let me summarize:

Yes, our device driver code was released directly to the Linux Kernel
We released the code under GPL v2
We are working with Greg Kroah-Hartman so it is ready for the next release of the Linux Kernel, version 2.6.32
We will continue to update the driver code to enhance interoperability on an ongoing basis, but it's our hope that other developers in the community will find the code useful and worthy of collaboration.

Fallen off your chair yet?

Microsoft developed the Linux device drivers to enhance the performance of Linux when virtualized on Windows Server 2008 Hyper-V. My team and I were responsible for testing and validating the driver components that were contributed for this first release.

Now, my team and I will be responsible for further developing this code going forward. (Yes, that does mean that I have gone back to leverage my very early roots as a Kernel programmer. Let the world be warned!!!!). Haiyang Zhang has been working on this code with me, and he will continue to work with me on this going forward.

When I joined Microsoft three years ago, the primary reason was to put my money where my mouth was. You see complaining about something is easy, but it becomes a little more complicated when somebody offers you the opportunity to be part of helping change what you have complained about.

So, three years after taking the job that made me put my money where my mouth was (and still often is!), I for one am EXTREMELY happy to see one of the most significant fruits of our work here in the Microsoft Open Source Technology Center (OSTC). But I have to say, even I would have been hard-pressed to think three years ago that we would consider contributing to the Linux Kernel.

As you know, two years ago Microsoft announced a partnership with Novell, and Tom Hanrahan ran the lab on a day to day basis till about 9 months ago. Since then I have had the pleasure of running the technical side of the execution of that lab under Tom Hanrahan for the OSTC. One of the primary tasks for that lab is to make sure Windows runs well on top of XEN and Linux runs well on top of Hyper-V, and we do this in very close cooperation with Novell.

We do most of this work as an extension to Mike Neil's Hyper-V team.

As part of this, we were asked to help develop and maintain a crucial part of this work called the Linux Integration Components. This code is designed so that Linux can run in an "enlightened mode" on top of Hyper-V (enlightened mode is roughly the Hyper-V equivalent of "paravirtualized mode" for the Xen hypervisor). Without this driver code, Linux can run on top of Windows, but without the same high performance levels. It is this device driver code that we are releasing today, directly to the Linux Kernel.

We're not talking a few hundred lines of code here; we're talking about roughly 20,000 lines of code.

Is this a Dump and Run from Microsoft? Absolutely not! We plan to enhance the functionality of this code, and we will continue to work with the Linux Community to support the drivers and to ensure continued interoperability.

As you can imagine, this was the result of a lot of hard work: Hiyang Zhang, who has been co-writing this code; Hashir Abdi, who has been testing all this stuff; as well as Vijay Tewari and Mike Sterling from the Hyper-V team who have been taking care of the Hyper-V side.

And last, but certainly not least, Greg Kroah-Hartman, who has been helping me to make all this code land in the right area in the kernel. He has patiently worked to help me correct my obvious mistakes and to get the code contributed into the kernel.

So where are we today? Well, Greg Kroah-Hartman will make the code visible to the outside world today. (For those who want to get a head start, the code will sit under <your kernel tree>/drivers/staging/hv). After it becomes visible, I will write a few more blogs this week that should help you to understand, build and run them.

The titles I am thinking for these blogs are:

Where do the Linux ICs reside in the kernel tree and how do I build them?

And

How do I install, configure and run the Linux IC's?

I had almost forgotten how wrapped up you can be once you start writing code again. So I have not gotten much sleep this past week, but it has been a joy to get back into coding again!

PHP|Tek in Chicago

hjanssen — Thu, 28 May 2009 00:29:00 GMT

Last week I got the perfect excuse to get out of the Planning and Budget process that we are going through right now, attending PHP|Tek, which was a welcome escape as planning and budgeting in any company is usually enough fun to make a grown man cry!

So last week I went to PHP|Tek in Chicago to speak and meet folk from the PHP community. As always, I greatly enjoy meeting the people who write and use PHP, and I have been to and spoken with enough of the speakers at past events that I know a lot of the core people by first name.

Kind of funny that we now have gotten to the point inside of Microsoft that we are almost old hats at Open Source conferences

There were two days prior to the conference where a group of core PHP developers and community people talked about the state - past, present and future - of PHP. It was super cool to be invited to that one!

Unfortunately I was only able to join one of those two days: amazing that flying from Seattle to Chicago takes the better part of a day!

The discussions there where very wide ranging, from whether there will be a PHP 5.4, what 6.0 will bring, which bugs are current show stoppers, where PDO is going, etc. etc.

For me PHP|Tek remains a very nice ‘community' conference, where the focus is on the community of PHP and not the business/vendors of PHP.

These kinds of conferences are the best way to network, and it would take too long to talk about all the people I spoke to. But Elizabeth Smith and I talked about us writing documentation for php.net (I have been wanting to write the ‘how to build PHP for Windows' part) so hopefully look for more documentation written by Microsoft for php.net soon.

As always I talked to a lot of the usual suspects: Scott MacVicar, Andrei Zmievski, Derick Rethans, Sebastian Bergmann, Chris Shiflett, Cal Evans and others.

Oh, and if you are really bored, check out the latest May issue of php architect, which has a bunch of really cool articles about PHP and Windows. Some of them were even co-written by me, which gives you an idea how far php | architect has sunk to have people write articles for them

I just checked out the Website, and the May issue is not posted yet. But everybody who attended PHP|Tek got a copy of that issue in their goodies bag.

I always enjoy giving sessions and the session I did give at PHP|Tek was ‘PHP 5.3 The best PHP on Windows Yet' , and I got some really good feedback. I think I had about 40+ people in my session. People are always surprised to see Microsoft's involvement with PHP and what we have done with the community so far.

It is a talk I have given before. It starts with describing what the organization I belong to (the Microsoft Open Source Technology Center) does and how we work inside of Microsoft. After that I go into some detail about why PHP 5.3 is the best PHP on Windows.

Did you know that, for example, with PHP releases prior to 5.3, the code was build with libraries that were more than 10 years old and for which nobody really had any idea where the source code went? So it was built - linked rather - with object files that were more than 10 years old.

It makes it really hard to fix/improve stuff that you do not have the source code for

Well, pretty much all the issues of the past are now gone. I will make sure I write a blog about what truly went into PHP 5.3 for Windows soon, if the budgeting and planning process doesn't kill me before that point. In the meantime, here is a link to phpfreaks where, a few weeks ago, I posted a bunch of what we have been doing.

One really interesting thing is that there were a lot of Microsoft people at this conference, specifically from the DPE (Developer Platform Evangelism) side of Microsoft.

These are the people who are very much field and customer focused. From my conversations with them, they enjoyed the conference and were glad to get the opportunity to speak with a lot of the OS crowd. It is amazing how much we all have in common once we talk about technology.

Thanks to the people who put on the conference: of course Marco Tabini, the man behind PHP|Tek, but especially Elizabeth Naramore, who is the unsung hero that is the real driver behind keeping PHP|Tek running smoothly!

On the Road in Europe - Take 1

hjanssen — Tue, 28 Oct 2008 16:25:00 GMT

Today is Tuesday - That must mean I am in Mainz. I am on day 12 of my European trip. I was in Rome and Amsterdam last week.

In Rome I attended the Moodle conference, which was pretty cool. It was put on by Roma Tre and was one of many destinations in which Moodle held conferences this past month. I went to talk to Martin Dougiamas, Helen Foster and Petr Skoda as part of our ongoing quest to get PHP on Windows to be the best experience possible.

The deployment numbers that Martin showed in his presentation are quite impressive! I have been digging around for his presentation to give these numbers, but I can't find them. I am sure that Martin must have posted his presentation somewhere, I just have not found it yet.

But the intention was to start a working relationship with the Moodle community, and this was a great start. One of the most interesting presentations was from the Italian Airforce, who described their experiences in finding better educational tools to train their personnel, and settled on Moodle to be a large part of that. It is always interesting to have a General in the audience.

It is really amazing to see how and where Moodle is used. It is a testament to the intention of Moodle and Martin and the Moodle community that is has become so popular.

The other thing I did was meet with a lot of open source influential's/Government/CTO/Journalists etc. Microsoft Rome asked me to give a bunch of presentations and interviews, which I love to do, but it turned into a 17 hour-long gauntlet. And I just want to go on record and say that I cannot be held accountable for what I talked about the last 4 or so hours of that day. It became a little blurry at that point.

The bad thing about these trips is that I am away from home for a long period of time. The really good thing about doing these trips is that I get to meet so many people. It is really cool to see the faces and have the dialogues when I talk about what Microsoft is doing in the OS world. By far it is very positive.

But I get the biggest bang for the buck when we have discussions on what people do, and want, from the OS and from Microsoft. We have been doing more and more in the OS world, but unless we work closely with the community we have no idea if we are on the right track.

One of the questions I asked in Rome in a meeting with government officials and OS influential's was ‘what does open source mean to you?' There were many responses, and most of them followed the same line. Some of the common responses where: sharing knowledge, collaboration, personal recognition, information that is easy to get to, allocation of rights, intend to make communications possible.

Well, for the next few days I will be at the IPC in Mainz. So I will blog more tomorrow. There are a lot of things we are doing right now, so I have a lot of content

PHP on Windows

hjanssen — Tue, 04 Mar 2008 13:50:00 GMT

I have this really odd feeling of Dejavu. The last blog I wrote (here), I started by saying that I have been very delinquent with writing blogs. So what do I do? I continue to be very delinquent with my blogs!

If anything, I am consistent! :)

So, why this sudden re-emergence of myself on port25? (Besides the fact that I just realized that it has been months since I last blogged, that is) Well we have been keeping ourselves very busy in the last few months. A lot of that work relates directly to PHP, so I wanted to talk about some of these efforts.

We have significantly increased our work in this area. And my group continues to find itself at the middle of pretty much all of these efforts. The SQL Server driver for PHP is now in its second release (Get the latest bits here). And we continue to make improvements and enhancements going forward to this driver.

We have also been working very closely with the PHP community on improving PHP performance on Windows. This is an effort that will be ongoing and probably accelerating in the months to come.

With the release of Windows Server 2008, I wanted to take a moment and highlight some of the things that have been done to make Windows an excellent PHP Deployment platform.

First of all, Microsoft late last year released FastCGI for IIS6 and IIS7. IIS7 is integrated with Windows Server. IIS7 has as part of its deployment FastCGI included (not an optional download as is the case with IIS6) this in effect means that with IIS7, Microsoft has added out of the box Microsoft-supported software designed to run PHP on IIS.

Equally as interesting, we have been working with Zend to help them certify Zend Core for Windows Server 2008. I think this makes it one of the first PHP products to be certified for WS2008. Which makes Windows PHP ready :). Imagine that, PHP running on Windows Server, fully certified!

To this end we will continue to work with Zend very closely to continue to improve PHP on Windows.

Secondarily (although not directly related to Windows Server) we have also been working with Zend to provide Cardspace functionality in the Zend Framework. You can get it here, and read more about how to use it at the Zend website here. Which is another way in which Microsoft has added support to PHP for Microsoft technologies.

This continues the close relationship with Zend into the future, and I am expecting to work with them on a whole host of other PHP related efforts going forward.

As you could read in earlier blogs by Garrett Serrack we hosted the Apache Software Foundation guys here in Redmond last week. So look for a blog from me later this week on the wrap-up of that visit.

Firefox Explorer (Or: Mozilla is Here)

MichaelF — Wed, 04 Oct 2006 11:47:00 GMT

So, as people are finding out, we at OSSL are doing things a little differently. One of those things Sam Ramji’s invite to the Mozilla guys to come up and help them making their products work on Vista better.

This was met with great criticism (Slashdot for example had people make all kinds of wild assumptions that started with suggestions all Mozilla folks flying on different planes to MS offering everybody jobs. And everything in between) and surprise.

Well, I am glad to report that the Mozilla guys are here. I have spent a really cool day in all kinds of meetings with the following guys from Mozilla; Robert Strong, Mike Schroepfer, Doug Turner, Vladimir Vukicevic, Scott Macgregor. (If I slaughtered anybody’s name I apologize for it in advance!!)

They are meeting with all kinds of people here, and it seems that both sides are learning greatly from each other. One thing that I noticed right away is that both the MS side and the Firefox side just love the technology and are hitting it off nicely.

The things we talked about today for example have been;

Cardspace; Mike Jones and Garrett Serack gave an overview of cardspace. Cardspace is a carrier of identity tokens. But you can read about it here as a nice starting point http://blogs.msdn.com/garretts/ . We spend some time talking about what it is providing, and that it is an effort from the industry to provide a way to provide identity management. Vista will include it out of the box, and .Net 3.0 will include the cardspace technology. Which will make it run on XP and 2003.

And we are currently looking to writing a plugin to make cardspace work inside of Firefox. (Like I said, we do things differently here at OSSL )

IE Lower Integrity Marked: IE will be running in a Low Integrity Marked setting, resulting in a greater security environment IE will run in. But it limits a whole bunch of places it can write to. (which is good) We talked with the IE people to the benefits and pitfalls of doing this, Mozilla is interested in checking into doing this for Firefox as well.

MS WPFE: the developers from the plugin had a conversation to make it work inside Firefox better. (I am not an expert on this, but this should give a better understanding of what it is; http://msdn.microsoft.com/winfx/reference/presentation/default.aspx )

MSI: MSI developers talked about the changes in and advantages of using the new MSI way of installing software in Vista. One of the things that came up was that some vendor have problems with their plugins (xpi files) such as Adobe. (Problem relating to global plugins) XPI can only be installed per user in Vista. I will talk more about this in a later blog

Windows Vista Shell: we went through a high level overview of the Windows shell in Vista.

If you want to know what the Mozilla folks thought of yesterday’s meeting, check out Vladimir’s blog http://blog.vlad1.com/

Well, I wanted to keep this short. But still let you all know what is going on. There is another day with them coming up tomorrow.

Btw, the chips have been implemented into the brains of all the Mozilla folks that have shown up here. And it looks like from Firefox 2.0 on it will be released on Windows exclusively.

Hank.

OPEN/SHARED SOURCE AT Microsoft

MichaelF — Tue, 12 Sep 2006 21:25:00 GMT

It has been a while since I posted a blog, and I really have no other excuse than that I have been very busy. I have had a whole bunch of blog ideas percolating in the back of my mind, and I will be writing them down soon.

When we started port25 and the OSSL it was met with great skepticism. But there have been a lot of changes going on around us here at Microsoft. And one of those I wanted to bring to your attention.

A few years ago the mere thought of Open Source at Microsoft was ridiculed both inside and outside of the company. But I am starting to see small and sometimes not so small changes. This blog describes a very positive change.

As you might all know, I went to the 2006 OSCON conference in Portland. And there I met another Microsoft employee, Sara Ford. She works in the Visual Studio and Power Toys area. She has been a very active blogger in the past (unlike myself, working on it though!) And you can find her page here.

We got to talking at the conference and I have worked with her a little since then and found her to be a very energetic person greatly interested in OSS. But why is this interesting??? Well she attended a session at OSCON given by James Howison. (See his OSCON session info here ) And his presentation was on open source communities.

She was so impressed by it that she is currently working on Open Sourcing the Power Toys. I had the pleasure to sit in the training she gave the team, you can see more of the training she gave (unfortunately I was there as well and probably messed up the whole video by opening my mouth. So ignore me!) here.

In any case, who would ever thought Microsoft would open sourcing anything. But it is happening, and in future blogs I will give you all more insight on my first 4 or so months here and the changes I am seeing both internal and external.

Till then!

Hank.