Port 25: The Open Source Community at Microsoft : Open Source, Dynamic Languages

Microsoft to Open Source the .NET Micro Framework

Peter Galli — Mon, 16 Nov 2009 15:16:00 GMT

I have great news to announce. Today, at the Microsoft Professional Developer Conference (PDC) here in Los Angeles, we announced not only the release of version 4.0 of the.NET Micro Framework, but also that we are open sourcing the product and making it available under the Apache 2.0 license, which is already being used by the community within the embedded space.

The .NET Micro Framework,a development and execution environment for resource-constrained devices, was initially developed inside the Microsoft Startup Business Accelerator, but recently moved to the Developer Division so as to be more closely aligned with the overall direction of Microsoft development efforts.

The result of this is that the .NET Micro Framework has become a seamless development experience, bringing a single programming model and tool chain for the breadth of developer solutions, all the way from small intelligent devices, to servers and the cloud. There are also no more time-limited versions.

Including the source code for almost all of the product also ensures that developers now also get access to the Base Class Libraries that were implemented for .NET Micro Framework and the CLR code itself.

However, both the TCP/IP stack and Cryptography libraries are not included in the source code. Program Manager Colin Miller told me this was because the TCP/IP stack is third party software that Microsoft licenses from EBSNet, so we do not have the rights to distribute that source code. If someone needs to access the source code for the TCP/IP stack, they can contact EBSNet directly.

As for the Cyptography libraries, they are not included in source code because they are used outside of the scope of the .NET Micro Framework. Customers who need to have access to the code in the cryptography functions will find that these libraries can be replaced, Miller said.

I asked Miller what the future plans for the .Net Micro Framework were, and he made clear that Microsoft intends to remain actively involved in its ongoing development, working alongside the community. While the license will allow customers to take the code and make specialized versions to fit their needs, customers told us they wanted Microsoft to stay involved to avoid any possible fragmentation of the platform.

"As such, we are planning on establishing a core technology team that is made up of both Microsoft and non-Microsoft contributors that continues the goals of producing a high quality product for very small devices. This group will act as the gateway to community contributions while, at the same time, Microsoft Developers will continue add functionality and coordinate with the overall .NET team," he said.

Microsoft is also in the process of forming a community of interested and involved members to help shape the future direction of the product. There will be a core technology team that is composed of Microsoft and external partners, and people will be encouraged to propose projects, which will be vetted before they are accepted.

"The site will also support people building extensions that exist alongside the platform rather than being integrated into it, " Miller told me.

The ECMA C# and CLI Standards

Peter Galli — Mon, 06 Jul 2009 17:27:00 GMT

I have some good news to announce: Microsoft will be applying the Community Promise to the ECMA 334 and ECMA 335 specs.

ECMA 334 specifies the form and establishes the interpretation of programs written in the C# programming language, while the ECMA 335 standard defines the Common Language Infrastructure (CLI) in which applications written in multiple high-level languages can be executed in different system environments without the need to rewrite those applications to take into consideration the unique characteristics of those environments.

"The Community Promise is an excellent vehicle and, in this situation, ensures the best balance of interoperability and flexibility for developers," Scott Guthrie, the Corporate Vice President for the .Net Developer Platform, told me July 6.

It is important to note that, under the Community Promise, anyone can freely implement these specifications with their technology, code, and solutions.

You do not need to sign a license agreement, or otherwise communicate to Microsoft how you will implement the specifications.

The Promise applies to developers, distributors, and users of Covered Implementations without regard to the development model that created the implementations, the type of copyright licenses under which it is distributed, or the associated business model.

Under the Community Promise, Microsoft provides assurance that it will not assert its Necessary Claims against anyone who makes, uses, sells, offers for sale, imports, or distributes any Covered Implementation under any type of development or distribution model, including open-source licensing models such as the LGPL or GPL.

You can find the terms of the Microsoft Community Promise here.

I told you this was good news!

Crafting a Better PHP Build Process on Windows – Part IV

Garrett Serack — Wed, 24 Jun 2009 00:12:00 GMT

In the previous post, I discussed what it took to use PGO on the Windows PHP build. That led to me building automated build scripts...

Automation as the root of all evil

"Anything that can be done for you, automatically, can be done to you, automatically." - David C. Wyland

First, I had to get the entire dependency stack into the mix. While some of the dependent libraries had VCProject files, some didn't. Worse, even if they had them, you couldn't tell with a degree of certainty that they were compiled with the same settings which would enable them to take advantage of PGO optimization.

I began taking each project, updating (or creating, using the Trace and mkProject tools) the Visual C++ project files that would use the same settings as the rest, and eventually came up with a solution file that had 74 projects in it - some of the projects generated more than one binary.

Next, I had to actually automate the process of creating the vcproject files. Once you've got the right dependencies, the PHP build process cranks out over 30 binaries when you include the PHP extensions that get built as part of the core.

After what seemed like a million compile-verify-tweak iterations, I had the tools that could generate VCProject files for the core PHP and all the extensions, provided it was all in the right place.

Next I wrote a .cmd batch script that went step-by-step, checking out the source, compiling the dependent libraries, building the PHP makefile, compiling PHP like the community did - and logging what it was doing, then switching to instrumentation, rebuilding the dependencies again, building the stack, PGO training it with test data and some applications (Wordpress, MediaWiki and phpBB) and then relinking it with optimization.

I got the .cmd script almost working, but it was fairly fragile. At that point I decided to switch batch scripting strategies and, in about a week, rewrote the batch script in JScript, which was far more flexible, and a lot more reliable.

What's next...

"The future always arrives too fast... and in the wrong order." - Alvin Toffler

During this process, I tweaked the build process that is generated quite a bit, adding in a few more applications to the PGO training, which cranks the performance up more and more.

Now, I can add in more scripts to assist with the training pretty trivially, but it still takes some effort to package up an entire application like MediaWiki or Wordpress and include it into the build process.

Even once I've added in an application, I end up doing a whole slew of comparative testing to see what impact it has on the final executables.

As time goes by, I'm sure there will be more tweaking to be done but, in all likelihood, any significant performance gains are going to be the result some modification of the PHP codebase itself.

Crafting a Better PHP Build Process on Windows – Part II

Garrett Serack — Thu, 11 Jun 2009 20:10:00 GMT

I talked about getting started in building the PHP stack in my last post, now I'm taking it...

One step further

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." - Donald Knuth

A chance conversation I had last summer at OSCON with Trent Nelson - who was building Python on Windows - planted the seeds of how to get PHP on Windows optimized further. Trent was using the PGO features of Visual Studio to generate Python binaries that run faster.

Rather than spend a lot of time optimizing all the little bits of PHP itself, I thought that this would be an ideal way to improve the overall speed of PHP, provided I could find the right scenarios to train PHP with. Little did I know that finding the right scenarios wasn't the hardest part.

What is PGO? (from Wikipedia)
Profile-guided optimization (PGO) is a compiler optimization technique in computer programming to improve program runtime performance. In contrast to traditional optimization techniques that solely use the source code, PGO uses the results of test runs of the instrumented program to optimize the final generated code. The compiler is used to access data from a sample run of the program across a representative input set. The data indicates which areas of the program are executed more frequently, and which areas are executed less frequently. All optimizations benefit from profile-guided feedback because they are less reliant on heuristics when making compilation decisions.

Adding PGO to the existing build process

"I have not failed, I've just found 10,000 ways that won't work." - Thomas Edison

I had downloaded the source to the dependent libraries off the PHP wiki, checked out the PHP source code, and begun the process of adding in PGO support to the existing build process. This proved to be extremely difficult.

Even limiting the scope to just the core of PHP itself - without the dependent libraries - I ran into trouble trying to compile using PGO instrumentation and then re-linking after running some tests. The make file that gets generated by the configure.js script (a JScript version of the automake configure script for the Windows platform) was just not built with what I had in mind.

I spent the better part of two weeks trying different approaches to tweaking the makefile so that I could use PGO to improve the PHP executable, but I kept running into roadblocks. Worse, the closer I got to a makefile that did what I wanted, the farther away from the current build process I was getting, and I wasn't sure that what I would end up with would even be close to what was being built today.

The long dark winter road

"Only the meek get pinched. The bold survive." - Ferris Bueller

I came to the conclusion that I'd have to build new Visual Studio project files from scratch. What worried me was that this would end up to be a completely different build process, and I'd never get the community to abandon what was already working, so I'd better be able to rebuild these new project files easily.

I started looking (inside Microsoft and out) for any tools which generated Visual C++ project files. I found someone internally who had used some JScript to create project files from text files, but after some experimentation, I found this was nowhere near what I needed. What I really needed was a way to convert the generated Makefile into a .vcproj file-and not just 'wrap' it.

Once I found there was no such tool* , I began trying to figure out how to create one. I had this idea a few times in the last decade or so: watch how a program is compiled, and create a project file that does the same thing. Having tossed around the idea in my head before, I knew it wasn't going to be trivial but, without it, I couldn't do what needed to be done.

* Let me tell you: you never want to think about writing a tool to parse out what a makefile does. It's rather like making a tool that tells you how sausage is made, in excruciating detail. Ugh.

In Part III, I'll talk about the trouble with observing the build process.

Crafting a Better PHP Build Process on Windows – Part I

Garrett Serack — Tue, 09 Jun 2009 18:22:00 GMT

The last several months, I've been working very deeply with PHP - specifically, compiling the PHP core itself, and looking for avenues for optimization. This is the first of four posts about the journey I've been on with PHP.

I get started building PHP

"It is a bad plan that admits of no modification" - Publilius Syrus

I started working with building PHP itself about a year ago. Initially, I was trying to put together an environment to compile up the PHP stack so that I could do some debugging, and track down a few faults that we were encountering in some of the PHP applications that we were trying to modify to use the SQL Server PHP driver that the SQL Server team here at Microsoft was creating.

Once I began to work with the source code, I found out very quickly that on top of having a hard time recreating the exact same binaries that the community build process generated, there were a large number of dependent libraries that were available in binary-only form and which were kept in a zip file that was passed around from developer to developer. That seemed a little odd for an open-source project, but I can certainly understand that over time, unless someone is working hard to keep it all together, these things happen.

Around the same time, the community had started to invest time and effort to 'clean up' the dependencies for building PHP on Windows, and move towards supporting VC9 (Visual Studio 2008) as an officially supported compiler.

In order to help in this process, I built out some testing environments in our Lab, which would let me compile up PHP on Windows and Linux, in order to get decent and reliable test results which we could use to identify any shortcomings that we could then address. This includes benchmarking not just the core PHP executable, but replicable and comparable testing of PHP applications such as Wordpress, MediaWiki, Gallery and phpBB.

PHP 5.3 on Windows: Not your father's PHP

"I'm looking for a lot of men who have an infinite capacity to not know what can't be done." - Henry Ford

For PHP 5.3, Pierre (and others) had gone out and found up-to-date versions of all the dependencies, brought them together, and managed to get them compiling with VC6 and VC9. They had posted these in binary and source form to the PHP Windows Internals site, which allows anyone to rebuild the PHP stack on Windows and, theoretically, get the same results as the 'official' build.

Jumping in at that point was much easier than it had been, as all you had to do was download the binaries of the libraries, check out the source code, run a few commands at the command line and, presto, you had your PHP executables.

At this point Pierre and I played around with the build flags on VC9 and found some settings that gave some pretty significant improvements to the speed of PHP vs. the speed of the VC6 version -and a lot of speed improvements vs. the old 5.2x line of PHP.

In Part II, I'll talk about going one step further with optimization.

PHP|Tek in Chicago

hjanssen — Thu, 28 May 2009 00:29:00 GMT

Last week I got the perfect excuse to get out of the Planning and Budget process that we are going through right now, attending PHP|Tek, which was a welcome escape as planning and budgeting in any company is usually enough fun to make a grown man cry!

So last week I went to PHP|Tek in Chicago to speak and meet folk from the PHP community. As always, I greatly enjoy meeting the people who write and use PHP, and I have been to and spoken with enough of the speakers at past events that I know a lot of the core people by first name.

Kind of funny that we now have gotten to the point inside of Microsoft that we are almost old hats at Open Source conferences

There were two days prior to the conference where a group of core PHP developers and community people talked about the state - past, present and future - of PHP. It was super cool to be invited to that one!

Unfortunately I was only able to join one of those two days: amazing that flying from Seattle to Chicago takes the better part of a day!

The discussions there where very wide ranging, from whether there will be a PHP 5.4, what 6.0 will bring, which bugs are current show stoppers, where PDO is going, etc. etc.

For me PHP|Tek remains a very nice ‘community' conference, where the focus is on the community of PHP and not the business/vendors of PHP.

These kinds of conferences are the best way to network, and it would take too long to talk about all the people I spoke to. But Elizabeth Smith and I talked about us writing documentation for php.net (I have been wanting to write the ‘how to build PHP for Windows' part) so hopefully look for more documentation written by Microsoft for php.net soon.

As always I talked to a lot of the usual suspects: Scott MacVicar, Andrei Zmievski, Derick Rethans, Sebastian Bergmann, Chris Shiflett, Cal Evans and others.

Oh, and if you are really bored, check out the latest May issue of php architect, which has a bunch of really cool articles about PHP and Windows. Some of them were even co-written by me, which gives you an idea how far php | architect has sunk to have people write articles for them

I just checked out the Website, and the May issue is not posted yet. But everybody who attended PHP|Tek got a copy of that issue in their goodies bag.

I always enjoy giving sessions and the session I did give at PHP|Tek was ‘PHP 5.3 The best PHP on Windows Yet' , and I got some really good feedback. I think I had about 40+ people in my session. People are always surprised to see Microsoft's involvement with PHP and what we have done with the community so far.

It is a talk I have given before. It starts with describing what the organization I belong to (the Microsoft Open Source Technology Center) does and how we work inside of Microsoft. After that I go into some detail about why PHP 5.3 is the best PHP on Windows.

Did you know that, for example, with PHP releases prior to 5.3, the code was build with libraries that were more than 10 years old and for which nobody really had any idea where the source code went? So it was built - linked rather - with object files that were more than 10 years old.

It makes it really hard to fix/improve stuff that you do not have the source code for

Well, pretty much all the issues of the past are now gone. I will make sure I write a blog about what truly went into PHP 5.3 for Windows soon, if the budgeting and planning process doesn't kill me before that point. In the meantime, here is a link to phpfreaks where, a few weeks ago, I posted a bunch of what we have been doing.

One really interesting thing is that there were a lot of Microsoft people at this conference, specifically from the DPE (Developer Platform Evangelism) side of Microsoft.

These are the people who are very much field and customer focused. From my conversations with them, they enjoyed the conference and were glad to get the opportunity to speak with a lot of the OS crowd. It is amazing how much we all have in common once we talk about technology.

Thanks to the people who put on the conference: of course Marco Tabini, the man behind PHP|Tek, but especially Elizabeth Naramore, who is the unsung hero that is the real driver behind keeping PHP|Tek running smoothly!

The Hidden Technology Decision-Maker

Mark Stone — Wed, 25 Mar 2009 15:00:00 GMT

On Monday Microsoft and SD Forum held the 3rd annual Open Source ISV Forum. In a day of interesting talks, I was particularly struck by Larry Augustin's talk. As an emphasizing example of the growth of open source projects partnered with a commercial endeavor, Larry mentioned DotNetNuke.

DotNetNuke is, of course, freely available for download and licensed under an open source license. But there is also a professional edition and a range of complimentary commercial services for those who want service, support, or customizations. The business model is classic open source: the free download seeds the market with potential customers, and as some of those run up against the limits of what they are willing to do on their own, they make inquiries about the professional edition. Thus open source creates an inbound channel of qualified sales leads, without the overhead and expense of a sales force working in the field.

This is a great business model, but it's important to think about the actual decision-makers in this adoption process. What's distinctive about DotNetNuke is that it's one of the few mature, open source Content Management Systems (CMSs) that is not written in PHP. It is, as the name implies, ASP.NET based. And this reminds me of a prdocut management position I was in some years ago when (a) the only mature CMS choices were PHP-based, and (b) PHP on Windows was not yet a viable alternative.

The company in question was in a typical position: they were not a technology company, but needed a strong web presence for their business and to connect to the online community of their customers. They had a home-grown CMS solution that wasn't scaling, wasn't secure, and wasn't stable. My product management team put together a good comparison chart of various CMS choices, many of them open source. I sat down with the web development manager to review the choices.

"Some of these are good systems," he said, "but I have a team of .NET developers, and these are all PHP-based. I don't have head count to go out and hire a PHP dev for this project."

Note the decision-making process at work here:

From an agreed list of candidate software, an engineering team will download something to "test drive";
One developer will do some testing and make a recommendation;
From a short list of recommended software a more thorough test will be done with a prototype or pilot project;
Finally a choice will be made, and money will be spent.

The person with the first vote in the process is not a CIO or any other traditional "IT Decision Maker". It's one dev, talking to his manager. If your software gets vetoed at that level, or -- worse -- never even gets a try-out, then your product isn't in the decision-making process.

A lot has changed in recent years. PHP is now much better supported on Windows, and .NET projects like DotNetNuke are available and much more mature. This is as it should be. For open source to spread to its full potential, it has to be available in the technology adoption decision-making process. And that availability starts with being available to the hidden technology decision-maker: that lone developer who will look at software written in a framework they understand for a platform they work with.

Japanese LAMP Engineers Visit Redmond

Peter Galli — Tue, 10 Mar 2009 21:30:00 GMT

I was fortunate enough to spend last Thursday with a group of LAMP engineers who have some experience with Windows Server and IIS, and who are based in Japan.

The three - Kimio Tanaka, the president of Museum IN Cloud; Junpei Hosoda, the president of Yokohama System Development; and Hajime Taira, with Hewlett-Packard Japan - won a competition organized by impress IT and designed to get competitive LAMP engineers to increase the volume of technical information around PHP/IIS and application compatibility. The competition was titled "Install Maniax 2008".

A total of 100 engineers were chosen to compete and seeded with Dell server hardware and the Windows Web Server 2008 operating system. They were then required to deploy Windows Server/IIS and make the Web Server accessible from the Internet. They also had to run popular PHP/Perl applications on IIS and publish technical documentation on how to configure those applications to run on IIS.

The three winners were chosen based on the number of ported applications on IIS, with the prize being a trip to Redmond. A total of 71 applications out of the targeted 75 were ported onto IIS, of which 47 were newly ported to IIS, and related new "how to" documents were published to the Internet. Some 24 applications were also ported onto IIS based on existing "how to" documents.

The first-place winner Kimio Tanaka managed to port 71 applications onto a single IIS server. His technical documents can be found here.

Kentaro Yoshikawa, the Platform Strategy CSI Lead for Microsoft Japan, put the competition together and brought the winners to Redmond, where we arranged for them to meet with folk from the Windows Azure, Windows Server and IIS development teams. They also spent time with Sam Ramji, the Senior Director for Platform Strategy, as well as with Tom Hanrahan and Hank Janssen of the Open Source Technology Center.

Kentaro told me that the three were really impressed by the depth of the discussions they had during the day, which was useful to them as they have, until now, mostly lived outside of the traditional Microsoft ecosystem.

They also appreciated the depth of technical thought, strategy and commitment to open source communities that exists within not only the Platform Strategy group, but across Microsoft.

Welcome Snakebite, the Newest Open Network in Town!

Peter Galli — Tue, 10 Feb 2009 13:38:00 GMT

There's a new Open Network in town, known as Snakebite, which is the brainchild of Trent Nelson, a committer to Python.

Snakebite is a network of some 37 servers of different shapes and sizes, spread over three sites and specifically geared towards the needs of open source projects like Python.

In short, and according to its Website, Snakebite is a network to provide open source developers unrestricted access to as many of the different platforms, operating systems, architectures, compilers, devices, databases, tools and applications that they need to optimally develop their software.

"Why do we develop open source software on closed networks? Why do open source developers only have access to a fraction of platforms that their software will eventually run on? And why the *@&# are the Windows buildbots always red?! Snakebite was created out of a desire to try and address problems like these faced by open source projects," the Website says.

Some months back, Nelson realized that while buildbots were fine when everything was running smoothly, nothing compared to actually having access to a system when a developer is trying to debug something.

"So, I thought to myself, why not buy a couple of clunky old boxes off eBay and donate them to the Python Software Foundation, such that all developers had access to them ... Ten months, seven trips to Michigan State University, six blown fuses and about $60,000 later, I'm proud to introduce you all to Snakebite: The Open Network!," he said.

This means that every CPython, Jython, IronPython and PyPy committer will have access to every development server on the network, Nelson says, adding that he has already extended the offer to prominent Python projects like Django and Twisted.

The end-goal is to ultimately invite other open source projects like Apache, Subversion, MySQL and Postgres, among others, but given that this network is Nelson's "gift to All Things Python, first and foremost," Python projects will always get preferential treatment.

Nelson also has big aspirations for Snakebite going forward, which you can read all about in this email he sent to the Snakebite list.

Nelson was also blown away by the level of support he received for the initiative: "Microsoft jumped on board and provided unlimited MSDN licenses in less time than it took me to write them an e-mail asking for stuff. Having the support of Microsoft from very early on has been a huge boost, and the MSDN licenses have already been invaluable ," he said.

An email to HP asking them for a Tru64 license and 2GB of RAM for the Itanium box he bought off eBay, also resulted in the company shipping two quad Itanium 2 RX-5670s, full of 73GB 15k disks and 78GB of RAM between the two servers; 32GB in one and 46GB in the other.

Sun, Google and Canonical have also expressed interest in the project, but Nelson has stopped asking for hardware as they have run out of space to host it all.

But, in a recent post to the Python Committers mailing list, Nelson notes that it will probably be a few weeks before users can start logging in and doing stuff.

"The HPCC/CSE server room at MSU is about to have walls knocked in and ramps built in order to accommodate a giant PDU that has been sitting outside it for about six months; the Snakebite rack is going to get shuffled around a bit so I figure there's not much point going live before that's taken care of," he said.