Port 25: The Open Source Community at Microsoft : Interop, Dynamic Languages

Roadmap Announced for Outlook Personal Folders Documentation

Peter Galli — Mon, 26 Oct 2009 23:20:00 GMT

Great news on the data portability front: today, Microsoft announced that it will be releasing documentation for the .pst file format - the format in which data is stored in Microsoft Outlook Personal Folders.

Even better is the fact that the documentation will be released under Microsoft's Open Specification Promise when complete. This lets anyone implement the .pst file format on any platform and in any tool, without concerns about patents, and without the need to contact Microsoft in any way.

But it is important for me to point out that this documentation is still in its early stages and the work is ongoing. However, Microsoft is also engaging directly with industry experts and interested customers to gather feedback on the quality of the .pst technical documentation to ensure that it is clear and useful.

As an increasing amount of information is stored and shared in digital formats, it is vital for people to be able to reuse their data across various applications and platforms. Giving access to the documentation will facilitate interoperability and let customers and vendors to access their data in .pst files across a variety of platforms.

This is also just another example of how Microsoft is listening to its customers and responding to their requests for greater interoperability, in this case around data portability.

Also, enabling customers and vendors to access the data in .pst files on a variety of platforms allows developers to read, create, and interoperate with the data in .pst files in server and client scenarios using the programming language and platform of their choice.

"We're excited about the possibilities created for our customers and partners by this kind of effort, and we look forward to continued collaboration with the industry in our pursuit of improved interoperability with Microsoft Office," said Paul Lorimer, the Group Manager for Office Interoperability at Microsoft, in a blog post.

Crafting a Better PHP Build Process on Windows – Part IV

Garrett Serack — Wed, 24 Jun 2009 00:12:00 GMT

In the previous post, I discussed what it took to use PGO on the Windows PHP build. That led to me building automated build scripts...

Automation as the root of all evil

"Anything that can be done for you, automatically, can be done to you, automatically." - David C. Wyland

First, I had to get the entire dependency stack into the mix. While some of the dependent libraries had VCProject files, some didn't. Worse, even if they had them, you couldn't tell with a degree of certainty that they were compiled with the same settings which would enable them to take advantage of PGO optimization.

I began taking each project, updating (or creating, using the Trace and mkProject tools) the Visual C++ project files that would use the same settings as the rest, and eventually came up with a solution file that had 74 projects in it - some of the projects generated more than one binary.

Next, I had to actually automate the process of creating the vcproject files. Once you've got the right dependencies, the PHP build process cranks out over 30 binaries when you include the PHP extensions that get built as part of the core.

After what seemed like a million compile-verify-tweak iterations, I had the tools that could generate VCProject files for the core PHP and all the extensions, provided it was all in the right place.

Next I wrote a .cmd batch script that went step-by-step, checking out the source, compiling the dependent libraries, building the PHP makefile, compiling PHP like the community did - and logging what it was doing, then switching to instrumentation, rebuilding the dependencies again, building the stack, PGO training it with test data and some applications (Wordpress, MediaWiki and phpBB) and then relinking it with optimization.

I got the .cmd script almost working, but it was fairly fragile. At that point I decided to switch batch scripting strategies and, in about a week, rewrote the batch script in JScript, which was far more flexible, and a lot more reliable.

What's next...

"The future always arrives too fast... and in the wrong order." - Alvin Toffler

During this process, I tweaked the build process that is generated quite a bit, adding in a few more applications to the PGO training, which cranks the performance up more and more.

Now, I can add in more scripts to assist with the training pretty trivially, but it still takes some effort to package up an entire application like MediaWiki or Wordpress and include it into the build process.

Even once I've added in an application, I end up doing a whole slew of comparative testing to see what impact it has on the final executables.

As time goes by, I'm sure there will be more tweaking to be done but, in all likelihood, any significant performance gains are going to be the result some modification of the PHP codebase itself.

Crafting a Better PHP Build Process on Windows – Part II

Garrett Serack — Thu, 11 Jun 2009 20:10:00 GMT

I talked about getting started in building the PHP stack in my last post, now I'm taking it...

One step further

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." - Donald Knuth

A chance conversation I had last summer at OSCON with Trent Nelson - who was building Python on Windows - planted the seeds of how to get PHP on Windows optimized further. Trent was using the PGO features of Visual Studio to generate Python binaries that run faster.

Rather than spend a lot of time optimizing all the little bits of PHP itself, I thought that this would be an ideal way to improve the overall speed of PHP, provided I could find the right scenarios to train PHP with. Little did I know that finding the right scenarios wasn't the hardest part.

What is PGO? (from Wikipedia)
Profile-guided optimization (PGO) is a compiler optimization technique in computer programming to improve program runtime performance. In contrast to traditional optimization techniques that solely use the source code, PGO uses the results of test runs of the instrumented program to optimize the final generated code. The compiler is used to access data from a sample run of the program across a representative input set. The data indicates which areas of the program are executed more frequently, and which areas are executed less frequently. All optimizations benefit from profile-guided feedback because they are less reliant on heuristics when making compilation decisions.

Adding PGO to the existing build process

"I have not failed, I've just found 10,000 ways that won't work." - Thomas Edison

I had downloaded the source to the dependent libraries off the PHP wiki, checked out the PHP source code, and begun the process of adding in PGO support to the existing build process. This proved to be extremely difficult.

Even limiting the scope to just the core of PHP itself - without the dependent libraries - I ran into trouble trying to compile using PGO instrumentation and then re-linking after running some tests. The make file that gets generated by the configure.js script (a JScript version of the automake configure script for the Windows platform) was just not built with what I had in mind.

I spent the better part of two weeks trying different approaches to tweaking the makefile so that I could use PGO to improve the PHP executable, but I kept running into roadblocks. Worse, the closer I got to a makefile that did what I wanted, the farther away from the current build process I was getting, and I wasn't sure that what I would end up with would even be close to what was being built today.

The long dark winter road

"Only the meek get pinched. The bold survive." - Ferris Bueller

I came to the conclusion that I'd have to build new Visual Studio project files from scratch. What worried me was that this would end up to be a completely different build process, and I'd never get the community to abandon what was already working, so I'd better be able to rebuild these new project files easily.

I started looking (inside Microsoft and out) for any tools which generated Visual C++ project files. I found someone internally who had used some JScript to create project files from text files, but after some experimentation, I found this was nowhere near what I needed. What I really needed was a way to convert the generated Makefile into a .vcproj file-and not just 'wrap' it.

Once I found there was no such tool* , I began trying to figure out how to create one. I had this idea a few times in the last decade or so: watch how a program is compiled, and create a project file that does the same thing. Having tossed around the idea in my head before, I knew it wasn't going to be trivial but, without it, I couldn't do what needed to be done.

* Let me tell you: you never want to think about writing a tool to parse out what a makefile does. It's rather like making a tool that tells you how sausage is made, in excruciating detail. Ugh.

In Part III, I'll talk about the trouble with observing the build process.

Announcing the PHP SDK for Windows Azure

Peter Galli — Wed, 13 May 2009 00:29:00 GMT

Vijay Rajagopalan, a Principal Architect here at Microsoft, is at TechEd India, where he will demo later this week a new set of interoperability projects related to PHP.

These projects include the PHP SDK for Windows Azure, an open source effort for which Microsoft has provided funding, with development by RealDolmen, whose goal is to provide high-level abstractions that enable PHP developers to interoperate readily with Windows Azure.

The PHP SDK for Windows Azure focuses on REST and provides PHP classes for Windows Azure blobs, tables and queue, helper classes for HTTP transport, AuthN/AuthZ, REST and error management, as well as manageability, instrumentation and logging support.

Rajagopalan will also announce the launch of a series of projects that offer samples and a toolkit that enable PHP developers to include Silverlight controls, Microsoft Virtual Earth maps and IE Webslices and Accelerators in PHP web applications; as well as automatically generated a simple "Create, Read, Update, Delete (CRUD)" PHP application from a table in SQL Server.

These projects, for which Microsoft has provided funding and which are available on Codeplex under a BSD license, are yet another proofpoint of the company's commitment to interoperability, and developers will be happy to know that the first batch of these have already been developed by Accenture.

Read Rajagopalan's full blog here for all the details.

The Azure Services Platform has been designed to be open, standards-based and interoperable, and its support for XML, REST and SOAP standards means that any of the Azure services can be called from other platforms and programming languages.

Microsoft has provided funding for two other SDKs that support third party programming languages: Java SDK for Microsoft .NET Services and Ruby SDK for Microsoft .NET Services so as to facilitate interoperability between the Azure Services Platform and non-Microsoft languages and technologies.

The inclusion of FastCGI in Windows Azure's hosting environment was announced at MIX 2009, and the protocol enables developers to run web applications on Windows Azure that were written using third party programming languages, including PHP. This opens up new options for PHP developers to deploy their applications.

A Technology Preview of the PHP SDK for Windows Azure will be released under a BSD license, while a functionally complete version of the SDK, which will support tables and queues, should be available for download by this fall of 2009, but the team is calling on developers to provide feature requests, test the toolkit, and join the user forum.

So, stay tuned, as there's a whole lot more to come!

Apache Stonehenge: Interoperability at Work

Peter Galli — Tue, 12 May 2009 17:24:00 GMT

The Stonehenge incubator project is approaching its first milestone: deploying the first set of samples and making them work together.

This is a really exciting development and continues to deliver on the project's primary goal: to provide practical applications that span languages and platforms and demonstrate how to achieve interoperability.

Multiple implementations of the Stonehenge Stocktrader sample application, including .NET, Java, PHP, Python and Ruby, have been committed to the repository. You can check the code here.

From a simplified architecture point of view, the Stonehenge Stocktrader application is built as follows:

A User Interface layer delivering the web front end (HTML)

A middle tier layer including a Business Services layer (login, account processing) and an Order Processing layer (buy/sell transactions)

A Data Access layer to provide access to the database for the middle tier layer (Business Services and Order Processing)

And, finally, the database where the application data lives

The work thus far has focused on the .NET, PHP, and Java interoperability scenarios, and the three Stocktrader implementations have been deployed in multiple configurations.

A series of tests were then run, mixing and matching the layers from the three implementations, playing with the configurations and leveraging the Web Services standards, including WS-Security, to provide message integrity and security.

A detailed "interoperability walkthrough" explaining all the different configurations has been posted here, while the full blog post by Kamaljit Bath, a Principal Program Manager in the Interoperability Technical Strategy Team at Microsoft, can be found here.

"Microsoft is pleased with its participation and the progress so far, and this new outcome from the Stonehenge project is very encouraging. With the implementation of the WS-* Standards, we get the benefit of distributed applications and platforms. We recognized that it is not always easy to achieve these goals, but I really feel this type of practical guidance will be helpful for these types of scenarios," said Bath.

The team is also actively soliciting comments and feedback, and encouraging both developers and users to participate in the project to ensure that the project continues to move in a direction that meets real people's needs.

Helping Facilitate Open Government

Peter Galli — Wed, 06 May 2009 19:03:00 GMT

Microsoft will announce on May 7 an initiative to help government agencies and developers publish and interact with their data in Windows Azure, the company's cloud computing platform.

One of these is the Open Government Data Initiative (OGDI), a cloud-based approach to housing public government data in Windows Azure, making it accessible in a programmatic manner via open standard protocols and application programming interfaces.

The source code for OGDI is being made publicly available through CodePlex, Microsoft's open source hosting site, so that developers may reuse it and provide feedback. Sample code is also being provided for technologies widely used on the Web, including PHP, Python, Flash, JavaScript, and Silverlight.

This initiative helps to provide government with tools focused on increasing responsiveness and access to critical services, streamlined processes and services.

For their part, Microsoft and its partners have developed a robust enterprise architecture approach that enables agencies to meet the technology requirements of government mandates with a familiar set of tools - built on an enterprise-ready, scalable, and easily-managed software-powered architecture.

So, in short, the goal of ODGI is to reduce the cost of publishing government data, and simplifying data access by leveraging cloud computing and open standards.

More information on Microsoft's Open Government Data Initiative can be found here.

To see an implementation of a data service in Windows Azure, using a sample of publicly available government data, visit this reference beta site.

These moves are part of Microsoft's ongoing open government efforts aimed at helping government organizations meet goals of transparency, participation and collaboration, particularly as an ever increasing amount of data becomes necessary and available.

As such, new methods need to be opened up to allow interaction with that data, and Microsoft's OGDI is designed to help public sector entities meet these goals.

This software, which underscores the importance of programmatic access to government data rather than having to download it, will give developers the ability to write programs that access data via Web-friendly programming methods without having to download or host the data; and let them write applications using any technology via open standards.

It also provides easier access to a broad array of government datasets, enabling the building of new and unique applications, while governments will be able to automatically refresh data without having to buy and maintain servers.

Cloud computing is the ideal platform for government data, and the technology is finally available to make it happen, says John Miri, Senior Fellow at the Center for Digital Government.

"The qualities that government looks for in an information management platform - things like flexibility, scalability, security, performance, and cost efficiency - are all better in a cloud model. As we see demands for government to become more transparent, collaborative, and interactive, a shift like this in technology architecture just has to happen, " he says.

For governments to become truly open, citizen access to public data in standards-based and interoperable ways is essential at all levels of government.

Given that most federal, state, local and education entities implement the American Recovery and Reinvestment Act (ARRA), they can meet open government goals of oversight, transparency and accountability through cloud and on-premises solutions such as Microsoft Stimulus360, which helps public sector agencies track, measure, and share information about federal stimulus programs through graphical dashboards and maps.

Japanese LAMP Engineers Visit Redmond

Peter Galli — Tue, 10 Mar 2009 21:30:00 GMT

I was fortunate enough to spend last Thursday with a group of LAMP engineers who have some experience with Windows Server and IIS, and who are based in Japan.

The three - Kimio Tanaka, the president of Museum IN Cloud; Junpei Hosoda, the president of Yokohama System Development; and Hajime Taira, with Hewlett-Packard Japan - won a competition organized by impress IT and designed to get competitive LAMP engineers to increase the volume of technical information around PHP/IIS and application compatibility. The competition was titled "Install Maniax 2008".

A total of 100 engineers were chosen to compete and seeded with Dell server hardware and the Windows Web Server 2008 operating system. They were then required to deploy Windows Server/IIS and make the Web Server accessible from the Internet. They also had to run popular PHP/Perl applications on IIS and publish technical documentation on how to configure those applications to run on IIS.

The three winners were chosen based on the number of ported applications on IIS, with the prize being a trip to Redmond. A total of 71 applications out of the targeted 75 were ported onto IIS, of which 47 were newly ported to IIS, and related new "how to" documents were published to the Internet. Some 24 applications were also ported onto IIS based on existing "how to" documents.

The first-place winner Kimio Tanaka managed to port 71 applications onto a single IIS server. His technical documents can be found here.

Kentaro Yoshikawa, the Platform Strategy CSI Lead for Microsoft Japan, put the competition together and brought the winners to Redmond, where we arranged for them to meet with folk from the Windows Azure, Windows Server and IIS development teams. They also spent time with Sam Ramji, the Senior Director for Platform Strategy, as well as with Tom Hanrahan and Hank Janssen of the Open Source Technology Center.

Kentaro told me that the three were really impressed by the depth of the discussions they had during the day, which was useful to them as they have, until now, mostly lived outside of the traditional Microsoft ecosystem.

They also appreciated the depth of technical thought, strategy and commitment to open source communities that exists within not only the Platform Strategy group, but across Microsoft.