Port 25: The Open Source Community at Microsoft : Interop, Bill Hilf

See Change

billhilf — Thu, 21 Feb 2008 12:00:00 GMT

I have always been a fan of Tim O’Reilly’s phrase the “architecture of participation” to describe “systems designed for user contribution.” I liked it so much that at last year’s OSCON I made this concept the focal point of a discussion about how Microsoft’s products, programs, and partnerships have evolved over time to further this idea of ‘participatory systems’. (The slide I used at OSCON in Portland on July 26, 2007 is shown below).

(Presented July 26, 2007, OSON, Portland, Oregon)

I also like the metaphor of an ascending curve that reminds me of a strand of DNA. To me it is a visual representation of the fact that as the number of examples of architecting for participation have increased, the mindset and the behaviors involved in doing so have increasingly become a part of the company’s core culture and software design and development practices.

Today we are making a set of broad-reaching changes that go above and beyond any prior incremental changes in Microsoft’s DNA, that opens the door on new horizons for what architecting for participation might mean in the future. I want to talk about what this means—and why an open source interoperability initiative is an important part of it.

To understand why Microsoft is making broad-reaching changes to its technology and business practices that will drive greater interoperability, it’s important to step back and look broadly at the way the emergence services oriented architectures, web services, and the growing importance of software plus services are transforming the world of technology.

Ray Ozzie described this sea-change well this morning:

…as we put more and more of our data into [technology] products, a new set of issues emerge. Whether it’s our health records, or our customer databases, we’ve progressively learned that our documents and data have a lifetime that potentially spans well beyond the lifetime of any specific application that might’ve been used to create it. For our records and our documents, issues such as preservation and portability have become vital concerns…Furthermore, as a byproduct of the internet’s ubiquity, virtually every system and product nowadays has become interconnected. From the mobile phone in your pocket, to your PC, to the heterogeneous systems within our enterprises, everything’s being interconnected – and connected to the Web as our “universal hub” for information sharing.

At MIX ’07 Ray put the sweeping implications of this overarching vision in the context of history, starting with what he called the “dawn of the PC revolution” in the 1980s. There is an important connection to bear in mind between some of the key inflection points in the technology landscape (like the availability of low cost PCs, and cheaper, faster connectivity) with changes in how aspects of openness and developer opportunity have evolved together. This resonates with me when reading today’s announcement (available here):

Historically, as lower cost hardware became widely available, documented APIs and free SDKs enabled developers to more quickly develop applications

As the number of applications exploded, and network connectivity became the norm, protocols enabled exchange of information between programs and over the wire.

And as many types of computing devices have proliferated and connectivity has become ubiquitous, data portability and standards have become key tools in the toolbox for a loosely-coupled, services-based world.

Long-term success for Microsoft depends on our ability to deliver a platform that is open, flexible, and provides customers and developers with choice. These choices include Microsoft and open source technologies working together, and this will continue to be the case in the future. By increasing the openness of high volume products across APIs, protocols, and standards, we can continue to provide the platform that offers developers and businesses, including those based on open-source technologies, the broadest range of opportunities to innovate, deliver value, and create seamless experiences for end users.

By building on and expanding existing facilities, events, and resources supporting interoperability, including labs, plug fests, technical content and opportunities for ongoing cooperative development, the open source interoperability initiative will ensure this fundamental change in how we run our business and share information is broadly inclusive of open source technologies. As Microsoft takes this significant step forward into the interconnected world of the future, we aspire to doing so with members of the open source community by our side now and for the long haul.

Today is an important day, full of change. A wise inventor once said: “The world hates change, yet it is the only thing that has brought progress” and although I’m sure today’s news will bring critics aplenty, it is with time and commitment that this change will manifest. I’m extraordinarily proud to be part of Microsoft and to be part of this change.

FAQs, updates and news about the initiative will be posted on www.microsoft.com/opensource/interop .

Layers

billhilf — Thu, 26 Apr 2007 23:56:00 GMT

When I started programming, it helped me a lot to think about the OSI model (Open Systems Interconnection Basic Reference Model). On the right is a simple example of a five layer OSI model. This type of model can help when coding or administering a system so you can effectively debug at the right ‘layer’. I’ve found that I use this same logic now in all sorts of other areas, as it helps me parse out the details of an issue. I also was reminded of this while reading one of Cory Doctorow’s new short stories, ‘When Sysadmins Ruled the Earth’ in ‘Overclocked’.

I’ve recently been looking at broadband statistics and, as usual, working on various business model issues. So let me parlay the OSI framework concept into a topic around mixed models and the Web. I often hear others try to simplify open source by comparing it to the Web or the Internet. This description is often used disingenuously but it did get me thinking about the relationship and it’s a fun thought experiment so let’s break the totality of the Web down for a minute to prove the point – and let’s use an OSI-like model from the bottom up (including, but not limited to, protocols).

Physical, Data, and Network Layers

For the Internet this would be not only Ethernet standards but also electrical specifications, bridges, switches, host adapters, and signals operating over copper and fiber. ATM, Frame Relay, IPv4/v6, IPSec, RIP, X.25, and other protocols also live at these layers. But the Internet isn’t just protocols. Companies such as AT&T, Quest and Level 3 have laid hundreds of thousands of miles of fiber-optic cable at the physical layer and infrastructure providers such as Foundry, Juniper Networks and Cisco build technologies that allow Internet exchange points and ISPs to interconnect.

Transport+Session, Presentation and Application Layers

Here we have the layers that move the data between end users and programs. Fundamental to the Internet are TCP/IP of course (and UDP for you gamers). TCP/IP is over 25 years old and being an open standard was critical for its dissemination and success. Other important protocols and services at this layer are POP3, SMTP, SSH, HTTP, DNS, instant messaging protocols (and many more). These protocols have been implemented in both open source and non-open source software, the key was having standard protocols for communication. Also at this layer are other infrastructure-like providers such as Akamai, VitalStream, BitTorrent, Amazon’s S3 and other caching and content delivery networks.

It is important to note that at all of these layers above were other once-relevant technologies that have since faded or altogether expired. When I worked for IBM I use to carry a Token ring adapter for my ThinkPad as many IBM offices didn’t have Ethernet, but only Token ring (this was true just four years ago). Anyone use much Token ring today? Or RUDP? Or FDDI? Or even telnet? These each have diminished or disappeared, IMHO, because either 1) something better came along and/or 2) lack of relevance or value to consumers, users and/or businesses. These are positive market forces: we want better, higher value, more relevant technologies and standards to replace lesser, lower value, irrelevant versions of the same.

There is an important, non-OSI layer above all of this and that’s the content that is driving the growth of the Web and broadband (global number of broadband connections rose 33% last year). My highly subjective distillation of ‘content’ is the YouTube, MySpaces, Yahoo, MSN, Google conglomeration of data that pumps across those layers above every day in all their data hungry glory. Oh, and all that advertising too. Certainly, there is a supply-demand correlation between the infrastructures at all levels of the stack and the content users are demanding (and supplying back). These are also positive market forces. Companies such as Level 3 (which was almost itself leveled in late-90s) are seeing growth in traffic on their fiber lines and also in their revenue – and they are buying more. Comcast has signed up over 12 million homes for cable-based broadband connectivity. Western Europe broadband penetration is growing faster than the U.S., and Japan now has 7.9 million fibre-to-the-home subscribers. The home media and phone technologies will also be tapping into these bigger pipes, from the TiVo to iPhone to Windows Mobile devices. And all sorts of amazing applications are sprouting up to take advantage of this broadband growth – for a test, think back to how many videos you watched online just three years ago compared to today.

So what is the relationship between all of this? Certainly, without useful and relevant standards like TCP/IP and HTTP we wouldn’t be very far. But we also wouldn’t have today’s Web without the physical fiber and backbone providers, IXPs/ISPs, and router manufacturers that provide the infrastructure. And without software such as Apache, IIS, Firefox, Internet Explorer, etc., we wouldn’t be using the Internet like we do today. And last but not least, without something to do on the Web, from reading news on Yahoo to auctions on eBay to Skype phone calls to videos on YouTube or social networking such as MySpace or doing business online, the Internet would have just been a neat technology experiment (or, minimally, as one of my favorite BBC columnists Bill Thompson points out, a tool for ‘computer scientists to find ways to share time on expensive mainframe computers’). Open source, proprietary, infrastructure, protocols and standards… and lots of hard work and innovation. It’s all in there – that’s the Web we have today. Just like there is a mix of content that makes up the Web, mixed software, hardware, infrastructure, and a community are all necessary parts of the body Internet.

If someone needs proof that open source and commercial models/software/hardware/etc. can be and are compatible, just look at the Web. Not only are they compatible, they have proven to be an amazing powerful combination. The challenge for the OSS pundits is to dig deep, don’t be superficial. I like how Stephen Walli challenges a lot of ‘stack’ thinking by explaining how the application is a network, and the network isn’t simple. It’s a good analogy, and although the OSI layer-thinking helps draw some lines, the network model is more realistic – which is why I am using the Web as subject matter here.

When I was a kid, my oldest brother used to sell me gravity insurance for $1 (for the record, I only bought one policy when I was six). It was his lesson that I shouldn’t forget about reality. He tried to sell me another policy again when I finished graduate school – likely worried I was getting lost in theoretical thinking. In reality, there are powerful combinations of mixed models in software design/development, licensing, and businesses. We can bury our heads in the sand, or in the clouds, and believe there are only two camps, two separate and foreign tribes – open source and commercial. It might even make us feel better to believe this. Or we can see that, in the real world, there is no spoon.

-Bill

Shipping Containers and Standardization

billhilf — Tue, 28 Nov 2006 04:51:00 GMT

I’m sitting in Istanbul watching ships – big ships with huge loads of containers on board. I recently finished a great book titled The Box, How the Shipping Container Made the World Smaller and the World Economy Bigger, Marc Levinson (2006), a fascinating analysis of the history of the shipping container. It may sound dry, but it’s a very interesting story – particularly of the entrepreneur, Malcom McLean, who challenged the norm and introduced standardized, packaged shipping; transforming commercial shipping from a costly, labor intensive and inefficient system into a booming industry that radically dropped the cost of transporting goods around the world.

If you are in the software or IT industry, it’s near impossible to read this book and not correlate many of the concepts of ‘containerization’ or standardization to the technology world. Almost all containers today that you see on ships, trains or in loading yards, are between 40 ft (12.2 m) and 45 ft (13.7 m). Typically, most are 40-ft and have a capacity of 2 TEU (twenty-foot equivalent units). The reason for the massive change in both transportation and the global economy is because of this simplicity of size – a small set of standard sizes that allowed ships, trucks, receiving bays, and all of the logistical systems related to easily adapt to an industry wide standard.

Prior to containerization, there were major inefficiencies in commercial shipping. Packaging and crating was inconsistent – “breakbulk’ cargo consisted of separate items that had to be handled individually, such as bags of sugar or flour packed next to steel piping. The type and shapes of the shipping vessels were not designed for efficiency: wide at the top and narrow at the bottom. All of this resulted in the large use of manual labor to absorb the inefficiencies in the process. See clip 3 to watch Levinson describe why trade was so difficult prior to containerization.

Standardization of containers resulted in tremendous efficiencies and unlocked the interoperability between the things that transported the containers and the tools, machines and facilities that need to send and receive these containers and their contained goods. How big of a change did this standardization make?

Today, approximately 90% of non-bulk cargo worldwide moves by containers stacked on transport ships. Some 18 million total containers make over 200 million trips per year. For the past ten years, demand for cargo capacity has been growing almost 10% a year. There are ships that can carry over 11,000 TEU ("Emma Mærsk", 1,300 feet long), and designers are working on freighters capable of 14,000 TEU. At some point, container ships will be constrained in size only by the Straits of Malacca, one of the world's busiest shipping lanes. There are plans underway for a quarter-mile-long ship, called the Malacca-Max, targeting a capacity of 18,000 containers (for scope, it would take $1 billion in cargo down with it if it sank).

When I talk with customers, repeatedly they are looking for a way to simplify. They are looking to get more things done (and faster) with less complexity and costs in their environment – science projects are not what they are looking to experiment with. Moreover, the last IT director I talked with is using this as an interviewing tool – testing the interested candidates for their ‘appetite’ for standardization versus highly customized, highly variant systems. Granted, there is always a need for some type of customization and variation, it’s not a question of ‘if’ but of ‘how much’, ‘where’, and ‘why?’ These questions are critical and each needs to have real business value described in the answer. Building a business on a standard platform helps facilitate this simplification and unlocks interoperability for IT. It is an important IT policy that I have employed in the past and many IT leaders I know use this as a core part of the IT strategy.

Standardization in shipping containers created an entire industry – literally grown around this concept of a standard ‘platform’. Beyond the core shipping industry itself (logistics, supply chain, services, ship-to-gound transport, etc.) people are starting to ‘hack’ the shipping container standardization model. Sun announced a data center in a shipping container – interesting proposal, but limited IMHO (maybe they should GPL it J). I find AquaSciences (picture on the left) one of the most fascinating ideas: using containers for fully-contained mobile freshwater generation systems. Quite easy to see the potential of this and it literally could save lives. Of course, there are people building living spaces out of containers, such as Freitag’s ‘skyscrapers’ in Zurich. Here’s a list of all sorts of shipping-container-as-house ideas.

Standardization was the key to success. As I sit here in Istanbul, watching literally hundreds of container ships make their way through the Bosporus from the Black Sea to the Sea of Marmara (which is connected by the Dardanelles to the Aegean Sea, and thereby to the Mediterranean Sea), I can’t help but marvel at how such a simple concept – the standardization of the shipping container – made such a radical and important change to the global economy and our lives. I don’t think it’s circumstantial that standardization made things cheaper and easier (two words Levinson uses in his closing sentence of ‘The Box’). Here’s to great ideas and to getting more work done with less complexity.

-bill

P.S:

1. video interviews with ‘The Box’ author Marc Levinson
2. Industry standards and platform standardization are important in software development as well as the datacenter – for an example see this article on “Using IEEE Standards to Support America’s Army Gaming Development”

Here's some big news

billhilf — Fri, 03 Nov 2006 00:30:00 GMT

Today we announced a very important partnership with Novell that represents a milestone for our industry. The partnership has technical, business and legal significance for Microsoft, Novell, our customers and the open source community. I want to break it down into reasonably sized chunks of information to help explain what it is and how we arrived here.

There is often a big ‘elephant in the room’ when discussing Microsoft and Open Source, and that issue is related to intellectual property. Our business at Microsoft is built on intellectual property, as are many businesses: Apple, Sony, BMW, Amazon, Nike, and the list goes on. What does this mean? It’s relatively simple – each day thousands of people come to work at Microsoft to build new technologies that (ideally) make it easier, more efficient, and more enjoyable to use a computer. We pay these people to do this and then we sell their work through our products. We then use that money to continue to pay these people to keep innovating and building new things that meet the above goal and the cycle goes on and on. Like the other companies I listed above, we protect the intellectual property from these innovations through patents, copyrights, trademarks and trade secrets (the four parts of intellectual property). Use an iPod? The menu structure driven by that little wheel is protected by patents that Apple owns. Wear Nike shoes? The layering of those soles and their contents are protected by patents and the Nike ‘swoosh is a protected trademark of Nike. Used Red Hat’s update service? That is also protected by a patent application. Even that soda you got at McDonald’s was likely delivered through this interesting invention. This is the world of intellectual property and apart from opinions on the governance of this system, this is how it works today.

Microsoft’s intellectual property is almost entirely in software innovations (no surprise) so it was very important for us to have a model where our intellectual property continues to be valued and respected by all constituencies, including the commercial open source world. Again, our business (as many are) is rooted in intellectual property so this is a non-trivial issue. Our agreement with Novell provides a way for Microsoft and Novell to keep square on one another’s patents and it provides a way for us to pass on the benefits to customers who want to know these complex issues have been sorted out. At the same time, we’ve got to do all this in a way that works for those open source developers who contribute code for the love of technology, not for the paycheck. Microsoft is stepping up and saying we won’t assert patents against individual, non-compensated developers for the code they write. What does all this mean? It means that we have a way to work through (or bridge) the open source and Microsoft issues around intellectual property by defining a line between commercial and non-commercial use of our intellectual property. Additionally Novell’s customers can feel confident that they are clear and compliant (at minimum with Microsoft) of the open source products they buy and deploy from Novell.

Figuring out this model unlocked a variety of new areas that we can collaborate on together, such as the work we’ll be doing around Virtualization, Office file formats and management technologies. Here are some of the highlights:

Patent coverage

The concern over potential patent infringements makes some people nervous about the deployment of open source technologies.
From the start, a design principle of the agreement was to be compatible with the GPL.
To do this, Novell and Microsoft are providing covenants to each other’s customers, therefore releasing each company from the other’s patent portfolio. This may sounds odd vs. a traditional patent cross-license agreement but it is one of the things that makes this deal so unique.
What it really means is that customers deploying technologies from Novell and Microsoft no longer have to fear about possible lawsuits or potential patent infringement from either company.

Development: Virtualization

Microsoft and Novell will collaborate in enhancing and developing the functionality required to efficiently virtualize Windows on Linux and Linux on Windows.
Both will now be first class citizens in data centers, addressing the needs of mixed environments. They will both enjoy optimized, supported and tuned device drivers to maximize their potential.

Development: Virtualization Management

As a plus, the companies will work together to implement the necessary standards to manage data centers that run mixed environments (WS-Management).
Novell will develop tools to manage virtualized Windows machines, and Microsoft will develop tools to manage virtualized Linux systems.

Office Open XML

Novell engineers have been working for the last year together with Microsoft engineers through the ECMA TC45 working group in producing a complete specification that would allow for interoperability across office suites.
Novell will develop the code necessary to bring support for Office Open XML into OpenOffice, and will contribute that support back to the OpenOffice.org organization. Novell will also distribute the Office Open XML plug-in in their own edition of OpenOffice. In addition, Novell will participate in the Open XML Translator open source project.

Mono, OpenOffice and Samba

Under the patent agreement, customers will receive coverage for Mono, Samba, and OpenOffice as well as .NET and Windows Server.-All of these technologies will be improved upon during the 5 years of the agreement and there are some limits on the coverage that would be provided for future technologies added to these offerings.
The collaboration framework we have put in place allows us to work on complex subjects such as this where intellectual property and innovation are important parts of the conversation.

We’re also building a joint research facility where Microsoft and Novell technical experts will architect and test new software solutions, and will work with customers and the community to build and support these technologies (read: cool big new lab with all sorts of gear and tinkering going on).

I’ve been in the open source world for over twelve years and I have worked with thousands of customers worldwide who use commercial and open source software. I have been part of the large deal team working on this partnership for a long time now. In my opinion, this is the most important bridge ever developed between Microsoft and open source and *significantly* helps customers and hobbyist developers have the peace of mind that they need and have asked for. As Steve Ballmer said today, “They said it couldn’t be done” but we did and I’m personally very proud to have been part of this bridge building.

It’s been a long and exciting day so now a few of us from Microsoft and Novell are going to go get some sleep

Interoperability Customer Council

admin — Wed, 21 Jun 2006 15:02:00 GMT

Last week we announced the formation of an Interoperability Customer Executive Council. This is something many folks here at Microsoft have been working on and thanks go to a wide variety of people. I was involved from an OSS perspective and will continue to be active with both our internal groups and this customer council to help this mission succeed. As you can imagine, interoperability is a very wide and loosely defined word – it can mean many things to many people. Particularly when you produce a lot of software! Rather than try to ‘pick’ certain areas to focus on (believe it or not, Microsoft is constrained by time and money like all other businesses) we decided the most effective and beneficial path would be to have our customers drive this conversation. Thus was born this council, formed by customers of different sizes, business types, and geographic location, in order to provide guidance on interoperability issues that are most important to customers, including connectivity, application integration and data exchange.

Interoperability in a heterogeneous environment doesn’t happen accidentally, nor does having your code open make everything ‘just work’ together. It takes thoughtful design and architecture; it takes time, effort and engineering discipline. It requires working closely with partners, competitors and customers. It also requires mature understanding that all things don’t necessarily require interoperability, and frequently those things that do, sometimes require different types of interoperability. Point being: it takes focus, energy and commitment. This council is a great step and example of this commitment from Microsoft.

I’m very excited by this step forward, below are some further news stories about this:

CNET News.com

Clients to advise Microsoft on software linking

http://news.com.com/Customers+to+advise+Microsoft+on+software+linking/2100-1001_3-6083664.html

eWeek

Microsoft Customer Council to Focus on Interoperability

http://www.eweek.com/article2/0,1895,1976394,00.asp

-Bill