Port 25: The Open Source Community at Microsoft : Open Source, Anandeep

A New Appreciation for Open Source in India - and Our Role in it.

anandeep — Sat, 29 Nov 2008 23:55:00 GMT

As I stated in my last blog, I am attending the premier Indian Open Source conference, FOSS.IN, in Bangalore. This conference had some very technical talks (which I will also blog about) but, like any other Open source conference, it was the people who were the most interesting.

There seems to be a large PHP contingent here. I met some folks from Piazza, a company that does PHP applications. Of course they asked the million dollar question - "Microsoft has an Open Source group?" But, very quickly, they soon started talking about how they could work with Microsoft technologies. They hadn't realized that we had a relationship with Zend, and that Microsoft would treat PHP as a first class language.

Once they heard about this, they excitedly started thinking of .NET applications that their customers were asking for, and how they could build them. These developers were not based in Bangalore, but in a small town in Kerala. They followed the true Open Source model of living and developing from where they wanted, and did not have to work in a large overgrown city with corporate offices. This is a BIG thing in India - and I am seeing this for the first time.

The other PHP linkage I made was with Damian Hickey, CEO of Zacware.com. Zacware is making an e-commerce server called Freeway, which is PHP based. His was more the traditional Open Source business, and his firm had already reached out to Microsoft's representative in Queensland, Australia to see how they could make this application run on Microsoft platforms.

We had a long discussion about community and how Microsoft could approach Open Source. Again, the theme of opening a two-way conversation came up. Damian's development team is in India and I was able to talk to his PHP dev lead, who was also excited to work on .NET.

I had asked one of the organizers why I hadn't heard of a well known contributor to the Linux kernel who hailed from India. I was promptly told that Balbir Singh and Ankita Garg were the two names to know. I ran into Balbir - who works for the IBM Linux Technology Center - and we had some good discussions on kernels and people we knew in common. Turns out he knows Tom Hanrahan, my current boss. (Tom, I mentioned you in my blog. I hope you are noticing for my review next year!)

I then chatted with Amit Shah from Qumranet (now Red Hat). He gave a talk in the conference on how he parleyed an interview question from Qumranet into a career in hypervisors. Again, he was keen to work with Microsoft, especially when he heard about our virtualization Interop efforts with Xen and Novell.

I also met a passionate young student, Abhishek, who very politely asked me all the tough questions about Microsoft. These involved questions on patents, open sourcing Windows, and OOXML. I answered them, presenting my own point of view. The reaction? He said he had been waiting to talk to someone from Microsoft who was so open about this stuff, even if he didn't agree with me on everything.

What did I learn about Microsofts role in Open Source in India? Our role here is to be the two way conduit, but we also need to engage the community with the same kind of passion for our own (Microsoft's) stuff that they have for Open Source. Anything less will appear fake.

More to come.

Anandeep

FOSS in India

anandeep — Tue, 25 Nov 2008 19:47:00 GMT

I am writing this during lunch at FOSS.IN, the most prominent Open Source conference for developers and FOSS advocates in India. FOSS.IN is in Bangalore, which is where I graduated high school. That was a long time ago - and Bangalore has been transformed from the "garden city" to the software capital of India. Lots more skyscrapers and cars on the road. And lots of software companies (including Microsoft) have development or R&D centers here.

It so happens that the venue for FOSS.IN is in the Indian Institute of Science (I.I.Sc) auditorium. I.I.Sc was right in the middle of the path I trudged from home to school, walking 12 miles uphill both ways (or so it seemed). I have seen the Indian cricket team there, when they were visiting the old auditorium. The new auditorium on the same site is nothing like the old one, it's actually pretty nice. I don't think they would allow me to perform the amateur play that I was forced into by my high school drama teacher at the venue any more!

The conference is all volunteer organized and does not have the polished feel of a larger conference in the US. I think this actually works in the favor of the conference organizers because people seem to interact more with one other than tends to happen at other conferences I have been to stateside.

But I have actually been disappointed that I have not seen more Open Source contributors from India gain prominence. That is kind of sad, since we have one of the (if not the) largest pool of software engineers in the world. So, I am keen to talk to the people involved and see what is happening around Open Source development in India.

There are a large number of people attending - my estimate is between 300 and 400 - with just three vendors: Sun, Noikia and VMWare(?).

So far I have attended the keynote by Harald Welte on how Embedded Linux was not taking the right path and why. I also attended a session that went through an "interview question" that was given to KVM contributor Amit Shah to solve by Qumranet prior to bringing him on board. It was to facilitate the migration of guests from an Intel based architecture host to an AMD based architecture host. The talk was fascinating.

I am starting to mingle and will have a number of stories from interacting with the people at the conference. So, more to come later.

Anandeep

Finally... dive-into-the-deep participation!

anandeep — Fri, 07 Nov 2008 16:54:00 GMT

Whenever people get to know that I work in the Open Source Lab at Microsoft, there are a few knee jerk questions they always have. The most common one is: "Microsoft has an Open Source Lab?", while another question often asked is: "So when is Microsoft actually going to be part of an Open Source project and actually participate?"

This question does not, of course, refer to our Codeplex projects, which are aplenty and Open Source, but rather about participating in an ongoing project that is not under Microsoft's umbrella. Something that organizations with strong Open Source credentials are behind: Mozilla, Perl, Linux, Apache, Samba, PHP, Eclipse ... you know, the usual suspects.

Fundamentally, as a developer, the answer I (truthfully!) gave: that we worked with all those communities and had actually done some important work with them, was not soul satisfying. You know as developers that one needs to jump in with both feet and actually stand up and be counted as being part of a project.

Given the kind of work we had been doing in changing realities both at Microsoft and in Open Source communities, I knew that it was just a matter of time before that happened. (My colleague Hank Janssen hates the word perception, and I would have used it here just to aggravate him, but that would not capture what I wanted to say. Darn!).

Well, that day is arrived - and I am excited beyond words to say that we (as Microsoft) will be participating in the Apache QPID project. This was announced by Sam Ramji today during his keynote address at the Apache Conference in New Orleans.

Apache QPID is middleware for message passing and is based on the AMQP standard. I will be the point person for our lab's participation in Apache QPID. Actually I should say "what used to be our lab" and which now has truly become the Open Source Technology Center at Microsoft.

In a previous life, I was with a small start up that did secure Web Services management. This was when the SOAP protocol sent unreliable, in-the-clear over the wire messages, and what we did was make SOAP a secure, guaranteed once and once only protocol between two Web Services end points without requiring any changes to web or application servers. That gave me some understanding of the complexities of message passing. Add the complexity of time constrained responses, huge volumes of data and interoperability between disparate systems and the technical problem becomes real juicy!

I often used to wonder why there weren't more successful Open Source messaging systems, since the primary message passing systems in use were proprietary and weren't built for interoperability. Imagine my surprise and joy when Tom Hanrahan (da boss) told me that Microsoft was joining AMQP and was considering participating in the Apache QPID project. I jumped on the opportunity to be involved with the project.

I want to reiterate that we will follow Open Source principles in being part of the Apache QPID project, and not antagonize people through typical big company execution. This means the following things to me:

We will approach the process of working in the QPID project as beginners in technology, process and relationships.
We will listen first in the forums, in conferences and events, and most importantly one-on-one with existing community members.
We will only do things that add value as validated by the QPID community
We will strive to be leaders in building QPID's future, but that leadership will be an invitation from the community

This all means that we will be slower of the mark than we would, say in one of our internal projects - but we would like to get it right.

For now I am going to download QPID and run it in the lab. Look, listen and linger in the forums, read the documentation, run the test suites, play around with APIs and learn how the joint runs (so to speak).

Looking forward to seeing you all there.

Is High Performance Computing naturally Open Source (ie. for tinkerers)?

anandeep — Wed, 18 Jun 2008 13:17:00 GMT

I have always been fascinated by clusters. Some people envision working with desktops or workstations when they think of “working with computers”. For me working with computers was always with a large collection of computers in a back room somewhere. And how cool if you could make all those computers collaborate with each other working to solve cool things like genome mapping, movie special effects, simulations of car crashes or simulations of molecules being formed!

So you can imagine I jumped at the chance to work with the Windows High Performance Computing team. This is the same team that builds Windows HPC Server 2008.

I think most of the people working in the team are from the “large collection of computers in back room somewhere” school. Would be really different in the Mac software division I assume!

I work with the Open Source Software Lab and we are all things “Open Source” to the rest of the company. The HPC Server team wanted us to make sure that their product played nice with Linux infrastructure and vice-versa. The usual suspects like AD, Samba, LDAP. CIFS etc were involved. We had to make sure that these recurrent interoperability themes were addressed in the HPC environment. I also got a chance to dig into ROCKS, OSCAR, MPI stacks and job schedulers etc etc.

This was a very rewarding experience not only for the technology exposure that I got but the pervasiveness of knowledge of Open Source within the team. They were far ahead of the other product groups in this regard and “got” the Open Source ethos. In fact, prior to my interactions with them they had released an open source MPI stack based on Argonne National Lab’s MPI implementation.

The other reason was that a lot of their customers were relentlessly open source! The conventional wisdom is that HPC applications and infrastructure require a lot of tinkering. Of course, there are some applications like FEM and CFD and that are well understood, but the general feeling was that complete control and access to the underlying infrastructure is a must for getting the most performance out of a cluster. And performance is the main thing in “High Performance Computing”.

Linux is seen providing that access by HPC customers and there is a large base of Linux for HPC in academia, the national labs and other institutions that use large clusters for doing their thing.

But is this really true?

I think that HPC has gone through a typical evolution – it starts with a few people who have a pressing need. There is a cross disciplinary team formed that builds software to do their job and a community grows around it. The community reaches critical mass and people start making building tools to make it more convenient. ROCKS is an example of this. Great skill, knowledge and ability is needed to get the job done.

However, these skilled people now become overloaded. The tools and the infrastructure that they created become so popular that everyone, including people who do not have background that was assumed before, wants to use it for their ends. So the community responds and builds standardized, easy-to-use infrastructure pieces that start to fit seamlessly together. Some control is lost, but ease-to-use is the primary focus.

The infrastructure for HPC has reached that stage (ROLLS with ROCKS). Windows HPC Server 2008 is built for this ease-of-use too.

However, the applications have not reached the stage of ease-of-use. They have to be coded with a lot of domain knowledge and have to built from scratch to truly scale while running on clusters. That means that the application writers demand more control of the underlying infrastructure and want more access to it than the users and maintainers want.

I am going out on a limb and making a prediction here – soon end users will be able to specify instead of coding applications, be it genome comparison or physics simulation. This is similar to accountants finding spreadsheets. There will probably be a few different models for different types of applications but that stage will come pretty quickly.

The infrastructure that runs these user-specified applications will be adaptive and will take these specifications and automatically tune them for high performance on the clusters.

This is where the perception of needing control to the lowest levels will be moot. The best adaptive infrastructure will be the one adopted.

Bold enough for you?

Can Open Source Scratch an Itch Before It Happens?

anandeep — Tue, 06 Feb 2007 00:25:00 GMT

I am an avid reader of Joel On Software – I find his insights great and very revealing.

I was reading a recent blog post by Joel entitled “The Big Picture” in which he has this to say about open source

“Open source doesn’t quite work like that. It’s really good at implementing copycat features, because there’s a spec to work from: the implementation you’re copying. It’s really good at Itch Scratching features. I need a command line argument for EBCDIC, so I’ll add it and send in the code. But when you have an app that doesn’t do anything yet, nobody finds it itchy. They’re not using it. So you don’t get volunteers.”

This was in the context of the review of the book “Dreaming in Code” by Scott Rosenberg. Scott was on campus at Redmond a few days ago, talking about his book. The book is a “Soul of a New Machine” type look into an open source startup that was trying to make a PIM (Personal Information Manager) code-named Chandler – and the travails that startup went through. Scott begins his introduction by saying ”… the art of creating it (software) continues to be a dark mystery”, so you can guess the whole thing didn’t go very well!

This got me thinking – what is open source really good for? Is it only good for copycat or scratch-an-itch type of software? Could there be a limit to what open source process can achieve in terms of software artifacts?

First of all – being “copycat” or “scratch-an-itch” type of software is not bad at all. It can be argued that Firefox falls into the former category being based on closed source browsers but while gaining feature parity with other browsers it added new features. I think this was the benefit of the general public because not only did Firefox get better in the spirit of competition other browsers (including Internet Explorer) got better. As for the latter category, what good is software if it isn’t working for its’ users or “scratching an itch”?

I think the point Joel was trying to make was that bootstrapping an open source project requires either a user need or the need for an alternative. A community is more easily built when there is a shared need for functionality or alternatives.

But something about that bothered me. After all isn’t Open Source all about the “love of the game”? Why wouldn’t a community want to do something that was experimental and didn’t have any immediate payoff? Coming from a university research environment, I knew there were people out there putting out experimental code into open source including everything from Robotics to the ALICE Educational Software Authoring System (I knew the person behind ALICE – Randy Pausch from Carnegie Mellon). ALICE has a pretty vibrant open source community behind it.

That said, all the top open source projects (based on a poll by O’Reilly) fall within Joel’s characterization.

So can futuristic experimental projects be developed using the open source process?

I think that the answer is yes. But these kinds projects cannot be developed in a pure open source community process like that of Linux. An institution like a university or a company has to bring to it critical mass. The US government paid for a lot of ALICE – before it could be put out there in a true community process.

BTW I just looked at the ALICE website – and Microsoft has also supported ALICE financially. That wasn’t the case back when I was at Carnegie Mellon – I remember thinking, “How is an educational software package which is a rage with art students going help the US defense department?”. Almost all its funding at the time came from DARPA (Defense Advanced Research Projects Agency)!

There has to be a lot of money/resources/people put into a software project to bring it to a stage so that a community sees that an itch is going to be scratched, and then gets on board.

I was chatting with Hank Janssen and Kishi Malhotra about the “top” open source projects and stated that the top open source project I wanted to see was a “Cloud OS” which wasn’t yet around. I was waiting for the day when a system call made on my laptop would kernel trap on a machine in a data center in India, without my knowing or caring to know which data center or which machine. Ruminating on this I postulated that some of the early components are already there with the Google File System and the Google Cluster Architecture. Then I realized that even though those were Linux based they were by no stretch of imagination open source!

Madison Digital Image Database: Art History Evolves Through Open Source

MichaelF — Wed, 27 Sep 2006 22:13:00 GMT

To continue the series of interesting people we met at OSCON Anandeep spends some time with Andreas Knab Computer Systems Engineer at the Center for Instructional Technology at James Madison University. Andreas works on an Open Sourced .NET project at the University called: Madison Digital Image Database. The project is an online slide management and teaching tool. Think of it as an online substitute for the 35mm projector from Art History class. In the podcast Andreas and Anandeep discuss the project, the development environment and Andreas' experience working on an Open Source project.

Links:

http://mdid.org (project wiki)
http://mdid.org/demo (demo site, requires registration)

[Edit] We just got the following note from Andreas:

"...we actually just received an IMLS grant(http://www.imls.gov/news/2006/092606_list.shtm#VA) to add an API to MDID to facilitate interoperability with other image management and presentation systems, so the project will stay in active development for the foreseeable future."

Congratulations!

Thoughts from OSCON: Development Practices

MichaelF — Mon, 14 Aug 2006 13:30:00 GMT

I recently attended OSCON 2006, and had a lot of fun being there. Microsoft’s presence in the Open Source World is still a novelty (to say the least), so I always got a reaction out of people at the conference when they saw my badge!

We had a large presence there, because we do believe that open source as a development model is here to stay. Bill Hilf was at the conference and Port 25 has some of the interactions he had with open source luminaries Tim O’Reilly and Matt Asay. While Bill was having these interesting conversations, we at the OSSL (Open Source Software Lab) were busy attending the talks at the sessions and collecting “swag” on the exhibition floor! I do have swag from HP, Google, Intel, Dell, AMD, Oracle, ActiveState, Solid and MindTouch but interestingly IBM was missing. Anyone out there have an IBM t-shirt to exchange for our Port 25 t-shirt? (see accompanying pictures)

The buzz in the air, appeared to me to be about open source both as and in business. The talks I gravitated towards (naturally) were about open source development practices. These ranged from taking closed source products and turning them loose as open source projects to driving pure open source development to using experts in a particular domain as contributors for a project not thought suitable for open source.

There was a common thread running through all these talks – the critical nature of development practices. No, there wasn’t anything earth shattering – these were development practices that are accepted as “goodness”. But the forces surrounding open source development made the use of these practices almost a necessity for projects to get of the ground. This is not to say that closed source companies do not follow these practices, but due to co-location, centralized management and other circumstances that go along with commercial development, some of the practices may not be rigidly enforced and the lack of these practices may not impact the product as much. Open source development does not have that luxury (I refer only to successful open source development projects, not the long tail of open source projects that are fossilized on SourceForge and other repositories).

The practices fall into the following categories

Easy “build”

Consider this, you browse to an open source project which is new to you and download it (could be from repositories such as Sourceforge or Codeplex). It doesn’t install and takes a lot of wrassling to run. More often than not, this first impression decides your level of participation. If you can’t find something cool, try it and run it – there are other fish in the open source sea!

The initial install is not the only thing that has to run right, as an open source developer working on a module, adding/modifying some source code, building the source code and running it are part of the iterative process that lets developers be productive. A system which doesn’t make the dependencies transparent and which doesn’t have a build system to include all the necessary files (and NOT include the unnecessary ones) will probably not get good developer input.

The easy build thing has been known for a while. At Microsoft, product groups have the concept of daily builds – if you as the developer “break” the build, you don’t go home till you fix it. In order for this to work, each developer should be able to build the system on his desk easily.

Quick iterative program development in the large without hassle is the name of the game. The very nature of open source development which needs to attract developers to gain momentum leads to a focus on easy builds.

There’s doc (or a blog or a newsgroup) for everything

Most open source developers don’t work in the same building. I am talking about open source developers in the community, not those employed full time by commercial open source companies. (Though most commercial open source developers have to interact with community developers on their virtual team). This means you can’t walk to the next office and ask the developer about how the API call really works!

It follows that at any time of day or night, answers to questions like “how does the API call really work?” should be available through internet accessible means. This could be doc, a newsgroup, a wiki , a blog or any other easily accessible repository.

Documentation by developers, you ask, isn’t that a mythical being?

That’s exactly the point – open source developers do write docs, they just don’t recognize that they are doing so. In order for an open source project to truly take off, education of new developers is a must - both when they are viewing code and when they are looking at documents explaining interfaces, how things work and the meaning of life! Ok, maybe the last thing is not strictly necessary, but it does make reading documentation much more fun. Who wouldn’t want to work on Ruby after reading “why’s (poignant) guide to ruby”?!!!

Every lifecycle stage and artifact is important

The way you work your way up the “committer” chain in open source projects is to prove yourself useful. The path to building credibility is to write documents, find bugs, review codes and make yourself useful in a pretty stiff meritocracy. Even when a developer achieves the golden “commit” privilege they continue to participate in those activities.

Not having departments with people exclusively devoted to test, doc or reviews makes the development of a “caste system” difficult! Development managers cannot put pressure on test managers to shortcut tests – because the development managers and test managers could be the same person!

This is a little bit more subtle than the “more eyes make all bugs shallow” argument – that is only true if those eyes don’t think of looking at bugs as work that is to be done by other eyes!

This is even truer of documents and education – it isn’t some tech writer with expertise only in writing who writes the important documentation, but the luminaries of the community. When the “Gurus” (which in Sanskrit means teacher) do what they are meant to do – then nirvana is attained (I loosely paraphrase from the Bhagavad Gita!).

Sprints not marathons

Consider having developers in the US, UK, India and Australia – when is the best time for a meeting? When it’s morning in the US, its night in India – and who knows what time it is in Australia? Software companies whose code is all developed by their own employees can have coordination meetings on a schedule decided by someone – not so in open source. This means that coordination can’t be complex and long drawn out.

So open source makes use of XP principles, work on a small feature that doesn’t take more than a month (ok, so it isn’t the extreme XP where the feature shouldn’t take more than two weeks!). Based on community pressures, priorities can be decided. Longer term projects are either done by a single person or by a co-located team (by commercial open source companies for example).

That means planning horizons are small, and mistakes can be rectified without huge loss of time. Releases happen when there is a critical set of features ready. The community is able to get their hands on new features early and give early feedback, which further cuts down the time for stable development.

Of course this means that customers are running hard just to stay in one place, if they accept all the releases! But at least they have the choice…!

Visibility into EVERYTHING

Open source is not just about the source being visible. More people do look at server side code, but even on servers the number of code readers is small compared to the number of contributors.

Visibility in open source is about everything – how many and what bugs are there, time to resolve bugs, prioritization of bugs, who is contributing what, what comments were made about whose code, whose code was included and whose wasn’t etc. etc. etc.

This not only acts as a great feedback mechanism to users, it provides for real and open debate about priorities and execution. As long as the project is handled on a rational basis, people can predict the state of the project. They can anticipate when a feature they want or a bug they have will be fixed. It also allows users to submit code to fix bugs for their own problems and see it transparently go through the system.

My full time job at a previous position was mediating features for a group of customers (numbering in the 10s). This required the full energy of a team, which was not a development team, to gather this information in a closed source environment and then disseminate this information. Of course there were mistakes in information gathering and communication, since the customers only got a view of the project through an intermediary. Building trust with the customers took the better part of the year and resulted in a development process that was not as efficient as it could have been.

So flame wars notwithstanding, visibility into everything is an advantage for open source projects.

Community, Community, Community

In order for developers to be productive they have to communicate with whom they want, when they want and get back what they want. This means there is a burden on the open source product’s leaders to make sure that this responsiveness is part of their community. The artifacts used are answering 250 e-mails a day, having IM on all the time and putting systems in place that make this possible. One open source company I know of uses categorization software just so that the appropriate person can look at an e-mail and has fast mail systems that allow sub-second previews of the e-mails!

What this gains the company is encapsulated in a quote from an OSCON presenter – “When we had a closed source product people worked 9 to 5, but with open source there is so much interaction with the community that our developers are strongly motivated to work on finding solutions and building features and they are much more productive!”

Actually come to think of it, there isn’t a thing here that I have said that wouldn’t work where I work! In fact these processes are already at work at Microsoft. I am not only talking only about the work that we are doing at Codeplex, releasing source for products such as Power Toys for Visual Studio Collection available on Codeplex or the WiX tool available at Sourceforge (You can get more info about these projects from the interviews on Port 25 with Sarah Ford and Rob Mensching) but also about the code sharing internally within Microsoft. Since Microsoft does both platform and application development, application developers often need and have access to the bug databases and source code of platform level components. There is a lot a give and take between teams of users within Microsoft. This visibility has also been expanded to users such as academic, government and enterprise users under license agreements with Microsoft.