Port 25: The Open Source Community at Microsoft : anandeep

A New Appreciation for Open Source in India - and Our Role in it.

anandeep — Sat, 29 Nov 2008 23:55:00 GMT

As I stated in my last blog, I am attending the premier Indian Open Source conference, FOSS.IN, in Bangalore. This conference had some very technical talks (which I will also blog about) but, like any other Open source conference, it was the people who were the most interesting.

There seems to be a large PHP contingent here. I met some folks from Piazza, a company that does PHP applications. Of course they asked the million dollar question - "Microsoft has an Open Source group?" But, very quickly, they soon started talking about how they could work with Microsoft technologies. They hadn't realized that we had a relationship with Zend, and that Microsoft would treat PHP as a first class language.

Once they heard about this, they excitedly started thinking of .NET applications that their customers were asking for, and how they could build them. These developers were not based in Bangalore, but in a small town in Kerala. They followed the true Open Source model of living and developing from where they wanted, and did not have to work in a large overgrown city with corporate offices. This is a BIG thing in India - and I am seeing this for the first time.

The other PHP linkage I made was with Damian Hickey, CEO of Zacware.com. Zacware is making an e-commerce server called Freeway, which is PHP based. His was more the traditional Open Source business, and his firm had already reached out to Microsoft's representative in Queensland, Australia to see how they could make this application run on Microsoft platforms.

We had a long discussion about community and how Microsoft could approach Open Source. Again, the theme of opening a two-way conversation came up. Damian's development team is in India and I was able to talk to his PHP dev lead, who was also excited to work on .NET.

I had asked one of the organizers why I hadn't heard of a well known contributor to the Linux kernel who hailed from India. I was promptly told that Balbir Singh and Ankita Garg were the two names to know. I ran into Balbir - who works for the IBM Linux Technology Center - and we had some good discussions on kernels and people we knew in common. Turns out he knows Tom Hanrahan, my current boss. (Tom, I mentioned you in my blog. I hope you are noticing for my review next year!)

I then chatted with Amit Shah from Qumranet (now Red Hat). He gave a talk in the conference on how he parleyed an interview question from Qumranet into a career in hypervisors. Again, he was keen to work with Microsoft, especially when he heard about our virtualization Interop efforts with Xen and Novell.

I also met a passionate young student, Abhishek, who very politely asked me all the tough questions about Microsoft. These involved questions on patents, open sourcing Windows, and OOXML. I answered them, presenting my own point of view. The reaction? He said he had been waiting to talk to someone from Microsoft who was so open about this stuff, even if he didn't agree with me on everything.

What did I learn about Microsofts role in Open Source in India? Our role here is to be the two way conduit, but we also need to engage the community with the same kind of passion for our own (Microsoft's) stuff that they have for Open Source. Anything less will appear fake.

More to come.

Anandeep

FOSS in India

anandeep — Tue, 25 Nov 2008 19:47:00 GMT

I am writing this during lunch at FOSS.IN, the most prominent Open Source conference for developers and FOSS advocates in India. FOSS.IN is in Bangalore, which is where I graduated high school. That was a long time ago - and Bangalore has been transformed from the "garden city" to the software capital of India. Lots more skyscrapers and cars on the road. And lots of software companies (including Microsoft) have development or R&D centers here.

It so happens that the venue for FOSS.IN is in the Indian Institute of Science (I.I.Sc) auditorium. I.I.Sc was right in the middle of the path I trudged from home to school, walking 12 miles uphill both ways (or so it seemed). I have seen the Indian cricket team there, when they were visiting the old auditorium. The new auditorium on the same site is nothing like the old one, it's actually pretty nice. I don't think they would allow me to perform the amateur play that I was forced into by my high school drama teacher at the venue any more!

The conference is all volunteer organized and does not have the polished feel of a larger conference in the US. I think this actually works in the favor of the conference organizers because people seem to interact more with one other than tends to happen at other conferences I have been to stateside.

But I have actually been disappointed that I have not seen more Open Source contributors from India gain prominence. That is kind of sad, since we have one of the (if not the) largest pool of software engineers in the world. So, I am keen to talk to the people involved and see what is happening around Open Source development in India.

There are a large number of people attending - my estimate is between 300 and 400 - with just three vendors: Sun, Noikia and VMWare(?).

So far I have attended the keynote by Harald Welte on how Embedded Linux was not taking the right path and why. I also attended a session that went through an "interview question" that was given to KVM contributor Amit Shah to solve by Qumranet prior to bringing him on board. It was to facilitate the migration of guests from an Intel based architecture host to an AMD based architecture host. The talk was fascinating.

I am starting to mingle and will have a number of stories from interacting with the people at the conference. So, more to come later.

Anandeep

Finally... dive-into-the-deep participation!

anandeep — Fri, 07 Nov 2008 16:54:00 GMT

Whenever people get to know that I work in the Open Source Lab at Microsoft, there are a few knee jerk questions they always have. The most common one is: "Microsoft has an Open Source Lab?", while another question often asked is: "So when is Microsoft actually going to be part of an Open Source project and actually participate?"

This question does not, of course, refer to our Codeplex projects, which are aplenty and Open Source, but rather about participating in an ongoing project that is not under Microsoft's umbrella. Something that organizations with strong Open Source credentials are behind: Mozilla, Perl, Linux, Apache, Samba, PHP, Eclipse ... you know, the usual suspects.

Fundamentally, as a developer, the answer I (truthfully!) gave: that we worked with all those communities and had actually done some important work with them, was not soul satisfying. You know as developers that one needs to jump in with both feet and actually stand up and be counted as being part of a project.

Given the kind of work we had been doing in changing realities both at Microsoft and in Open Source communities, I knew that it was just a matter of time before that happened. (My colleague Hank Janssen hates the word perception, and I would have used it here just to aggravate him, but that would not capture what I wanted to say. Darn!).

Well, that day is arrived - and I am excited beyond words to say that we (as Microsoft) will be participating in the Apache QPID project. This was announced by Sam Ramji today during his keynote address at the Apache Conference in New Orleans.

Apache QPID is middleware for message passing and is based on the AMQP standard. I will be the point person for our lab's participation in Apache QPID. Actually I should say "what used to be our lab" and which now has truly become the Open Source Technology Center at Microsoft.

In a previous life, I was with a small start up that did secure Web Services management. This was when the SOAP protocol sent unreliable, in-the-clear over the wire messages, and what we did was make SOAP a secure, guaranteed once and once only protocol between two Web Services end points without requiring any changes to web or application servers. That gave me some understanding of the complexities of message passing. Add the complexity of time constrained responses, huge volumes of data and interoperability between disparate systems and the technical problem becomes real juicy!

I often used to wonder why there weren't more successful Open Source messaging systems, since the primary message passing systems in use were proprietary and weren't built for interoperability. Imagine my surprise and joy when Tom Hanrahan (da boss) told me that Microsoft was joining AMQP and was considering participating in the Apache QPID project. I jumped on the opportunity to be involved with the project.

I want to reiterate that we will follow Open Source principles in being part of the Apache QPID project, and not antagonize people through typical big company execution. This means the following things to me:

We will approach the process of working in the QPID project as beginners in technology, process and relationships.
We will listen first in the forums, in conferences and events, and most importantly one-on-one with existing community members.
We will only do things that add value as validated by the QPID community
We will strive to be leaders in building QPID's future, but that leadership will be an invitation from the community

This all means that we will be slower of the mark than we would, say in one of our internal projects - but we would like to get it right.

For now I am going to download QPID and run it in the lab. Look, listen and linger in the forums, read the documentation, run the test suites, play around with APIs and learn how the joint runs (so to speak).

Looking forward to seeing you all there.

Is High Performance Computing naturally Open Source (ie. for tinkerers)?

anandeep — Wed, 18 Jun 2008 13:17:00 GMT

I have always been fascinated by clusters. Some people envision working with desktops or workstations when they think of “working with computers”. For me working with computers was always with a large collection of computers in a back room somewhere. And how cool if you could make all those computers collaborate with each other working to solve cool things like genome mapping, movie special effects, simulations of car crashes or simulations of molecules being formed!

So you can imagine I jumped at the chance to work with the Windows High Performance Computing team. This is the same team that builds Windows HPC Server 2008.

I think most of the people working in the team are from the “large collection of computers in back room somewhere” school. Would be really different in the Mac software division I assume!

I work with the Open Source Software Lab and we are all things “Open Source” to the rest of the company. The HPC Server team wanted us to make sure that their product played nice with Linux infrastructure and vice-versa. The usual suspects like AD, Samba, LDAP. CIFS etc were involved. We had to make sure that these recurrent interoperability themes were addressed in the HPC environment. I also got a chance to dig into ROCKS, OSCAR, MPI stacks and job schedulers etc etc.

This was a very rewarding experience not only for the technology exposure that I got but the pervasiveness of knowledge of Open Source within the team. They were far ahead of the other product groups in this regard and “got” the Open Source ethos. In fact, prior to my interactions with them they had released an open source MPI stack based on Argonne National Lab’s MPI implementation.

The other reason was that a lot of their customers were relentlessly open source! The conventional wisdom is that HPC applications and infrastructure require a lot of tinkering. Of course, there are some applications like FEM and CFD and that are well understood, but the general feeling was that complete control and access to the underlying infrastructure is a must for getting the most performance out of a cluster. And performance is the main thing in “High Performance Computing”.

Linux is seen providing that access by HPC customers and there is a large base of Linux for HPC in academia, the national labs and other institutions that use large clusters for doing their thing.

But is this really true?

I think that HPC has gone through a typical evolution – it starts with a few people who have a pressing need. There is a cross disciplinary team formed that builds software to do their job and a community grows around it. The community reaches critical mass and people start making building tools to make it more convenient. ROCKS is an example of this. Great skill, knowledge and ability is needed to get the job done.

However, these skilled people now become overloaded. The tools and the infrastructure that they created become so popular that everyone, including people who do not have background that was assumed before, wants to use it for their ends. So the community responds and builds standardized, easy-to-use infrastructure pieces that start to fit seamlessly together. Some control is lost, but ease-to-use is the primary focus.

The infrastructure for HPC has reached that stage (ROLLS with ROCKS). Windows HPC Server 2008 is built for this ease-of-use too.

However, the applications have not reached the stage of ease-of-use. They have to be coded with a lot of domain knowledge and have to built from scratch to truly scale while running on clusters. That means that the application writers demand more control of the underlying infrastructure and want more access to it than the users and maintainers want.

I am going out on a limb and making a prediction here – soon end users will be able to specify instead of coding applications, be it genome comparison or physics simulation. This is similar to accountants finding spreadsheets. There will probably be a few different models for different types of applications but that stage will come pretty quickly.

The infrastructure that runs these user-specified applications will be adaptive and will take these specifications and automatically tune them for high performance on the clusters.

This is where the perception of needing control to the lowest levels will be moot. The best adaptive infrastructure will be the one adopted.

Bold enough for you?

OSCON 2007 Trip Report

anandeep — Fri, 17 Aug 2007 15:35:00 GMT

My overall impression was that OSCON was lower key than last year. There seemed to be fewer booths in the Exhibition floor and less palpable excitement in the venue. A lot of people were complaining about the quality of the tutorials and the talks. Or it may just be that this was my second time around attending OSCON and it didn’t have the same quality of excitement for me compared to the very first time!

Microsoft’s name was on the bag that they gave us. That was good to see. Also Microsoft’s name was on the marquee as a diamond sponsor. The other sponsors were all big names. Intel was a diamond sponsor as was Zimki. Zimki? Wait a minute, Zimki isn’t a big name! How did an open source web development platform company spend so much money? According to their web site “Zimki is Fotango's flagship product, a JavaScript web application development platform that enables developers and businesses to build and deploy web apps quickly without the need for tools, equipment or hosting. With no set-up fees and subscription charges,users simply pay for the computing resources that they consume on a utility basis.” Their blog also states that they were holding back their plans to roll out Zimki at OSCON . So a small company spent all this money and had no presence at OSCON. Shades of the dot com boom. Hopefully open source web development platforms are not today’s dot coms!

I attended two keynotes (that I remembered an hour after I attended them) – one involved Eben Moglen, the lawyer dude for the FSF, tearing into Tim O’Reilly. Tim O’Reilly was asking Eben questions about whether GPL V3 gave Google a free ride. Eben went into how he wanted to protect freedom and how the Open Source people had “wasted ten years” not making “freedom” the issue. But as persuasive and articulate as Eben is, I think he left the feeling with the audience that the FSF had given in to Google to get GPL V3 passed. Eben even used the words “diplomacy” to describe the process of building GPL V3.

Some dramatic moments such as Eben pointing to Tim and then to the sign behind him and saying,” Take down that sign with YOUR name on it and put “Freedom” there instead”. Tim even went to say to him – “I will ignore the personal attacks”. To which Eben said – “This is not a personal attack, it’s an invitation to diplomacy”. Wow! I could have watched a musical and not had so much drama.

Don’t get me wrong, I admire the FSF’s devotion to its cause. They have been consistently practicing what they preach. I am glad that there are people like them to keep even big companies like Microsoft honest. But I feel let down with their inconsistency with respect to Google.

The other keynote that I enjoyed was Bill Hilf. Bill is our GM, and we know all the stuff he said in speech. We try not to say one thing and practice another (surprise, surprise!). But he said it all so clearly, and so well in front of a largely skeptical audience. It was a masterful and engaging performance. Even when ambushed by Nat Torkington with questions that were not on the agenda, he didn’t lose his verve and kept on emphasizing what we do instead of what we say. It feels great to have a official place in the Microsoft firmament @ www.microsoft.com/opensource . Wait! Or was it nice to be the exclusive open source guys in such a big company? (From a purely selfish point of view)

I attended a few talks. One was Sam Ramji’s talk about our Interop efforts in virtualization, identity and management. To tell you the truth, I went because Sam is my boss. But I stayed because he did a great job of simplifying and presenting the information. I learnt and reinforced a ton about virtualization and the Interop challenges around it. I now have a firm grasp on the subject – not isolated chunks of information unconnected to each other. Ok Sam, when are you doing a talk on High Performance Computing? J

The other significant talk I attended was about Hadoop. Hadoop is an open source software platform that lets one easily write and run applications that process vast amounts of data. Or basically it implements Google’s MapReduce. According to Google’s original paper on MapReduce - “Programs written in the MapReduce style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system”.

Hadoop has been created by Doug Cutting, the creator of Lucene and Nutch. In order for him to do MapReduce effectively he had to do a “Google File System (GFS)” like system called “Hadoop Distributed File System (HDFS)”. HDFS was originally built as infrastructure for the Apache Nutch web search engine project.

Now, Yahoo is using Hadoop and HDFS for its back end. There is now an open source implementation of Google’s Open Source based proprietary stuff. If the community get’s behind it, it may be that the truly open source stuff may outshine the open source but proprietary stuff. Makes your head spin.

Oh, and why the name Hadoop? Doug Cutting’s son’s favorite elephant was named Hadoop. A name that came from the son’s imagination. I love Open Source!

An Interview with Ruby on Rails Core Team Member: Michael Koziarski

anandeep — Tue, 03 Apr 2007 20:49:00 GMT

Michael Koziarski (a.k.a) Koz is one of the core group of about 12 people who holds the keys to the code repository for the Rails framework (also known as Ruby on Rails). They’re all listed on the Rails core page with name and mug-shot. (I checked, a guy who looks like the Michael I interviewed has his mug shot on the page!).

According to his bio there “Michael Koziarski (nzkoz) is a software consultant based in Wellington, New Zealand. After a successful stint as an enterprise Java developer, he switched to rails shortly after the first public release. He’s a contributor to The Rails Way and maintains a personal blog.” That’s as good a intro as any so we thought we wouldn’t improve on it!

Michael was invited to the Microsoft Technical Summit and I was able to spend some really entertaining and educational time with him. I even showed him the lab! I think he may have enjoyed his time here as he wrote about us in a blog entry.

I enjoyed talking to him – and he stayed to talk to us about Rails, the dev process and other stuff despite being very close to missing his flight!

Anandeep

Video: An Interview with Ruby on Rails Core Team Member: Michael Koziarski

Decoding Liberation: An interview with Samir Chopra

anandeep — Fri, 30 Mar 2007 23:32:00 GMT

Samir Chopra is an Associate Professor of Computer and Information Science at the City University of New York (CUNY), Brooklyn. He has a Master’s degree in Computer Science and a Doctorate in Philosophy and is interested in the intersection of politics and technology – specifically information technology.

Samir and I share more than our interest in Open Source (or Free Software as Samir prefers to call it!). Our fathers trained as Air Force pilots in the Indian Air Force in their teens and knew each other very well. In fact Samir (being the renaissance man that he is) has co-authored one of the most successful books on Indian Air Force history “The India-Pakistan Air War of 1965”. We contribute to an “open source” site on Indian armed forces history – you can read one of my articles (shameless plug!) on air war history here.

Samir is a Free Software proponent and recently he was invited to the Microsoft Technical Summit to exchange ideas with a wide audience from both within Microsoft and from all around the globe. I jumped at the opportunity to interview him about his forthcoming book “Decoding Liberation” and to debate with him our views of how software would evolve.

Samir’s site is http://www.sci.brooklyn.cuny.edu/~schopra/ (his co-author, Scott Dexter's site is http://www.sci.brooklyn.cuny.edu/~sdexter) and he writes a blog on Free Software which has the same title as his book.

An introduction to the book is here.
Routledge has also given us permission to post the first chapter of this book which is available now for pre-order.
Video: Decoding Liberation: An interview with Samir Chopra

Cross-Platform Access to Codeplex Compliments of our Friends at Teamprise

MichaelF — Tue, 20 Mar 2007 22:40:00 GMT

Awhile ago we interviewed Martin Woodward from Teamprise regarding their suite of client applications that provide cross-platform access to Visual Studio Team Foundation Server.

Today Teamprise announced that they are providing complimentary licenses for the Teamprise Client Suite to developers who plan to use the license to access Codeplex. This allows projects with developers on the Eclipse IDE along with anyone on the Mac, Linux or Unix platforms to use Codeplex for hosting their projects.

To obtain access to the client go to the sign-up page.

To provide some additional information on this release and Teamprise, Anandeep sat down with Martin to catch up.

Video: Cross-Platform Access to Codeplex Compliments of Teamprise

Martin's Blog about this announcement can be found here.

Who really needs to gather crash information and what do they need to do with it?

anandeep — Wed, 07 Mar 2007 04:06:00 GMT

I just got back from Cambridge (in the United Kingdom, not the one by the Charles river) from the Microsoft Research / Technische Universitat Darmstadt “Reliability Analysis of System Failure Data” conference.

Microsoft Research has a very nice lab near Cambridge University. This was my first visit to Cambridge and I was able to drink in (literally!) some of the local color. I went to the pub (called the “Eagle”) where Watson & Crick had their “aha!” moment about the double helix structure of DNA. I also visited the pub next to Queen’s College called the “Anchor” which Pink Floyd’s Syd Barrett used to frequent.

But it was not all pub-crawling, we had some serious stuff to deal with at the conference. The objective of the conference was to bring academia and industry together to deal with a problem in the field of reliability analysis. The problem is that industry (there were representatives from Sun, Cisco, IBM and of course Microsoft) had the failure data but NOT the models and techniques for solving the problem overall. Academia had the models and techniques but NOT the kind of failure data that it needs to solve the overall problem. This conference was an attempt to get the two sides together and find a solution to this conundrum.

After we had presented our position papers (our lab’s paper is here), we split into workshops on Data Collection, Data Repositories and Data Analysis. The idea was to come up with the “next step” in taking reliability analysis of system failure data forward.

In the Data Collection section, we were stuck for a bit. We were trying to look for compelling reasons for data collection by end-users (rather than by the software makers like Microsoft, Sun or IBM). What reason would an IT department have for implementing mechanisms to collect failure data?

At first the reasons seemed obvious – to monitor for failure and to correct defects of course. But then the representatives from the software makers spoke up and said that they did collect failure data and were using it exactly for the purpose of correcting defects. Vince Orgovan from Microsoft stated in his paper that almost 400 million PCs provide data to Microsoft. Not only was the data available but Windows XP supported corporate error reporting in exactly the same way that it supported error reporting to Microsoft. It just needs changing a few registry keys to do this. The Windows debugger “!analyze” can be used on the error data, much as it is used internally.

This took the wind out our collective sails. If shipping software was providing all these mechanisms what could we suggest as a next step that had compelling value? Most corporations would like to leave the job of correcting defects to the software makers (proprietary or open source) anyway! The software makers were in a much better position to look across many deployments and correct defects in the software.

The only compelling reason we could come up with to build a mechanism for data collection was to help with deployment on the user side. The data collection mechanism would collect failure data during the preliminary testing. This data would then be fed into a model that could be used to judge the maturity level of the deployment. Kind of a CMM (Capability Maturity Model) for reliability. We even suggested that we have an ITIL management practice around this. This would potentially allow the ITIL model to not only give good qualitative measures like it does today but to quantify the reliability of a deployment.

This in itself is a very useful thing to have, but I cannot believe that it is the only reason that we would collect failure data at the end user level. Let me know if you think of any others.

Open Source would have much the same issues but for the fact that there is not a central organization that collects all this failure data. The situation in Open Source may be the reverse of the situation for proprietary software makers in that the failure data is collected at the IT organization level and not centrally. How does this failure data really result in code defect corrections? I guess that it is either pre-analyzed and submitted as a bug or people patch their own instances of the source code. But my opinion is that eventually open source software systems will have to build central repositories of failure data in much the same way that commercial software vendors have built them.

Conversations are what it's all about...LinuxWorld OpenSolutions Summit

anandeep — Thu, 15 Feb 2007 23:18:00 GMT

I am writing this from the Big Apple. The Linuxworld Open Solutions Summit is in New York and running from February 13th through February 15th. I questioned the wisdom of holding a conference in February on the East coast rather than the milder shores of the West Coast - since I arrived the middle of a snowstorm after a day's delay caused by flight cancellations. But once I got here the bright lights on Broadway dissipated all doubts!

The Open Solutions Summit is a much smaller conference than Linuxworld conference (which I covered in a previous blog). But it has many more IT professionals who show up than vendors. This makes for more OSCON-like ambience, except that it is IT focused rather than developer focus.

Microsoft held an evening mixer in the conference hotel (the Marriot Marquis) in the cool rotating restaurant on top of the building. The NYC skyline was spectacular, but what I really enjoyed were the conversations with the people.

I got to chat with people such as Jeremy White of CodeWeavers. When he asked me if I knew what WINE was, I answered "Wine is Not an Emulator?". I think Jeremy appreciated me knowing that!

Lincoln Durey from EmperorLinux was there and told me about the new tablet laptop that used Jarnal based handwriting recognition. He did say it wasn't as good as the Windows Tablet PC handwriting recognition! I think OneNote is the killer app for the tablet. Is there a open source application like OneNote out there? What was cool about the tablet was that it has a WACOM film on top of a regular LCD that talks to a serial port. He even had a toughbook with the same stuff on it.

It was a pleasure to meet Tony Luck who is a Principal Engineer at the Open Source Technology Center at Intel. I think we could identify with each other because we worked in similar organizations (OSTC / OSSL).
He is a kernel maintainer - which means he is royalty as far as Open Source goes. He works on making sure that Intel chip features work with and are fully utilized by Linux in his day job. Way cool - and he's a nice guy to boot.

Another person I met was Gianluca Brigandi who is with a small company called Novascope in Argentina. His company is building out a product based on Java Open Single SignOn (JOSSO) standard besides providing open source code for JOSSO. He is very concerned with interoperability and was keen to hear about Microsoft's open source efforts. I pointed him to http://www.codeplex.com/ and probably will be interacting with him on interoperability further. We spoke about everything from APIs for developers to use to interact with identity standards to open source licenses and what they mean for companies like his.

What brought home the power of conversations to me was a conversation with Rob Donath from SpikeSource. He approached us with an interoperability and packaging questions about Windows and open source software in a virtualization environment. And he said that he wouldnt have known who to talk to if we hadn't been reaching out to the Open Source community through events like this and Port 25.

I love my job! :-)

Can Open Source Scratch an Itch Before It Happens?

anandeep — Tue, 06 Feb 2007 00:25:00 GMT

I am an avid reader of Joel On Software – I find his insights great and very revealing.

I was reading a recent blog post by Joel entitled “The Big Picture” in which he has this to say about open source

“Open source doesn’t quite work like that. It’s really good at implementing copycat features, because there’s a spec to work from: the implementation you’re copying. It’s really good at Itch Scratching features. I need a command line argument for EBCDIC, so I’ll add it and send in the code. But when you have an app that doesn’t do anything yet, nobody finds it itchy. They’re not using it. So you don’t get volunteers.”

This was in the context of the review of the book “Dreaming in Code” by Scott Rosenberg. Scott was on campus at Redmond a few days ago, talking about his book. The book is a “Soul of a New Machine” type look into an open source startup that was trying to make a PIM (Personal Information Manager) code-named Chandler – and the travails that startup went through. Scott begins his introduction by saying ”… the art of creating it (software) continues to be a dark mystery”, so you can guess the whole thing didn’t go very well!

This got me thinking – what is open source really good for? Is it only good for copycat or scratch-an-itch type of software? Could there be a limit to what open source process can achieve in terms of software artifacts?

First of all – being “copycat” or “scratch-an-itch” type of software is not bad at all. It can be argued that Firefox falls into the former category being based on closed source browsers but while gaining feature parity with other browsers it added new features. I think this was the benefit of the general public because not only did Firefox get better in the spirit of competition other browsers (including Internet Explorer) got better. As for the latter category, what good is software if it isn’t working for its’ users or “scratching an itch”?

I think the point Joel was trying to make was that bootstrapping an open source project requires either a user need or the need for an alternative. A community is more easily built when there is a shared need for functionality or alternatives.

But something about that bothered me. After all isn’t Open Source all about the “love of the game”? Why wouldn’t a community want to do something that was experimental and didn’t have any immediate payoff? Coming from a university research environment, I knew there were people out there putting out experimental code into open source including everything from Robotics to the ALICE Educational Software Authoring System (I knew the person behind ALICE – Randy Pausch from Carnegie Mellon). ALICE has a pretty vibrant open source community behind it.

That said, all the top open source projects (based on a poll by O’Reilly) fall within Joel’s characterization.

So can futuristic experimental projects be developed using the open source process?

I think that the answer is yes. But these kinds projects cannot be developed in a pure open source community process like that of Linux. An institution like a university or a company has to bring to it critical mass. The US government paid for a lot of ALICE – before it could be put out there in a true community process.

BTW I just looked at the ALICE website – and Microsoft has also supported ALICE financially. That wasn’t the case back when I was at Carnegie Mellon – I remember thinking, “How is an educational software package which is a rage with art students going help the US defense department?”. Almost all its funding at the time came from DARPA (Defense Advanced Research Projects Agency)!

There has to be a lot of money/resources/people put into a software project to bring it to a stage so that a community sees that an itch is going to be scratched, and then gets on board.

I was chatting with Hank Janssen and Kishi Malhotra about the “top” open source projects and stated that the top open source project I wanted to see was a “Cloud OS” which wasn’t yet around. I was waiting for the day when a system call made on my laptop would kernel trap on a machine in a data center in India, without my knowing or caring to know which data center or which machine. Ruminating on this I postulated that some of the early components are already there with the Google File System and the Google Cluster Architecture. Then I realized that even though those were Linux based they were by no stretch of imagination open source!

Can the Linux Kernel be More Maintainable? Anandeep interviews Professor Stephen Shach

anandeep — Thu, 18 Jan 2007 20:15:00 GMT

Prof Stephen R. (Steve) Schach is an Associate Professor of Computer Science and Computer Engineering at the Vanderbilt University.

Port 25 met up with him while he was visiting Seattle, Washington in picturesque Kirkland, Washington on the shores of Lake Washington.

Steve (he hates being called Prof Schach!) believes in gathering data to make predictions. While he accepts that there may be interpretations of data he thinks gathering the correct data is paramount.

He credits Open Source Software with kick starting the Empirical Software Engineering movement saying “We could count the number of lines of code in gcc and Linux – we couldn’t do that with Windows 95!”.

In this interview we discussed empirical software engineer/computer science and some of the work Steve has been doing. This includes his work on the proportion of time that code is in bug fixing mode and his work on global variables in Linux.

The latter work was found to be controversial by the Open Source Community. Steve thought that all he was doing was counting the number of global variable in Linux vs BSD and stating that Linux had far more than is considered wise! This was surprising to Steve, but isn’t that much of a surprise to the people who know how much passion Open Source can generate!

Steve’s website is here http://www.vuse.vanderbilt.edu/~srs/ and you can find his publications on his website.

Video: Anandeep interviews Professor Stephen Shach”

Crash Data Collection and Analysis

anandeep — Tue, 19 Dec 2006 18:35:00 GMT

Archana Ganapathi is a Computer Science graduate student at the University of California, Berkeley.

Archana has been working in the area of Empirical Computer Science (which relies on real data rather than theory or simulation) and some of her research is on computer crashes. She worked on collecting data on Windows crashes and is in general interested in the idea of using real data to advance Computer Science.

Here home page is http://www.cs.berkeley.edu/~archanag/ . Her paper on Windows crashes is “Crash Data Collection: A Windows Case Study” and another interesting paper she has written is “Why do Internet services fail, and what can be done about it? “

The Open Data Repository link referenced in the video is http://institutes.lanl.gov/data/. This is a temporary link with a public data set as there is currently not an official link for the repository that will eventually be hosted by USENIX. We'll be sure to pass along the official link as soon as it is available.

-Anandeep

Video: Crash Data Collection and Analysis

Experimental or Production?

anandeep — Tue, 05 Dec 2006 18:33:00 GMT

There's two things people figure out about me (mainly because I tell them!) - one that I am crazy about airplanes and two that I love stirring controversy! And in this blog I get an opportunity to bring those two favorite things together.

There are two kinds of light or General Aviation airplanes out there - the "production/certified" airplanes (referred to as "Spam Cans") and "homebuilt/experimental" airplanes.

You probably have heard of the manufacturers of the "Spam Cans" - they have names like Cessna, Piper and Beechcraft. These are large companies with lots of engineers who mass produce airplanes and sell them to you if you part with large sums of money. They also give them to you in any color as long as it it's creamish. These are faithful, reliable if boring airplanes. Nothing wrong with them but they are not fun. They also make a lot of compromises in speed, manueverability, weight carrying ability or runaway length requirements - and usually don't excel in any of those criteria. They are the airplanes every commercial enterprise uses though. Almost everybody learns to fly in them. Some of them are bush planes in Alaska and Africa and are the lifeline of a lot of people there - nothing to sneeze about! Below is a picture of a Cessna 150 (my ex-airplane) that I used to build up hours and tour the Pacific Northwest. One of the "Spam Cans" but beloved nevertheless.

Then there are people who accept no compromises. They decided they didn't want to accept hired engineer’s opinion of the best design. They went to work designing their own planes and then offering plans or kits so that other people could build them. One of the early pioneers of this was Burt Rutan, now famous as the designer of the first private spaceplane "Spaceship One", who offered a kit for an airplane christened "VariViggen" that had its tailplanes in front (in a configuration called a "canard"). It could go faster than any production plane on much less power and was stall proof- which meant that it was a lot safer than the regular planes. The other success story is Richard VanGrunsven - whose company "Vans Aircraft" has built a family of aircraft called "RV"s (there is still some debate as to whether that means "Recreational Vehicle" or "Richard VanGrunsven"). As of the time of writing there were 4861 RVs built and flying - more RVs ship every year than any commercial light plane manufacturer in the world can produce! These aircraft are speedier, more manueverable, have better weight carrying ability or have less runaway length requirements than comparable production aircraft with the same horsepower. These planes are known as “homebuilts” or “amateur-built” or “experimental”. The “experimental” title comes from the placard that they have to exhibit by law – this is also the placard all manufactured planes and military aircraft have to exhibit till the time they get certified by the FAA. It doesn’t necessarily mean that the aircraft is an experiment in progress.

Ok -where's the controversy? I am saying that the Open Source Software movement is like the experimental aircraft movement and make an assertion that commercial software companies are like production aircraft companies.

After all there is a community among experimental builders that rivals the OSS community. They share ideas freely, give each other plans for improvements and are very loyal and committed to the cause. One instance of such a community is "Van's Air Force". These communities and the experimental manufacturers also are on the cutting edge of technology, pioneering cheap "all glass" computer screen instrumentation in light planes among other things. Like Linux, most successful experimental aircraft have a solid “kernel” that is built and maintained one way but like Linux the “distributions” abound based on builder’s personal preferences. For instance from the very successful Vans RV-4 came the Harmon Rocket. Ubuntu and RedHat aren’t THAT different! J

But its not all applehood and mother pie. Building these airplanes (even if you do all the work yourself) is not much cheaper than buying a general aviation airplane. You do have to build them to get all the advantages and it is considered a truism in the community "build if you like to build, buy if you like to fly!". Which means that it takes serious commitment to build one of these things and you better take a lot of pleasure in just the act of building. Of course, you could buy one of these already built, but would you trust the builder? Build quality is very variable! Certification standards are conservative and lengthy for a reason - a small variation can result in a catastrophic outcome. These aircraft are also more demanding to fly than the boring old "Spam Can".

I fly a Cessna 182 for the Civil Air Patrol - and I wouldn't want to fly an experimental airplane that I myself hadn't built. (Even if I built one - would I?). Because we fly in the mountains with heavy loads (survival gear, direction finding equipment, and individuals who are - shall we say - "weight challenged"). I know it won't do things spectacularly but will do its standard thing as long as I follow the manual. Its heavy on the controls, isn't that fast and has a high fuel consumption - but it can carry a heavy load and land in a reasonable distance. And I can be sure that all the improvements that Cessna has mandated have been incorporated, since it would be illegal not to. Not so for the experimentals since the builder not the kit manufacturer is the legal manufacturer and can make his own decisions! It isn’t the “experimental” placard that scares me, its’ the fact that I would have to form a judgment on my own on every INSTANCE of what is fundamentally the same design.

Am I taking the analogy too far? To be truthful, I don't know - but it is certainly worth thinking about!

Now if you send e-mail to Sam Ramji telling him how much you liked this blog - I might be able to afford a house in the Puget Sound Area and this RV-8 kit that I want at the same time! :-)

Learning to Read!

anandeep — Fri, 17 Nov 2006 01:39:00 GMT

One of the great things about my job at the Open Source Software Lab (OSSL) here at Microsoft (besides being able to work with both Linux and Windows!) is that I get to go computer science research conferences. I try not to attend the purely academic ones, but the ones in which both industry and academic research issues are addressed.

I just got back from ISSRE (pronounced “is-ree”) i.e. the 17th IEEE International Symposium on Software Reliability Engineering, 2006. This conference talks about everything that impacts the reliability of computers – this includes everything from “drivers of reliability” to “testing to ensure reliability” to “doing static analysis of programs”.

Skeptical that anything they talk about here would be useful to y’all? Well, think again! They have all kinds of practical advice on doing things right. The talks I really enjoyed included

empirical evidence on the positive impact of using assertions during development (a very development oriented talk)
considering stabilization time of an application (i.e. stability during installation and immediately after) as part of a reliability metric
the use of “operational profiles” to reduce the number of test cases by over 30% without significantly impacting the reliability of the tested product

Only one of the above talks was from an academic institution, the other two were based on experience with software being widely used in the consumer and application server space.

The one thing that I enjoyed the most was a tutorial on “Software Productivity and Reliability – Tools and Techniques” given by Prof S C Kothari of Iowa State University. The tutorial title is appropriate but I think what it should have been is “Learn to Read Programs Properly!”

Kothari believes that a lot of attention has been paid to what he calls “Program Writing” – developers tools and such. This has resulted in the creation of very complex software artifacts. Most real world applications today are built on these already built complex software systems.

The problem is that almost all academic institutions and programs focus on the inventive aspects of programming. This means that they teach algorithms and techniques assuming that everything will be written from scratch. Real life is of course never like this – it is difficult if not impossible to be a computer software professional these days and work just with your own code. More often than not, most developers have to wade through other people’s code to understand, use or modify it. Developing software today involves a lot more than just writing it.

The skills to “read programs” are acquired the hard way – and sometimes never fully mastered. Kothari suggested that there needs to be an emphasis on program reading in training and that tools need to be built to aid in reading programs and forming the proper mental model of them. The barrier to future software productivity is not machines or algorithms but human mastery of the complexity of the vast amount of critical software out there.

Program reading is not easy, as most people in open source know! This is due to

The complexity of the semantic analysis of the program – figuring out what the module is trying to do, is it part of the scaffolding put in place to support the execution or is it domain knowledge embodied in the module?
A lack of domain knowledge – how would a programmer know how a complex business or legal transaction needs to be done or a certain application level protocol executed– yet this information has to be embodied in the code written by the programmer.
Non-localized relations between software artifacts – the module does one thing in one context and another thing in the other – in some it has to maintain data integrity and in others it has to undo something that has occurred somewhere else in the program.

There are some tools that are available to assist in program reading such as CScope (BTW Hank Janssen of our lab wrote parts of CScope) but there has not been a lot of attention paid to WHAT program reading needs in order to address the complexity issues raised above. Kothari has a company Ensoft that provides some very cool tools to do the kinds of things that are needed for reading complex programs. The tools are based on abstractions that are used in program comprehension (there is a IEEE Conference on Program Comprehension held every year). Kothari illustrated one that he called “matching pair” (MP). Matching pairs are defined by a syntactic pattern – which could be artifacts (such as matching parentheses) or events ( such as locking or unlocking a resource). There are many types of such matching pairs and to make a program correct a matching pair can be defined with respect to control flow, data flow or both. A control flow matching pair means that a function f would need to be followed by a function f-inverse in EVERY execution path that the program could take. Looking through every execution path is hard (and it is proven that to do it via automated static analysis of programs is an intractable problem) – especially in something like the Linux kernel.

Using the tool that Kothari demonstrated – a call graph was generated and a “query language” defined over call graphs. Looking for matching pairs using the tool became unbelievably simple. This was just one of the things that can be done to reduce the complexity and time taken to figure out what a very complex program was doing.

I think this is a real breakthrough – and I am now a confirmed advocate of program reading. I am hoping to work with Prof Kothari to do some more stuff with this – I hope to share the results if I do end up doing that.

Why do I mention this on this forum? This is something that open source developers and IT Pros have been doing for a long time. Open source developers have a culture wherein a lot of code reading is encouraged. And IT Pro’s have to constantly update and upgrade scripts that they use to control and run their infrastructure. The cultural advantage lies with open source developers and IT Pros but given the complexity of software is increasing exponentially everyone could do with a little help