Port 25: The Open Source Community at Microsoft : Software Reliability

Who really needs to gather crash information and what do they need to do with it?

anandeep — Wed, 07 Mar 2007 04:06:00 GMT

I just got back from Cambridge (in the United Kingdom, not the one by the Charles river) from the Microsoft Research / Technische Universitat Darmstadt “Reliability Analysis of System Failure Data” conference.

Microsoft Research has a very nice lab near Cambridge University. This was my first visit to Cambridge and I was able to drink in (literally!) some of the local color. I went to the pub (called the “Eagle”) where Watson & Crick had their “aha!” moment about the double helix structure of DNA. I also visited the pub next to Queen’s College called the “Anchor” which Pink Floyd’s Syd Barrett used to frequent.

But it was not all pub-crawling, we had some serious stuff to deal with at the conference. The objective of the conference was to bring academia and industry together to deal with a problem in the field of reliability analysis. The problem is that industry (there were representatives from Sun, Cisco, IBM and of course Microsoft) had the failure data but NOT the models and techniques for solving the problem overall. Academia had the models and techniques but NOT the kind of failure data that it needs to solve the overall problem. This conference was an attempt to get the two sides together and find a solution to this conundrum.

After we had presented our position papers (our lab’s paper is here), we split into workshops on Data Collection, Data Repositories and Data Analysis. The idea was to come up with the “next step” in taking reliability analysis of system failure data forward.

In the Data Collection section, we were stuck for a bit. We were trying to look for compelling reasons for data collection by end-users (rather than by the software makers like Microsoft, Sun or IBM). What reason would an IT department have for implementing mechanisms to collect failure data?

At first the reasons seemed obvious – to monitor for failure and to correct defects of course. But then the representatives from the software makers spoke up and said that they did collect failure data and were using it exactly for the purpose of correcting defects. Vince Orgovan from Microsoft stated in his paper that almost 400 million PCs provide data to Microsoft. Not only was the data available but Windows XP supported corporate error reporting in exactly the same way that it supported error reporting to Microsoft. It just needs changing a few registry keys to do this. The Windows debugger “!analyze” can be used on the error data, much as it is used internally.

This took the wind out our collective sails. If shipping software was providing all these mechanisms what could we suggest as a next step that had compelling value? Most corporations would like to leave the job of correcting defects to the software makers (proprietary or open source) anyway! The software makers were in a much better position to look across many deployments and correct defects in the software.

The only compelling reason we could come up with to build a mechanism for data collection was to help with deployment on the user side. The data collection mechanism would collect failure data during the preliminary testing. This data would then be fed into a model that could be used to judge the maturity level of the deployment. Kind of a CMM (Capability Maturity Model) for reliability. We even suggested that we have an ITIL management practice around this. This would potentially allow the ITIL model to not only give good qualitative measures like it does today but to quantify the reliability of a deployment.

This in itself is a very useful thing to have, but I cannot believe that it is the only reason that we would collect failure data at the end user level. Let me know if you think of any others.

Open Source would have much the same issues but for the fact that there is not a central organization that collects all this failure data. The situation in Open Source may be the reverse of the situation for proprietary software makers in that the failure data is collected at the IT organization level and not centrally. How does this failure data really result in code defect corrections? I guess that it is either pre-analyzed and submitted as a bug or people patch their own instances of the source code. But my opinion is that eventually open source software systems will have to build central repositories of failure data in much the same way that commercial software vendors have built them.

Can the Linux Kernel be More Maintainable? Anandeep interviews Professor Stephen Shach

anandeep — Thu, 18 Jan 2007 20:15:00 GMT

Prof Stephen R. (Steve) Schach is an Associate Professor of Computer Science and Computer Engineering at the Vanderbilt University.

Port 25 met up with him while he was visiting Seattle, Washington in picturesque Kirkland, Washington on the shores of Lake Washington.

Steve (he hates being called Prof Schach!) believes in gathering data to make predictions. While he accepts that there may be interpretations of data he thinks gathering the correct data is paramount.

He credits Open Source Software with kick starting the Empirical Software Engineering movement saying “We could count the number of lines of code in gcc and Linux – we couldn’t do that with Windows 95!”.

In this interview we discussed empirical software engineer/computer science and some of the work Steve has been doing. This includes his work on the proportion of time that code is in bug fixing mode and his work on global variables in Linux.

The latter work was found to be controversial by the Open Source Community. Steve thought that all he was doing was counting the number of global variable in Linux vs BSD and stating that Linux had far more than is considered wise! This was surprising to Steve, but isn’t that much of a surprise to the people who know how much passion Open Source can generate!

Steve’s website is here http://www.vuse.vanderbilt.edu/~srs/ and you can find his publications on his website.

Video: Anandeep interviews Professor Stephen Shach”

Crash Data Collection and Analysis

anandeep — Tue, 19 Dec 2006 18:35:00 GMT

Archana Ganapathi is a Computer Science graduate student at the University of California, Berkeley.

Archana has been working in the area of Empirical Computer Science (which relies on real data rather than theory or simulation) and some of her research is on computer crashes. She worked on collecting data on Windows crashes and is in general interested in the idea of using real data to advance Computer Science.

Here home page is http://www.cs.berkeley.edu/~archanag/ . Her paper on Windows crashes is “Crash Data Collection: A Windows Case Study” and another interesting paper she has written is “Why do Internet services fail, and what can be done about it? “

The Open Data Repository link referenced in the video is http://institutes.lanl.gov/data/. This is a temporary link with a public data set as there is currently not an official link for the repository that will eventually be hosted by USENIX. We'll be sure to pass along the official link as soon as it is available.

-Anandeep

Video: Crash Data Collection and Analysis

Experimental or Production?

anandeep — Tue, 05 Dec 2006 18:33:00 GMT

There's two things people figure out about me (mainly because I tell them!) - one that I am crazy about airplanes and two that I love stirring controversy! And in this blog I get an opportunity to bring those two favorite things together.

There are two kinds of light or General Aviation airplanes out there - the "production/certified" airplanes (referred to as "Spam Cans") and "homebuilt/experimental" airplanes.

You probably have heard of the manufacturers of the "Spam Cans" - they have names like Cessna, Piper and Beechcraft. These are large companies with lots of engineers who mass produce airplanes and sell them to you if you part with large sums of money. They also give them to you in any color as long as it it's creamish. These are faithful, reliable if boring airplanes. Nothing wrong with them but they are not fun. They also make a lot of compromises in speed, manueverability, weight carrying ability or runaway length requirements - and usually don't excel in any of those criteria. They are the airplanes every commercial enterprise uses though. Almost everybody learns to fly in them. Some of them are bush planes in Alaska and Africa and are the lifeline of a lot of people there - nothing to sneeze about! Below is a picture of a Cessna 150 (my ex-airplane) that I used to build up hours and tour the Pacific Northwest. One of the "Spam Cans" but beloved nevertheless.

Then there are people who accept no compromises. They decided they didn't want to accept hired engineer’s opinion of the best design. They went to work designing their own planes and then offering plans or kits so that other people could build them. One of the early pioneers of this was Burt Rutan, now famous as the designer of the first private spaceplane "Spaceship One", who offered a kit for an airplane christened "VariViggen" that had its tailplanes in front (in a configuration called a "canard"). It could go faster than any production plane on much less power and was stall proof- which meant that it was a lot safer than the regular planes. The other success story is Richard VanGrunsven - whose company "Vans Aircraft" has built a family of aircraft called "RV"s (there is still some debate as to whether that means "Recreational Vehicle" or "Richard VanGrunsven"). As of the time of writing there were 4861 RVs built and flying - more RVs ship every year than any commercial light plane manufacturer in the world can produce! These aircraft are speedier, more manueverable, have better weight carrying ability or have less runaway length requirements than comparable production aircraft with the same horsepower. These planes are known as “homebuilts” or “amateur-built” or “experimental”. The “experimental” title comes from the placard that they have to exhibit by law – this is also the placard all manufactured planes and military aircraft have to exhibit till the time they get certified by the FAA. It doesn’t necessarily mean that the aircraft is an experiment in progress.

Ok -where's the controversy? I am saying that the Open Source Software movement is like the experimental aircraft movement and make an assertion that commercial software companies are like production aircraft companies.

After all there is a community among experimental builders that rivals the OSS community. They share ideas freely, give each other plans for improvements and are very loyal and committed to the cause. One instance of such a community is "Van's Air Force". These communities and the experimental manufacturers also are on the cutting edge of technology, pioneering cheap "all glass" computer screen instrumentation in light planes among other things. Like Linux, most successful experimental aircraft have a solid “kernel” that is built and maintained one way but like Linux the “distributions” abound based on builder’s personal preferences. For instance from the very successful Vans RV-4 came the Harmon Rocket. Ubuntu and RedHat aren’t THAT different! J

But its not all applehood and mother pie. Building these airplanes (even if you do all the work yourself) is not much cheaper than buying a general aviation airplane. You do have to build them to get all the advantages and it is considered a truism in the community "build if you like to build, buy if you like to fly!". Which means that it takes serious commitment to build one of these things and you better take a lot of pleasure in just the act of building. Of course, you could buy one of these already built, but would you trust the builder? Build quality is very variable! Certification standards are conservative and lengthy for a reason - a small variation can result in a catastrophic outcome. These aircraft are also more demanding to fly than the boring old "Spam Can".

I fly a Cessna 182 for the Civil Air Patrol - and I wouldn't want to fly an experimental airplane that I myself hadn't built. (Even if I built one - would I?). Because we fly in the mountains with heavy loads (survival gear, direction finding equipment, and individuals who are - shall we say - "weight challenged"). I know it won't do things spectacularly but will do its standard thing as long as I follow the manual. Its heavy on the controls, isn't that fast and has a high fuel consumption - but it can carry a heavy load and land in a reasonable distance. And I can be sure that all the improvements that Cessna has mandated have been incorporated, since it would be illegal not to. Not so for the experimentals since the builder not the kit manufacturer is the legal manufacturer and can make his own decisions! It isn’t the “experimental” placard that scares me, its’ the fact that I would have to form a judgment on my own on every INSTANCE of what is fundamentally the same design.

Am I taking the analogy too far? To be truthful, I don't know - but it is certainly worth thinking about!

Now if you send e-mail to Sam Ramji telling him how much you liked this blog - I might be able to afford a house in the Puget Sound Area and this RV-8 kit that I want at the same time! :-)

Unsexy Development

anandeep — Fri, 27 Oct 2006 14:27:00 GMT

I loved doing development in a research and university environment. You got to write cool code, prove new ideas, break new ground and generally ended up with bragging rights to say “I did an image recognition algorithm on a multi-layer architecture implementing reactive and planning parallelism on an autonomous robot!” The code had to work on your workstation or maybe on a demo machine once. Once you wrote the code, the only people who touched the system were hapless graduate students implementing the next big idea. They had to come to you and you could then dazzle them with your insight! This was “sexy development”!

When I moved to industry and wrote software for day to day use – things changed. Now you had all those people with “manager” titles telling you what to do, and those people called “testers” who told you why your code sucked (you couldn’t logically argue your way out of that because the weasels usually had proof)!. Of course being consummate professionals you adapted. You got the religion of “bullet proof code” and worked on making sure the testers only had “fit and finish” bugs filed against you. Which the intern could work on. That was still fun - a different challenge maybe not as “pure” as designing a neat new algorithm but pretty good nevertheless!

You got past the testers but when they integrated the components that you had bullet-proofed to run end-to-end or user acceptance tests, unexpected stuff happened. Who would have thought that they would configure the machine that way or that another non-surface component could pass you null strings. Now you had to plan not only for the testers – but also for other developers and those pesky sys admin guys. How did they become sys admins? They couldn’t tell a polynomial solution from a log n solution anyway! But being nothing if not adaptable you adapted. You now built bullet proof AND idiot proof code. (My father, a military pilot and flight instructor, when teaching flight safety used to say “Nothing is foolproof because fools are so ingenious!”). It got a little boring at times but you still had the satisfaction of building something that was “engineered”.

I thought I had shipped the product but I found couldn’t sit back and relax. The support guys were making insinuations against my code. It didn’t work they said – and you hadn’t put in the right level of granularity in the logs for them to do a diagnosis. This had nothing to do with Computer Science – any bozo could write stuff to the log. Why didn’t the intern do it? What do you mean he can’t make sense of my code? Yeah, I do know my code best. I guess it’s the right thing to do. Certainly not as fun as designing, bullet proofing and idiot proofing new code but good supportability is “sine qua non” for a well done project!

Is that the end of it? No, further design and coding needs to be done for making software more manageable, to make the logs more systematic, to make sure that the product works when its deployed to multiple configurations, that it performs well and fails gracefully.

Unless you specialize in a certain aspect of manageability, reliability or diagnosis – this is not “sexy” development. I probably wouldn’t get as much satisfaction from designing event logs as I would from designing a new search algorithm.

I was getting paid to do all this (ok, so it was my own startup but I was getting paid in VC money!) and it was still very hard. We did do it but it took lots of coaxing of our developers to pay attention to this. They all preferred to work on the next release that had all the sexy features. Even though they knew that to make the startup successful and still have a job, the unsexy stuff needed to be done and done RIGHT!

When you are working for the “love of the game” and not money, like in Open Source – who coaxes you? Who does the unsexy stuff? Are there enough people who specialize in the esoteric aspects of event logs, that this is not a problem? Or do users who need the feature “just do it” and add the code to the community version? Or are things slipping through the cracks?

I did a sweep of the usual suspect Linux developer mailing lists and found that there is concern about whether unsexy stuff gets done. Here is a typical comment that I saw

“I think that the only issue with Open Source boils down to this:

The things that nobody wants to do, but somebody has to.

Nobody wants to think about documentation. Or user interfaces. These things are hard, tedious, and a hell of a lot more boring than actually coming up with stuff to "make things work".” (from here)

Documentation is famously one of those things that is considered “unsexy” (well, ok in commercial software too). There are efforts like Grokdoc to make documentation of Open Source projects sexy by making it a priority. But the “who does unsexy?” issue is a real concern in Open Source.

We ran into a similar issue with event logs. You know the text stuff you write so that you can find out later what happened. At the lab we just did an investigation of whether we could tell if one of our boxes had crashed from the syslog and from console messages. We were a little taken aback by how many times we couldn’t tell what states the machine had gone through.

On doing some investigation we found that the most influential project that was addressing this issue, the Evlog project (most supported by IBM) has been quiet since 2004. This code is used internally within IBM but was not mainstreamed into the Linux kernel.

How does one get unsexy stuff like this into the Linux kernel so that is comparable to UNIX/VMS/Windows?

I contend that it is critical to Open Source that attention be paid to the event logs. They are critical in making any operating systems reliable. VMS/UNIX/Windows all went through the process of making their event logs more meaningful – and this has helped make them much more reliable.

We will be addressing this further in the next couple of weeks – keep tuned!

What does business readiness of software really mean?

jcannon — Tue, 18 Jul 2006 16:15:00 GMT

There is a buzz word floating out there – “business readiness”. It seems everyone (including people here at Microsoft) are trying to capture something important to organizations and people that are responsible for selecting, deploying and maintaining software for businesses. What does it really mean though?

Does it mean that a software package, distribution or application meets a benchmark? Does it mean that it is supportable without getting the Big Three consulting companies involved? Does it mean that all its functionality has been tested using regression test cases? Does it mean performance and scalability of the software meets needs? Does it mean that the software will be kept alive into the future by a vibrant community?

In my opinion it means all of the above.

So, what is the problem?

The problem is that business readiness is in the eye of the beholder! (Definition of beholder – the dude who happens to be holding the software when the music stops!)

I think this is a complex problem for two reasons

It is really hard to objectively measure “business readiness” for any piece of software. How would you assign a rating to software? What all would you need to consider? More about this later.
Organizations may decide for business reasons that “business readiness” is not the most important factor in adopting software.

In the good old days, when the startup I founded was in the throes of finding its first customers – a mid-sized company decided it liked us. We made B2B software – basically a marketplace engine /procurement engine / supply side engine. The customer decided that using our software would be advantageous for them because our engine could be easily adapted to run many different models that they wanted to try out, at little cost. They knew we were still early in our product cycle but they were willing to act as guinea pigs for the business advantage that they would gain. They also benefited from the fact that they wouldn’t have to hire the engineers or the operational support people to maintain the software and that their features would find priority in our product roadmap. I would say that our software was not completely “business ready” at the time (more like early beta quality) but that we were able to help attain the business objectives of the customer, while helping our product move towards business readiness.

I will concentrate on the first point – how do you objectively measure business readiness, and suggest a way to look at this. This is not a recipe, just a few thoughts on what we should pay attention to. Hopefully you can dive into the suggested links and find stuff that helps you evaluate the business readiness of some software you are considering.

There are many levels at which software must be evaluated – I assume here that the functionality of software is not the issue. Of course this is a big assumption, but the evaluation of software “features” is a better understood art than the non-functional aspects of software. (There is even a term called “non-functional requirements” while doing requirements and specifications – I never quite got my head around how something that didn’t function could be a requirement!).

What is the state of the art?
This is a question that is very hard to answer. For any piece of software the best most people can do is to compare it to its competitors in the marketplace. Most organizations that use open source would not have the luxury of having the commercial software to compare against. They would have to rely on word of mouth or other such imperfect evaluations. Even for most commercial software it is hard to get a good grasp of how that software compares with other software.

There are some organizations such as ISBSG (International Software Benchmarking Standards Groups) that is a non-commercial organization that collects data about software projects and quality. This data is submitted voluntarily by organizations that are software organizations all across the world in many different areas of software. The software for which such data is submitted is largely proprietary and commercial.

A good use of ISBSG data would be to compare defect density within an open source project to the benchmark for that kind of application within the ISBSG data. This would serve as an indicator of the quality of the open source software.

Other data available includes “cost per function point” for a project – this can help evaluate if the cost of the project/product to your organization is close to the “standard” price for good quality projects for the application area chosen.

Evaluating the software
Once the gold standard is known other evaluation criteria for the software at hand can be applied. The gold standard provides an quantitative upper bound in terms of number of defects and cost. But IT departments do not run on cost alone….

For open source software there are a number of evaluation benchmarks/certifications being made available. However, the criteria used to evaluate open source doesn’t exist in a vacuum – it is based on hard earned lessons in software development in general. I think that these criteria apply to all software whether open source or commercial software.

Some of the standards bodies out there include:

OpenBRR (www.openbrr.org)
This organization is proposing a standard model for rating open source software software. The criteria proposed include Functionality, Usability, Quality, Security, Performance, Scalability, Architecture, Support, Documentation, Adoption, Community and Professionalism. It will be interesting to see how this evolves – I think a lot of work needs to be done in this area and a promising start has been made. What OpenBRR has going for it is that it is an industry wide effort incubated by a respected university (Carnegie Mellon – please excuse my bias! J) and is committed to involving open source committed companies to the process of generating the model.
OSMM (Open Source Maturity Model) by NavicaSoft
This has been proposed by Navicasoft a professional services firm focused on open source, providing strategy, implementation, and training services to its clients. The OSMM model considers the following factors Software, Support, Documentation, Training, Integration and Professional Services. Practitioners calculate overall product OSMM scores for products. OSMM has a little bit more momentum, being around longer than OpenBRR, but is less comprehensive or “academic” in its approach – being tied to one company rather than being from an independent organization may not play in its favor.
OMMM (Open Source Maturity Model) by CAP-GEMINI
This is pretty comprehensive model which aims to generate a “score card” for open source products. It applies the criteria of Age, Licensing, Human hierarchies, Selling points, Developer community, Modularity, Collaboration with other products, Standards Support, Ease of Deployment, User community and Market penetration to generate the score card. Since the model has been developed by a consulting organization there is well framed process to apply this model. They have recently moved the project to the www.seriouslyopen.org repository.

There is nothing stopping you from considering criteria from each of those models to evaluate the “business readiness” of the software you are concerned with. I suspect that any good model will show comparable results, or the discordant models will fall by the wayside!

Show me the money
In their “Expert Letter” ,CAP Gemini - developers of the OMMM model, try to make the point (somewhat unconvincingly in my opinion) that because commercial software is developed differently from open source it has to be evaluated differently.

In my opinion, its all about the value the software provides. If the value can be derived down to dollars, that may be the best way of convincing people.

Khaled El-emam, recently interviewed in Port 25 has this cool ROI process that starts with software metrics such as number of bugs and ends up with a dollar calculation about how much a software product/project will cost the users in terms of “cha-ching”. Maybe every product needs to be put through this “business readiness” measurement!

I am now thinking about visualizing the business readiness using some cool graphic tools – “be the software, be the soooooooftware” (apologies to “Caddyshack”!)

Khaled El Emam, University of Ottawa, on the ROI of Code Quality

jcannon — Fri, 14 Jul 2006 20:01:00 GMT

Anandeep interviews Khaled El Emam, Professor at the University of Ottawa, Canada. Khaled's research looks very deeply at how bugs and software defects introduced in software development impact an organization after purchase and deployment. His assertion, the more bugs and issues in deployed software, the more downtime, support and maintenace costs are incurred post-purchase. Thus, evaluating code quality upfront can be a helpful tool in predicting ROI. The less bugs, the less code maintenance. Khaled's research looks at both commercial and open source software development projects and has uncovered some interesting insights.

Video: Khaled El Emam, University of Ottawa, on the ROI of Code Quality

Related Links:
ROI From Software Quality - Samples from Khaled's Book
Check out ROI From Software Quality (Amazon)
Learn more about Khaled El Eman (Short Bio)

Alternative Video Format:
- Download in MPEG4

Researching Source Code Quality

admin — Thu, 25 May 2006 23:37:00 GMT

Sam Ramji interviews Brendan Murphy to discuss his research on source code quality.

Video: Sam Ramji interviews Brendan Murphy to discuss his research on source code

Alternative Video Format
Download MPEG4 Format

Software Dependability with Brendan Murphy, Microsoft Research

admin — Fri, 19 May 2006 17:10:00 GMT

Bryan Kirschner, Research Strategist with the Open Source Software Lab, interviews Brendan Murphy from Microsoft Research. The two look at software dependability on applications, systems & platforms and how errors relate to software design and human oversight.

Video: Software Dependability with Brendan Murphy, Microsoft Research

Alternate Video Format:
- Download in MPEG4

Watson and the ISV: Sharing the Wealth

admin — Mon, 15 May 2006 20:18:00 GMT

Sam interviews Jason Hardester, Program Manager for the Watson ISV Portal, to discuss how Watson is used by ISV's to improve the customer experience across the Windows ecosystem...

Video: Watson and the ISV: Sharing the Wealth

Alternative Video Format:
-Download MPEG4

Stuff we think is cool: Improving the Windows experience, it's elementary!

admin — Fri, 31 Mar 2006 06:21:00 GMT

Ben Canning, Group Product Manager from the Office team talks Watson and how this unique solution is helping improve the Windows experience.

View Video
Format: WMV
Duration: 30:22

Updated:
Download this interview in MPEG4 format.
Download the transcript (PDF)