< Back to Blogs
Can Open Source Scratch an Itch Before It Happens? by anandeep on February 05, 2007 09:25PM

I am an avid reader of Joel On Software – I find his insights great and very revealing.

I was reading a recent blog post by Joel entitled “The Big Picture”  in which he has this to say about open source

“Open source doesn’t quite work like that. It’s really good at implementing copycat features, because there’s a spec to work from: the implementation you’re copying. It’s really good at Itch Scratching features. I need a command line argument for EBCDIC, so I’ll add it and send in the code. But when you have an app that doesn’t do anything yet, nobody finds it itchy. They’re not using it. So you don’t get volunteers.”

This was in the context of the review of the book “Dreaming in Code” by Scott Rosenberg.  Scott was on campus at Redmond a few days ago, talking about his book.  The book is a  “Soul of a New Machine” type look into an open source startup that was trying to make a PIM (Personal Information Manager) code-named Chandler – and the travails that startup went through.  Scott begins his introduction by saying ”… the art of creating it (software) continues to be a dark mystery”, so you can guess the whole thing didn’t go very well!

This got me thinking – what is open source really good for? Is it only good for copycat or scratch-an-itch type of software? Could there be a limit to what open source process can achieve in terms of software artifacts?

First of all – being “copycat” or “scratch-an-itch” type of software is not bad at all.  It can be argued that Firefox falls into the former category being  based on closed source browsers but while gaining feature parity with other browsers it added new features. I think this was  the benefit of the general public because not only did Firefox get better in the spirit of competition other browsers (including Internet Explorer) got better.  As for the latter category, what good is software if it isn’t working for its’ users or “scratching an itch”?

I think the point Joel was trying to make was that bootstrapping an open source project requires either a user need or the need for an alternative. A community is more easily built when there is a shared need for functionality or alternatives.

But something about that bothered me. After all isn’t Open Source all about the “love of the game”? Why wouldn’t a community want to do something that was experimental and didn’t have any immediate payoff? Coming from a university research environment, I knew there were people out there putting out experimental code into open source including everything from Robotics to the ALICE Educational Software Authoring System (I knew the person behind ALICE – Randy Pausch from Carnegie Mellon).  ALICE has a pretty vibrant open source community behind it.  

That said, all the top open source projects (based on a poll by O’Reilly) fall within Joel’s characterization.

So can futuristic experimental projects be developed using the open source process?

I think that the answer is yes.  But these kinds projects cannot be developed in a pure open source community process like that of Linux.  An institution like a university or a company has to bring to it critical mass. The US government paid for a lot of ALICE – before it could be put out there in a true community process.

BTW I just looked at the ALICE website – and Microsoft has also supported ALICE financially. That wasn’t the case back when I was at Carnegie Mellon – I remember thinking, “How is an educational software package which is a rage with art students going help the US defense department?”.  Almost all its funding at the time came from DARPA (Defense Advanced Research Projects Agency)!

There has to be a lot of  money/resources/people put into a software project to bring it to a stage so that a community sees that an itch is going to be scratched, and then gets on board.

I was chatting with Hank Janssen and Kishi Malhotra about the “top” open source projects and stated that the top open source project I wanted to see was a “Cloud OS” which wasn’t yet around. I was waiting for the day when a system call  made on my laptop would kernel trap on a machine in a data center in India, without my knowing or caring to know which data center or which machine. Ruminating on this I postulated that some of the early components are already there with the Google File System and the Google Cluster Architecture. Then I realized that even though those were Linux based they were by no stretch of imagination open source!

Comments RSS
  1. orcmid said:

    I think the other example is when open-source development is employed to address a community need.  There are activities among academic organizations to provide shared solutions for on-line course administration and open-source is being used.  There are similar activities around D-Space and its friends.  I am not sure where the "starter" development arose in all cases: there probably does have to be some seed development.

    A community solution that I work with came about directly from an expressed need that applied in a particular software integration setting.  In this particular case, many vendors had the problem, it was making them crazy (because of having to keep track of updates of other-people's software and follow with updating of all their integrations), yet it was not something that it was useful for anyone to own and make a product of.  

    The product, an integration middleware that was open for anyone to adopt, was essentially an open-source with no commercial-use barriers, and it is now officially under the BSD template license.

    I'm thinking of ODMA, but TWAIN has had a similar history (and ODMA has similarities with both TWAIN and ODBC).

    In these last cases, I notice that some vendors tended to drive the community effort and contribute worked code that was shaken out in interoperability fests with others.  The artifacts have outlived the initial flurry of enthusiasm as the original, critical contributors moved on (and integration models changed).  But these niche efforts are remarkably long-lived.

    posted at 11:46AM 02/06/2007
  2. einhverfr said:

    I have been involved in a large number of open source projects.  Some of these have been wholely innovative, others have been reference implementations, and others have been closer to the copycat type.

    But the basic thing is:  every piece of software that is not explicitly a clone begins life as an attempt to do something new. It may be deeply based on other projects, but that is a strength not a weakness because there is an attempt to learn from the past.

    I am fairly conservative as a software engineer and I tend to think that innovation is almost a dirty word.  Instead, where valid innovation occurs best, it is an incrimental and nearly invisible process which results in the smallest changes necessary to meet the need.  Larger changes are then built on smaller changes until the entire application is revolutionized.

    So I don't think that it is either/or but rather both, when properly managed.

    Best Wishes,

    Chris Travers

    posted at 09:16PM 02/06/2007
  3. anandeep said:

    Orcmid and Einhverfr

    Good points - there is a thought I hadn't considered. While scratching an itch OSS may starting scratching itches that haven't happened yet. Or in more technical terms, experimental breakthroughs can be generated by a community wthat is working on what many may consider "solved" problems. Since "copycat" approaches would have the advantage of knowing more about the domain this may allow for a more innovative design to start with, which would address a more general class of problems.

    posted at 09:59PM 02/06/2007
  4. fluke said:

    The article is only about the issue of FOSS development during the initial stage of development from scratch.  In terms of his article being a "big picture," there is a picture frame that excludes the rest of the software life cycle.  There will be several similarities between commerical and FOSS development models for a start up project:

    - Both need to have an understanding of their target audience

    - In addition to knowing the target audience, it is best to use established formats and UI (or "copycat") the user is familar with

    - Both work best with a small development team to create the initial skeleton

    - And both commerical and FOSS projects tend to suffer from false starts where it fizzles out before producing much of anything

    But if you take a look at another area of the life cycle, it becomes more clear where the changes in FOSS and commerical development occur.  Take the life cycle of MS Bob.  As a finished product ended up being treated as a joke in the popular press.  However, as a set of re-usable modules, MS Bob lives on.  For example, XP's default search assistant (the dog) appears to be the dog from Bob's house.  Yet, for the life cycle in the re-use phase, only members of MS get to decide what continues on and what parts of the product remain boxed away to collect dust.  End users that invest not only their money but their time in learning a product are in this case left with something that will forever remain a static non-evolving "finished" Win16 application.  The choices to reuse parts, port to Win32 or even port to Windows CE on ARM are not things an end user can do without starting again from scratch.

    With a FOSS application, code migration or re-use becomes a more realistic option.  In this way, FOSS delivers "Cairo" style concepts today.  So, developer A covers an itch to interface with Roomba vacumm robots.  And then there is a developer B covers his itch to interface a computer with the Nintendo Wiimote.  While neither developer did anything exciting by themselves, the fact that both are FOSS makes it much easier for developer C to control his Roomba with a Wiimote and produce a fancy looking YouTube video of it.

    I wondered if you could go into detail what you consider a "pure open source community process"?  Keep in mind that FOSS is about the freedoms the software license provide the reciever of the software and not about money.  So, why is ALICE less "pure"?  Did the Linux kernel become a less pure project by accepting code from the NSA which clearly was also developed using goverment funding?  I personally am glad that goverment and MS provide funding to FOSS projects and am uncomfortable with you saying that somehow introduces an impurity to the development model.  Instead, the biggest problem with ALICE seems to be it's use of the legacy 4 clause BSD license which renders it incompatible with the GPL.  If they switched to the 3 clause BSD license, I would be more inclined to get involved.

    Lastly, your "Cloud OS" defination is overly vague.  There are several existing FOSS projects and concepts that could contribute to a "cloud" like methodology by breaking the hardware boundary at different levels of abstraction.  These include Inferno OS, Beowulf, Mosix, RPC, SOAP, SSH/shell pipes, X11, Network Block Device, iSCSI, ATA over Ethernet, Global File System, General Parallel File System and Lustre.  

    However, abstracting right between users space and kernel space across a commonity network connection to go half way around the world would introduce both security and performance conserns.  It is more likely that a couple of I2 members might try something like this but personally it sounds like a "solution" looking for a problem.  Theoritically, one could use ethernet between a processor core and a L1 cache too, but why?  There are advantages to being able to float what hardware is used at different layers but to do that with every system call is likely to be of limited usefulness.  If you get a chance, have a tech from BindView show you a graph of a common application's use of system calls and then consider the latency that would be added.

    posted at 04:34PM 02/13/2007
  5. anandeep said:

    Fluke,

    I am actually talking about the part of the lifecycle of a project where it takes off and gains critical mass. My hypothesis is that for an open source project started by a single developer or small group of developers, that critical mass can be achieved if (i) there is a product that is familiar to the masses that the project can be an alternative for OR (ii) it meets a customer requirement OR (iii) it is neither (i) or (ii) and has a large organization supporting it.

    The project will not reach the point of the lifecycle where other users are adding on to it, like you suggest, if it hasn't met the conditions (i)-(iii)

    I would love a counter-example, after all "learning occurs at impasses" as an old advisor wrote in his AI thesis on "repair theory"!

    When I said "pure" open source process I meant something that was sustained by a few individuals. Perhaps pure wasnt the right word to use, but when I think of the open source movement, Linus's solitary creation of Linux springs to mind.  When you have companies and organizations doing open source it somehow doesn't have the same romantic allure.

    The Cloud OS definition is a little vague - but I think the best way to address that is to point to Ray Ozzies Disruption memo - which is an amazing piece of work. It's like reading Tannenbaum's "Structured Computer Architecture" for the first time. It is the kind of writing that gives structure to the voices in your head.

    A good definition might be "The Cloud OS is the platform or OS that enables Ray Ozzie's vision of the services platform". The following link is the first from an MSN search of his memo

    http://blogs.zdnet.com/web2explorer/?page_id=54

    The example I gave of the kernel trap was an extrapolation (or is it intrapolation) of Ray's tentet  - "The demand for compelling, integrated user experiences that “just work”".

    I will look at the list of projects you mentioned and spend some more time thinking about the Cloud OS. Maybe part of a future blog

    Anandeep

    posted at 01:08AM 02/20/2007
  6. Pretty much ALL wiki development is, and has been, Open Source.  Only recently have there been proprietary offerings.  SocialText ended up opening up.  WikiMedia, (Open Source,) which runs Wikipedia (Open Content,) is also OpenSource.

    Was it an innovation? Absolutely. Was it useful? Yes.

    Also, language development, such as Perl and Python, has been innovative, and also done out in the open, as open source.

    posted at 02:12PM 02/23/2007
  7. I think it comes down to the people.

    My physics research is centered around two detectors at Fermilab and CERN. Each one has 1000-3000 people working directly on it and many more indirectly working on it. In some sense all software we develop is open-source: everyone can access it, use it, modify it (well, all members of the experiment).

    Many crazy projects are started. A projects success seems to depend on two things -- that it does something better or much more easily than anything else out there, and that you have a person behind it. You have to have both. This is science, so the first is obvious: finds electrons better, calculates energies better. The second is similar. If the project is big then you need a person with a vision who is able to carry others along with them.

    My limited experience with the external open source movement is the same. There are lots of great (innovative) ideas out there. And you need someone with the vision and the time to keep a project on track. In that case I think you can have a very successful project.

    Of course, the real way to test this is list all the projects on some site like sourceforge, classify them as copy-cat or new, and then rank by downloads. You then have to normalize for things like "everyone knows what a browser is, so you don't have to teach them that they need it" problem. Sorry. I don't have the time. :-)

    posted at 05:47PM 02/24/2007
  8. kram1024 said:

    You should look at the kde 4.x projects and its plasma and dolphin applications.  There was sooooo much experimentation with it, that kde 4.0 looks nothing at all like kde 3.5.9  and kde 4.1-beta2 now allows as many panels as you want, any size you want, and wherever you want them.  You can place the applets on your desktop too. kwin also improved by adding support for visual effects, such as causing the window on the screen to explode when you close it or fall to pieces instead.

    posted at 07:46AM 07/08/2008
Post a Comment
*
*