Port 25: The Open Source Community at Microsoft : Dev Center

Part 2: Lessons I Learned as a Project Manager Converting to Agile

saraford — Tue, 20 Oct 2009 21:11:00 GMT

In my first post for this series, I stated that "Agile is the single greatest things a team could do to significantly improve the user experience and quality of their website" and "not designing the full 100% is a true blessing in disguise." By putting these two concepts together, you can fully embrace the power that is Agile: Course Correction.

Making Design Changes in Waterfall

In the traditional waterfall, a Program Manager writes a specification, in which all members of the team and the associated business owners review and approve. Then, the development cycle begins to code to the specification word for word. Once code complete, the test team takes over reviewing the code to ensure it matches the specification.

But, what if half way through, someone on the team realizes the specification is wrong? Or more practically, what if the requirements change sometime during development? In my Microsoft experiences, this is called a Design Change Request (or a DCR), and it is extremely costly. New designs need to be made and reincorporated with the rest of the specification. Dev needs to re-code the feature. And the test team has to restart any prior testing, especially for regressions. In other words, DCRs are not good things.

Making Design Changes in Agile

One of the light bulb moments for me on CodePlex.com was accepting the fact that Agile is really just a series of mini waterfalls. Instead of designing a waterfall release cycle that will span the course of several years, you're only designing for a waterfall process of a few weeks. There's a Planning Phase, a Development Phase (which encapsulates the Test Phase, but is beyond the scope of this blog post), and the Deployment Phase. It's waterfall, but waterfall moving at an incredibly fast pace.

Now let's reconsider those Design Change Requests, but on an Agile team. The work is very, very scoped, which implies less ambiguity. (Yes, you could have a perfect specification that isn't ambiguous at all, but if that spec is 60 pages, the risk of human error is higher. Trust me, I've seen this, where I was the human in error.) And, recall that you're only doing 80% of the work. The cost of the design change is already accounted for in the remaining 20% of the development cycle. In other words, it's in the spirit of Agile that you're going to make changes along the way to better the product, or "course correct," as it was initially described to me. In agile, DCRs are good things.

Regardless whether you discover that you need to make a course correction during the Development Phase or the Deployment Phase, it's okay because Agile by its very nature can adapt and respond to these changes quickly.

An Example of Course Correction

Below is a screenshot from the original ratings and reviews designs. Notice how I circled the downloads count.

During implementation, a developer realized that the download count could be confusing. Did it mean total downloads for the entire project, total downloads for the release, or just recent (past 7 days) downloads for the release? We realized that given these designs we couldn't really convey the information we felt necessary to provide the right user experience to our visitors. So, we went back to the design and made a few tweaks. Again, we weren't trying to completely redesign the project directory. Even if we wanted to, there was no time to do anything radical. At the time the developer noticed this issue, we were going to finalize the build for deployment in a little over a week.

What we did instead was "course corrected" by making slight changes to the project metadata on the left-hand side to finish the current feature (or user story, if you're more aware of that terminology). Circled below is the metadata we added. We also added the * next to the number of page views and downloads, where we state at the bottom of the page "in the past 7 days."

Aside: CodePlex shows past 7 days data as a way of reinforcing release early, release often, which we continue today in the project directory. The screenshot below was taken at the time of this writing.

Conclusion of Part 2: Course Correction

This concludes my second post on Program Managing an Agile team.

If you like what you see, let me know! And if you don't like what you see, please don't hesitate to let me know. Seriously, I love discussing my Love / Hate Relationship with Agile development, as depicted on my personal blog.

Up Next: Putting it All Together - How the CodePlex team Builds Software

Part 1: Lessons I Learned as a Project Manager Converting to Agile

saraford — Mon, 19 Oct 2009 19:16:00 GMT

Before I became the Program Manager for CodePlex.com, Microsoft's open source project hosting site, I worked on the Visual Studio team on four different product cycles. Since Visual Studio uses traditional Microsoft product lifecycle releases, I had to learn about Agile development alongside learning about open source development when I joined the CodePlex.com team. Making the switch from releasing every three years to every three weeks didn't happen overnight!

One of the things I discovered is that Agile is used a lot in open source communities. In fact, many of the people who I've worked with personally on learning Agile have strong roots in OSS. Also, the fact that Microsoft is starting to adopt Agile philosophies shows how the company is changing, becoming more transparent, finding more ways to connect with the community, and embracing other schools of thought. And this is why I am here, to be on the inside to push for these cultural changes within Microsoft.

Although Agile is the single greatest thing a team could do to significantly improve the user experience and quality of their website, I believe it can be quite challenging for anyone not in a developer role to get accustomed to. I hope that by sharing my experiences, I can help others in non-developer disciplines on an Agile team.

In this series of blog posts, you'll discover how I learned to program manage an Agile team after six years of waterfall (the traditional method of software development).

Three Major Takeaways

If I could go back in time, here are the three things I would tell myself about Agile.

1. Design and plan for the very next step.

It's not about reaching the moon, but getting out of your front door. Sure, you can design the perfect feature, but if it is going to take you six months to get there, it is useless to an Agile team. It's about designing the journey towards the perfect feature that matters.

2. Break down work into the smallest possible functional sets.

Adding work is fun and rewarding, but removing incomplete work due to a lack of development time is painful and risky. But, you can't deploy a half-written feature either. First, break down the work into the smallest pieces. Then, put together the smallest functional sets that have to be deployed together for the feature to make sense. Your development team will tell you how many sets they can do per release.

3. Design and plan only 80% of the way.

Not designing the full 100% is a true blessing in disguise. Since you have another release right around the corner, you have the time to collect user feedback and incorporate it into the next design. Not only does this solve the remaining 20% (getting you closer to the perfect design with less cost), but also allows your customers to be virtual members of your team.

The Program Manager Release Cycle

To begin, here's an Agile release cycle from the point of view of the Program Manager. For simplicity, this illustration only depicts a single release cycle, without any overlap of previous or future cycles.

Ratings and Reviews: An Example

To further illustrate, consider the ratings and reviews feature on CodePlex.com as an example. Users can rate a release and write a review for projects on CodePlex.

One quick aside about ratings and reviews: CodePlex users rate an individual release instead the entire project. For example, consider Stephen King as an author. What does it mean for me to rate Stephen King 4 out of 5 stars? I find some of his books to be awful, like the Tommyknockers. I want those 7 hours of my life back. 1 out of 5 stars. Yet, for me, some of his other books are incredible, like The Dark Tower. 5 out of 5 stars. Hence, we allow users to rate an individual release to provide more relevant information to potential downloaders of the project.

Let's explore the ratings and reviews feature step-by-step in the Program Manager shoes.

1. Design Phase Part 1: Limit the scope to designing the minimum to make the feature useful and meaningful. For ratings and reviews, the feature must have the following:

a. User can rate a release. User can view the rating.

b. User can write a review. User can read the review.

c. User can sort by highest rated releases in project directory.

2. Design Phase Part 2: Bucket into smallest deployable functional sets. For me, personally, I use sticky notes to illustrate the "must have" pieces for each set.

a. Sticky Note #1: Rate releases / View rating

b. Sticky Note #2: Write a review / read review

c. Sticky Note #2: Sort by highest rated releases

3. Iteration Planning Meeting: To start the development cycle, meet with the development team to discuss costing.

a. In the case of ratings and reviews, my devs said they could do Sticky Notes #1 and #2, but the project directory sorting feature would have to wait for the next release.

4. Dev Cycle: Because the designs are closer to 1-page specifications rather than fully-documented implementations, questions will come up from the dev team. This is where you, the Program Manager, will:

a. Answer any questions about the missing 20% of the specifications / wireframes

b. "Course correct" (more on that later)

c. Add more feature work if time allows

5. Deployment: The release goes live. Now you can collect user feedback and incorporate it into the next development cycle.

a. The very first tweet I saw regarding the ratings and reviews feature was "Sara, is there a way to sort by highest rated?" Here, I was able to ask the user questions about how this feature should work to confirm our designs. Most of the time we don't inform users what's coming up next (we like surprises.) But in this case, it was pretty obvious.

Here's the visual representation of the ratings and reviews feature in the Program Management agile release cycle:

Conclusion of Part 1: The Agile Program Management Cycle

This concludes my first post on Program Managing an Agile team. Since I could go on endlessly writing about topics I'm passionate about (and those who know me will confirm this is not an exaggeration), I'm going to pause here to conclude this initial train of thought.

Up next: The concept of Course Correction.

Expanding the interoperability of eclipse4SL: Mac support comes to the Eclipse for Silverlight project

Vijay Rajagopalan — Wed, 18 Mar 2009 18:15:00 GMT

In October 2008, when Microsoft announced the general availability of Silverlight 2.0, and, as part of Microsoft’s ongoing commitment to interoperability, Microsoft and Soyatec unveiled the Eclipse Tools for Silverlight (Eclipse4SL).

Today, Microsoft and Soyatec are expanding our interoperability collaboration by releasing a Community Technology Preview (CTP) of the eclipse4SL project that includes support for the Macintosh platform. Mac and Windows developers can now collaborate on Silverlight projects using the Eclipse or Visual Studio IDEs. We have also added several new features to the project that should please all developers like C# code generation and improved XAML auto-completion (check out our Interoperability Team blog for more details and a demo)

As an open source initiative sponsored by Microsoft (funding and architectural guidance) and led by Soyatec (development), the eclipse4SL project is released under the Eclipse Public License Version 1.0 on SourceForge.net and was submitted by Soyatec it to the Eclipse Foundation as an open Eclipse project. Since its inception the project has received lots of feedback and made significant progress. eclipse4SL has been among the “Top Rated” projects on www.eclipseplugincentral.com for weeks:

(Screenshot taken on 03/16/2009)

For more information, visit the Eclipse for Silverlight web site, or join the discussion at http://www.eclipse4sl.org/community/.

Vijay Rajagopalan, Principal Architect in the Interoperability Strategy Team at Microsoft.

Moonlight 1.0 Hits the Street

Peter Galli — Wed, 11 Feb 2009 19:42:00 GMT

Moonlight 1.0 is now available.

Moonlight is an open source project that gives Linux users access to Microsoft Silverlight content, and is available for all major Linux distributions, including openSUSE, SUSE Linux Enterprise, Fedora, Red Hat, and Ubuntu. This milestone release is part of the technical collaboration between Novell and Microsoft.

Microsoft has worked with the Moonlight team and Novell to enable interoperability between Windows and Linux platforms and extend the high-quality interactive Web and video experience for the benefit of the Linux community, said Scott Guthrie, corporate vice president of Microsoft's .NET Developer Division.

Microsoft has provided Novell with access to its test suites for Silverlight, and provides Linux end users of Moonlight with free access to the Microsoft Media Pack, a set of licensed media codecs for video and audio that bring optimized and licensed decoders to every Linux user using Moonlight. Windows Media Video (.wmv), Windows Media Audio (.wma) and MP3 files are supported through the Microsoft Media Pack.

A pre-release of Moonlight was made available on January 19, 2009 to allow Linux users to stream Barack Obama's Inauguration, and more than 20,000 Linux users downloaded Moonlight to watch that Silverlight broadcast.

"Microsoft Silverlight offers the most comprehensive and powerful solution for the creation and delivery of rich internet applications and media experiences, and is used by hundreds of thousands of developers worldwide," Guthrie said.

For his part Miguel de Icaza, the founder of the Mono project founder and vice president of Developer Platforms at Novell, said Moonlight brings the benefits of Silverlight's popular multimedia content to Linux viewers. "This first release delivers on the goal of breaking down barriers to multimedia content and creating parity in the user's viewing experience regardless of whether the user is on Windows or Linux."

Welcome Snakebite, the Newest Open Network in Town!

Peter Galli — Tue, 10 Feb 2009 13:38:00 GMT

There's a new Open Network in town, known as Snakebite, which is the brainchild of Trent Nelson, a committer to Python.

Snakebite is a network of some 37 servers of different shapes and sizes, spread over three sites and specifically geared towards the needs of open source projects like Python.

In short, and according to its Website, Snakebite is a network to provide open source developers unrestricted access to as many of the different platforms, operating systems, architectures, compilers, devices, databases, tools and applications that they need to optimally develop their software.

"Why do we develop open source software on closed networks? Why do open source developers only have access to a fraction of platforms that their software will eventually run on? And why the *@&# are the Windows buildbots always red?! Snakebite was created out of a desire to try and address problems like these faced by open source projects," the Website says.

Some months back, Nelson realized that while buildbots were fine when everything was running smoothly, nothing compared to actually having access to a system when a developer is trying to debug something.

"So, I thought to myself, why not buy a couple of clunky old boxes off eBay and donate them to the Python Software Foundation, such that all developers had access to them ... Ten months, seven trips to Michigan State University, six blown fuses and about $60,000 later, I'm proud to introduce you all to Snakebite: The Open Network!," he said.

This means that every CPython, Jython, IronPython and PyPy committer will have access to every development server on the network, Nelson says, adding that he has already extended the offer to prominent Python projects like Django and Twisted.

The end-goal is to ultimately invite other open source projects like Apache, Subversion, MySQL and Postgres, among others, but given that this network is Nelson's "gift to All Things Python, first and foremost," Python projects will always get preferential treatment.

Nelson also has big aspirations for Snakebite going forward, which you can read all about in this email he sent to the Snakebite list.

Nelson was also blown away by the level of support he received for the initiative: "Microsoft jumped on board and provided unlimited MSDN licenses in less time than it took me to write them an e-mail asking for stuff. Having the support of Microsoft from very early on has been a huge boost, and the MSDN licenses have already been invaluable ," he said.

An email to HP asking them for a Tru64 license and 2GB of RAM for the Itanium box he bought off eBay, also resulted in the company shipping two quad Itanium 2 RX-5670s, full of 73GB 15k disks and 78GB of RAM between the two servers; 32GB in one and 46GB in the other.

Sun, Google and Canonical have also expressed interest in the project, but Nelson has stopped asking for hardware as they have run out of space to host it all.

But, in a recent post to the Python Committers mailing list, Nelson notes that it will probably be a few weeks before users can start logging in and doing stuff.

"The HPCC/CSE server room at MSU is about to have walls knocked in and ramps built in order to accommodate a giant PDU that has been sitting outside it for about six months; the Snakebite rack is going to get shuffled around a bit so I figure there's not much point going live before that's taken care of," he said.

Web Sandbox Source Now Available Under Apache License 2.0

Peter Galli — Tue, 27 Jan 2009 02:48:00 GMT

Microsoft has released more source code under an OSI-approved license: this time it has made the source code for the Web Sandbox runtime available under the Apache 2.0 open source license.

The Web Sandbox project explores how to advance the web platform to improve security, isolation, quality of service and extensibility capabilities for web developers and website users.

More information on the licensing details, as well as comprehensive documentation for experimenting and integrating with the Web Sandbox, can be found here.

But, while developers are being encouraged to help define and refine the Web Sandbox, it is not recommended for those developers creating production sites as it is still under development.

The Web Sandbox was created in response to limitations found in the current web platform, and is designed to explore potential solutions. Having a more secure and robust architecture as a foundational building block will help drive the next wave of Web innovation.

The Sandbox is a framework that works on most modern browsers that support the"ECMA-262, 3^rd Edition" (JavaScript) standard, and provides the same features in all modern web browsers. No browser add-ons or changes are required to leverage this technology. Beyond security, the Web Sandbox normalizes the different browsers and provides consistent W3C DOM support.

Since the initial release of Web Sandbox at PDC 2008, the team has received a lot of useful feedback from the web security community, and has also been collaborating with a number of customers, partners and the standards communities, all of whom want to adopt the technology when it is ready.

The goal? An open and interoperable standard that will help foster interoperability with complementary technologies like script frameworks and drive widespread adoption of the Web Sandbox.

This move is good news for Microsoft and the open source communities. But, it is important to note that while an Apache license is being used, the Web Sandbox project is not an Apache Software Foundation project and is not sponsored or endorsed by the ASF.

Microsoft does, however, already have an active relationship with the ASF. In fact, last year the company announced it had become a sponsor of the ASF so as to help enable the Foundation pay administrators and other support staff so that its developers can focus on writing great software.

Sam Ramji, the senior Director of Platform Strategy at Microsoft, also delivered a keynote address at ApacheCon in New Orleans last November.

Microsoft's Interoperability Technical Strategy Team already participates as a code contributor to the Apache Stonehenge incubator project; the company has also contributed a patch to ADOdb, a popular data access layer for PHP used by many applications and which is licensed under the LGPL and BSD; while Microsoft's Powerset team contributes to HBase, an open-source, column-oriented, distributed database written in Java.

Update: Stonehenge Incubation Project

Peter Galli — Mon, 19 Jan 2009 20:00:00 GMT

As Microsoft continues the drive for interoperability between different implementations on various platforms, the Interoperability Technical Strategy Team is, for the first time, participating as a code contributor to an Apache project: the Stonehenge incubator project.

Microsoft first talked about Stonehenge at ApacheCon 2008, which was held in New Orleans last November. Since then, it has been approved as an incubator project within Apache Software Foundation, and WSO2 and Microsoft have already contributed code for a web-services based sample application, known as StockTrader, to this effort.

That code can be found here, along with the contributions from WSO2.

Stonehenge has attracted some very prominent committers so far, Kamajit Bath, a Principal Program Manager in the Interoperability Technical Strategy Team and the lead for Microsoft's participation in Stonehenge, says in a recent blog post.

"I hope that the momentum will be sustained, and I am looking forward to seeing code contributions from other folks and seeing the StockTrader sample application enhanced with new features. I also hope that new sample applications will be developed to cover other areas of the WS-* standards that are not best represented by the StockTrader application. I look forward to participating in this discussion with the Stonehenge community," he says.

There are three Microsoft committers on the Stonehenge incubator project: Greg Leake, who wrote the original StockTrader application; Drew Baird, who worked to get it ready for contribution to Stonehenge; and Mike Champion, who will play an active role in this effort.

In a recent blog post, Champion says Microsoft have heard from customers that they want sample applications based on real-world scenarios and challenges, as these will help them realize the potential of these technologies that have been developed and standardized for the last 8 years or so.

The initial response from the Apache community has also been quite favorable, and "I have a personal commitment to invest in helping make Stonehenge a success, and look forward to digging in," he says.

Champion also notes that Stonehenge is being championed by Paul Fremantle, co-founder and CTO of WSO2, "which has been a great partner in helping to improve and demonstrate the interoperability of the WS-* standards across platforms."

He cites, as an example of this, TechEd 2008, where Jonathan Marsh of WSO2 and Greg Leake of Microsoft demonstrated how separate WSO2 and Microsoft components implementing a mutlti-tier stock trading application can interoperate and be substituted for one another, he says in the blog.

StockTrader is also just the starting point for the broader goals of Stonehenge, which aims to develop a set of sample applications to demonstrate seamless interoperability across multiple underlying platform technologies by using currently defined W3C and OASIS standard protocols.

Stonehenge can also help wire up the ‘last mile' between the standardized web services infrastructure that is now implemented across key platforms, and a new generation of service oriented applications that will span them, he says.

Existing WS-* interoperability work such of the sort done by WS-I and in our "plugfests" will continue to solidify the platform-level interoperability. The new work, exemplified by Apache Stonehenge, should attract a wider community of users who can exploit the hard standardization and platform interoperability work without having to wallow in as many nasty details as in the past.

For Bath, projects like Stonehenge are important in enhancing interoperability between different software implementations. While standards organizations do a great job and the roll out of various WS-* standards is a testimonial to the fact that they can work efficiently, interoperability work doesn't stop at the end of the standardization process but, rather, that is where it really starts, he says.

SMB2: a Complete Redesign of the Main Remote File Protocol for Windows

Peter Galli — Mon, 08 Dec 2008 16:24:00 GMT

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's.

Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time.

Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

In this blog Jose Barreto, a senior technical evangelist in Microsoft's Storage Solutions Division, explains some of the history behind the protocol and outlines important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Introduction

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's. Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time. Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

A History of SMB and CIFS

When it was first introduced to the public, the remote file protocol was called SMB (Server Message Block). SMB was used, for instance, by Microsoft LAN Manager in 1987 and by Windows for Workgroups in 1992. Later, a draft specification was submitted to the IETF under the name Common Internet File System (CIFS). The CIFS specification is a description of the protocol as it was implemented in 1996 as part of Microsoft Windows NT 4.0. A preliminary draft of the IETF CIFS 1.0 specification was published in 1997. Later, extensions were made to address other scenarios like domains, Kerberos, shadow copy, server to server copy and SMB signing. Windows 2000 (released in 2000) included those extensions. At that time, some people went back to calling the protocol SMB once again. CIFS/SMB has also been implemented on Unix, Linux and many other operating systems (either as part of the OS or as a server suite like Samba). A few times, those communities also extended the CIFS/SMB protocol to address their own specific requirements.

One important limitation of SMB was its "chattiness" and lack of concern for network latency. It would take a series of synchronous round trips to accomplish many of the most common tasks. The protocol was not created with WAN or high-latency networks in mind and there was limited use of compounding (combining multiple commands in a single network packet) or pipelining (sending additional commands before the answer to a previous command arrives). This even led to products created to address the specific issues around SMB WAN acceleration. There were also limitations regarding the number of open files, shares and users. Due to the large number of commands and subcommands, the protocol was also difficult to extend, maintain and secure.

Introducing SMB2

The first major redesign of SMB happened with the release of SMB2 by Microsoft. SMB2 was introduced with Windows Vista in 2007 and updated with the release of Windows Server 2008 and Windows Vista SP1 in 2008.

SMB2 brought a number of improvements, including but not limited to:

Reduced complexity, going from over 100 commands and subcommands to just 19 (see details below)
General mechanisms for data pipelining and credit-based flow control (see details below)
Request compounding, which allows multiple SMB requests to be sent as a single network request(see details below)
Larger reads and writes make better use of faster networks, even with high latency
Caching of folder and file properties, where clients keeps local copy of information on folders and files
Durable handles allow an SMB2 connection to transparently reconnect to the server if there is a temporary loss of network connectivity
Message signing improved (HMAC SHA-256 replaces MD5 as hashing algorithm) and configuration/interoperability issues simplified
Improved scalability for file sharing (number of users, shares and open files per server greatly increased)
Protocol works well with Network Address Translation (VC count is gone)
Extension mechanism (for instance, create context or variable offsets)
Support for symbolic links

It is important to highlight that, to ensure interoperability, SMB2 uses the existing SMB1 connection setup mechanisms, and then advertises that it is capable of a new version of the protocol. Because of that, if the opposite end does not support SMB2, SMB1 will be used.

The SMB2 protocol specification was published publicly by Microsoft and you can find the link at the end of this post.

Reduced Complexity

One of the ways to showcase the reduced complexity in SMB2 is to make a comparison to the commands and subcommands in the old version.

Here is the complete list of the 19 opcodes (or commands) used by SMB2 in the message exchanges between the client and the server, grouped in three categories:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP, LOGOFF, TREE_CONNECT, TREE_DISCONNECT)
File, directory and volume access (CANCEL, CHANGE_NOTIFY, CLOSE, CREATE, FLUSH, IOCTL, LOCK, QUERY_DIRECTORY, QUERY_INFO, READ, SET_INFO, WRITE)
Other (ECHO, OPLOCK_BREAK)

When you try to get a similar list for the old SMB, things get a little more complex. I tried to make a list of all commands and subcommands using only the documents linked below and came up with over 100:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP_ANDX, TRANS2_SESSION_SETUP, LOGOFF_ANDX, PROCESS_EXIT, TREE_CONNECT, TREE_CONNECT_ANDX, TREE_DISCONNECT)
File, directory and volume access (CHECK_DIRECTORY, CLOSE, CLOSE_PRINT_FILE, COPY, CREATE, CREATE_DIRECTORY, CREATE_NEW, CREATE_TEMPORARY, DELETE, DELETE_DIRECTORY, FIND_CLOSE, FIND_CLOSE2, FIND_UNIQUE, FLUSH, GET_PRINT_QUEUE, IOCTL, IOCTL_SECONDARY, LOCK_AND_READ, LOCK_BYTE_RANGE, LOCKING_ANDX, MOVE, NT_CANCEL, NT_CREATE_ANDX, NT_RENAME, NT_TRANSACT, NT_TRANSACT_CREATE, NT_TRANSACT_IOCTL, NT_TRANSACT_NOTIFY_CHANGE, NT_TRANSACT_QUERY_QUOTA, NT_TRANSACT_QUERY_SECURITY_DESC, NT_TRANSACT_RENAME, NT_TRANSACT_SECONDARY, NT_TRANSACT_SET_QUOTA, NT_TRANSACT_SET_SECURITY_DESC, OPEN, OPEN_ANDX, OPEN_PRINT_FILE, QUERY_INFORMATION, QUERY_INFORMATION_DISK, QUERY_INFORMATION2, READ, READ_ANDX, READ_BULK, READ_MPX, READ_RAW, RENAME, SEARCH, SEEK, SET_INFORMATION, SET_INFORMATION2, TRANS2_CREATE_DIRECTORY, TRANS2_FIND_FIRST2, TRANS2_FIND_NEXT2, TRANS2_FIND_NOTIFY_FIRST, TRANS2_FIND_NOTIFY_NEXT, TRANS2_FSCTL , TRANS2_GET_DFS_REFERRAL, TRANS2_IOCTL2, TRANS2_OPEN2, TRANS2_QUERY_FILE_INFORMATION, TRANS2_QUERY_FS_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_REPORT_DFS_INCONSISTENCY, TRANS2_SET_FILE_INFORMATION, TRANS2_SET_FS_INFORMATION, TRANS2_SET_PATH_INFORMATION, TRANSACTION, TRANSACTION_SECONDARY, TRANSACTION2, TRANSACTION2_SECONDARY, UNLOCK_BYTE_RANGE, WRITE, WRITE_AND_CLOSE, WRITE_AND_UNLOCK, WRITE_ANDX, WRITE_BULK, WRITE_BULK_DATA, WRITE_COMPLETE, WRITE_MPX, WRITE_MPX_SECONDARY, WRITE_PRINT_FILE, WRITE_RAW)
Other (ECHO, TRANS_CALL_NMPIPE, TRANS_MAILSLOT_WRITE, TRANS_PEEK_NMPIPE, TRANS_QUERY_NMPIPE_INFO, TRANS_QUERY_NMPIPE_STATE, TRANS_RAW_READ_NMPIPE, TRANS_RAW_WRITE_NMPIPE, TRANS_READ_NMPIPE, TRANS_SET_NMPIPE_STATE, TRANS_TRANSACT_NMPIPE, TRANS_WAIT_NMPIPE, TRANS_WRITE_NMPIPE)

I make no claim that the list above for SMB is exact or complete, but it does make a point. As an interesting exercise, check the lists above to verify that, while SMB2 has a single WRITE operation, there are 14 distinct WRITE operations in the old protocol.

SMB2 also requires TCP as a transport. SMB2 no longer supports NetBIOS over IPX, NetBIOS over UDP or NetBEUI (as SMB version 1 did).

Pipelining

A key improvement in SMB2 is the way it makes it easy for clients to send a number of outstanding requests to a server. This allows the client to build a pipeline of requests instead of waiting for a response before sending the next request. This is especially relevant when using a high latency network.

SMB2 uses a credit based flow control, which allows the server to control a client's behavior. The server will start with a small number of credits and automatically scale up as needed. With this, the protocol can keep more data "in flight" and better utilize the available bandwidth.

This is key to make a large transfer go from hours (in SMB) to minutes (in SMB2) in a "long and fat pipe" (high bandwidth, high latency network).

For an example of how pipelining in SMB2 can improve performance, check out this blog post.

Compounding

When you look at the command set for the new SMB2 protocol, you notice that they are all simple operations. The old SMB1 protocol had some complex commands and subcommands that combined a set of simple operations as required in specific scenarios.

One of the important changes in SMB2 is the ability to send an arbitrary set of commands in a single request (single network round trip). This is called compounding and it can be use to mimic the old complex operations in SMB1 without the added complexity of a larger command set.

For instance, an old SMB1 RENAME command can be replaced by a single request in SMB2 that combines three commands: CREATE (which can create a new file or open an existing file), SET_INFO and CLOSE. The same can be done for many other complex SMB1 commands and subcommands like LOCK_AND_READ and WRITE_AND_UNLOCK.

This compounding ability in SMB2 is very flexible and the chain of commands can be unrelated (executed separately, potentially in parallel) or related (executed in sequence, with the output of one command available to the next). The responses can also be compounded or sent separately.

This new compounding feature in SMB2 can be used to perform a specific task in less time due to the reduced number of network round trips.

Conclusion

I hope this post has helped you understand some of the important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Reference

Below is a list of important links that document SMB2, SMB and CIFS, including the latest protocol specifications published by Microsoft:

Microsoft at AJAXWorld

Peter Galli — Mon, 20 Oct 2008 22:05:00 GMT

The AJAXWorld Conference and Expo got underway in San Jose today, under the broad theme of "Rich Web Technologies for Enterprise Web 2.0 & Social Web."

Scott Guthrie, a corporate vice president at Microsoft, delivered the keynote address today, while Brad Abrams, the Product Unit Manager for the AppFx team here at Microsoft, will also be presenting a couple of sessions over the next few days.

While Microsoft is not announcing anything new at the show, Abrams will point to the cumulative effect of all the work that has been done so far. He will talk about how JQuery will ship with future versions of Visual Studio, as well as how standards based JavaScript, CSS, and HTML are all now supported in Visual Studio 2008.

Attendees will also get to see how JQuery interoperability and usage is now on the same page with ASP.NET Ajax, as well as demos of the cross-browser, cross-platform Silverlight plug-in, which works on Firefox, Safari and Internet Explorer, and on both Macs and Windows machines.

Abrams tells me he also plans to show off a couple of Silverlight sites running on Linux with the Novell Moonlight implementation of Silverlight, as well as a demo of its Silverlight tools for Eclipse, which underscores how developers can use the tools they want to leverage Silverlight.

He has also posted a lot more detail on his talk, which was titled "Building a Great Ajax application from Scratch," in this blog post.

Also, Bryan Kirschner of the platform strategy group, will be delivering a keynote address on how Microsoft participates in a world of choice at GOSCON, the Government Open Source Conference, in Portland tomorrow.

In addition, two other colleagues from Microsoft will also be participating. Stuart McKee, the National Technology Office for the U.S., will be on the Government Open Collaboratives Panel with Brian, and Kathleen Connor from Microsoft's Health Solutions Group, will also be speaking.

CodePlex launches support for TortoiseSVN

jcannon — Tue, 16 Sep 2008 17:24:00 GMT

From our very own Sara Ford, word comes of Codeplex now offering server support for SvnBridge. SvnBridge enables TortoiseSVN to talk to Team Foundation Server. From Sara, "Support for Subversion has been our number one requested feature, and by hosting SvnBridge, our users can now use their favorite Subversion client with any project." Read the full blog here.

A picture says a thousand words:

Featured Project: Family.Show on Codeplex

jcannon — Tue, 27 May 2008 16:09:00 GMT

Over the course of the past year, we've highlighted various community and open source projects on Codeplex. This morning, I wanted to do the same with a very cool open source project that I came across over the weekend called Family.Show. Family.Show is a genealogy project that visualizes family mapping and relationships. The project was released in July 2007 and it's currently at version 2. While the project is backed by Vertigo Software, it is Open Source and licensed under the Microsoft Public License.

Here's a small screen grab from the project home page:

Over the weekend, I used some free time & a family BBQ to start building my own family tree. Frankly, while I'm impressed with the application and it's ease-of-use, I was more amazed at the interest and enjoyment everyone in my family experienced using it. Everybody from my 83 year old grandmother to my 19 year old sister were excited to explore and add to the family tree. It gives new meaning to the collaborative potential of open source :>

See above, the Cannon family tree (yikes) - but infinitely fascinating to self-described family historians. It's worth noting that Family.Show also supports the import/export of GEDCOM standard files, so your work can interchange with other genealogy software packages as well.

I encourage anyone interested to download & tinker with it. Additionally, here are some additional resources:

Visit the Codeplex project home page
Family.Show at MIX07
Watch a Channel9 Video with the core development team

-Jamie

Code Parallel or Die, Part 2

Frank Chism — Tue, 15 Apr 2008 19:40:00 GMT

It’s multicore time. Do you know where your parallelism is?
Do you know where your parallelism is? Well you better find it! This is because you must or your users will not see the doubling of performance that they have seen for the last seven or eight generations of microprocessors. Why? Well the chip makers ran out of physics, clocks just couldn’t go a lot faster every few months the way they used to and more cores, even if they double or triple the performance of the processor do not lead to a faster feeling system if all the applications are serial. So, is your application serial? It better not be.

Two examples from my own life demonstrate the user and buyer frustration we are about to hit head on. First my sister, a long time Mac user, bought a new dual socket dual core Mac. On paper this system was easily three to five times “more powerful” then her previous system, a dual processor Mac. Her first complaint was, “It doesn’t feel faster. It certainly doesn’t feel ‘wicked fast’ compared to my five year old system.” I pointed out to her that what she had was more power, not more ‘faster’. I told her to try doing more than one thing at once. It took awhile for her to get the hang of this, but one day recently she called and started raving about how ‘powerful’ her new system was compared to her old one. What she had learned how to do was to get three or four long running tasks to run in the background while continuing to do her interactive day to day tasks without any noticeable degradation in interactive performance!

As a second example, for Christmas I built my wife a new dual core system because she was complaining about her system being too slow and crashing all the time. It was a thing of beauty with twice the CPU, twice the memory, and five times the capacity and twice the bandwidth of her old disks. The processor was two generations newer with a 20% faster clock and over four times the memory bandwidth. I was ready for a big hug and profuse thanks for my intelligence and craftsmanship. Boy was I in for a disappointment. What I got was, “It isn’t any faster.” I am a victim of the core war!

The lights are on, but there’s no parallel home.
Well my personal tragedy was nothing compared to what will happen to any software developer or corporation that depends on their fortune or fame to flow from people using their product and liking the experience. That is, it will be tragic for those who do not start to think parallel for every step of every phase of every program that they write and design in parallel not try to graft it in afterwards. Just as with security, and network awareness, you don’t get good results trying to add in parallel later. You must start in parallel and continue in parallel and only go serial as a last drastic measure and then only for the shortest possible time. Sadly, even in my own backyard of High Performance Computing the number of programs that have tried to graft in parallel rather than design it in is appalling. What they end up with is often what I call the ‘Silence of the Lames’. That is, a lame parallel port that doesn’t scale or doesn’t speed up their program nearly as much as is possible.

Now you might say that this only works for scientific programs with massive amounts of data. Well, sorry Charlie, if your product is something as ‘serial’ as a document processing program like Microsoft Word, you had better be thinking how you can use those extra cores to improve the user experience. Why? Well as I said in the beginning: Code parallel or die!

For the want of a nail, the shoe was lost; for the want of a shoe the horse was lost; and for the want of a horse the rider was lost, being overtaken and slain by the enemy, all for the want of care about a horseshoe nail. - Benjamin Franklin

That’s all for now. I enjoyed writing this and hope to hear from some of you about what you think of my ‘Parallel Imperative’ and what you can personally do about it.
So, never stop studying and I’ll blog at you later.

- Frank

Code Parallel or Die

Frank Chism — Mon, 14 Apr 2008 17:22:00 GMT

“What we have here is a failure to communicate.”
- Luke in “Cool Hand Luke”

A fist full of cores in a server rich environment.
If you haven’t noticed that the world has changed, you had better wake up and smell the coffee. The era of killer apps written in serial code is ending. Like the norm in species extinction, the reason many ISVs will not survive this change is that the environment in which we develop code has changed rapidly and not all species can adapt to this new world. The change was preceded by a hint of ‘The Shape of Things to Come’ in the Beowulf and compute cluster revolution. This revolution created the first generation of commodity clusters based on microprocessor nodes connected by commodity networks. An evolutionary message from this generation is that if you are building a cluster you really ought to use server class nodes. You _can_ build a cluster from desktop parts, but the higher you scale your node count the more you want the convenience and reliability of server parts. I remember seeing a look of outrage on the face of an Intel marketing person when I commented that I thought their then brand new Itanium Tiger platform was a really nice part for building a cluster. He thought of the new platform as a High Performance Server, not as a ‘part’. Well get used to it. We don’t build computers out of discreet components like transistors any more (which is good for him because Intel makes really big Integrated Circuits, not transistors) and we have just moved up to the point where the level of integration is the server, not the processor.

And Then There Were Cores
So into this environment where servers are parts and many high performance computing applications were written to a distributed memory communications aware parallel programming model, the microprocessor has changed in the most radical way I can remember. Our equilibrium has been punctuated (see: "Punctuated equilibria: an alternative to phyletic gradualism" (1972) pp 82-115 in "Models in paleobiology", edited by Schopf, TJM Freeman, Cooper & Co, San Francisco. Eldredge, N. & Gould, S.J.) and we can expect rapid change to a whole new kind of parallelism. The change I refer to is the introduction and near ubiquity of multiple core processors. For years software developers have been able to count on dramatic increases in CPU speed and processing power on a Moore’s Law schedule. But, Moore did not say that the processing speed and power of CPUs would double every eighteen months, he said that the number of gates on a single chip would double every eighteen months. So, all was good for a long time because that doubling of gates meant smaller _and_ faster gates and greater architectural complexity and thus ‘faster’ CPUs. So by about 2005, just about every good idea for a better computer processor architecture ever thought of was incorporated into the major microprocessors on the market. So, when 2006 rolled around the obvious thing to do was to use the next doubling of gates to double the number of ‘processors’ on a microprocessor. Dual core was here and the trend will continue for at least a couple of doublings. Who knows what strange beasties will emerge when we get to the point where ‘many core’ processors become ‘too many core’ processors. Until then, though we can be sure: Multicore killed the serial star....

More on this tomorrow.

Supernova

Sam Ramji — Wed, 19 Mar 2008 16:22:00 GMT

I’m writing this from EclipseCon in Santa Clara, California, where I’m going to announce the beginning of Microsoft’s collaborative work with the Eclipse Foundation.

This started about a year ago when I met Mike Milinkovich at an open source event (the Open Source Software Think Tank 2007) where we were seated at the same table, and assigned to discuss “key issues inhibiting the growth of open source”. We found we had pretty similar ways of looking at problems – I found Mike to be very pragmatic and straightforward in his thinking. That discussion led to a conversation about what we could do to help Eclipse developers building software for Windows.

At the same time, the CardSpace team at Microsoft was already working actively with the Higgins Project to establish a secure, interoperable framework for user identity on the web – an architecture known as the Identity Metasystem. Since the inception of Higgins, the CardSpace team has worked very closely with the Higgins team, providing them the protocol documentation they needed to be able to build an identity selector that is interoperable with CardSpace, as well as placing those protocol specifications under the OSP so that they knew that it was safe to do so. We share a commitment to building a user-centric, privacy-preserving, secure, easy-to-use identity layer for the Internet.

Currently, Higgins, Microsoft, and dozens of other companies and projects are in the midst of the third OSIS-sponsored user-centric identity interop, where we all try our code together, providing the data needed to improve both our implementations and the interoperability between them.

Among a range of other opportunities (which we’re still working on), we discovered that Steve Northover (the SWT team lead) had gotten requests to make it easy for Java developers to write applications that look and feel like native Windows Vista. He and a small group of developers built out a prototype that enables SWT to use Windows Presentation Foundation (WPF). We’re committing to improve this technology with direct support from our engineering teams and the Open Source Software Lab, with the goal of a first-class authoring experience for Java developers.

This is exciting to me – as a Java developer in my prior life (as well as the first technical marketing manager for BEA’s WebLogic Workshop, now Apache Beehive) it just makes sense to enable Java on Windows. We started a collaborative effort with JBoss two years ago that continues to this day. At the end of the day, it’s all about the developer. There will be more to come from the conversations that Eclipse and Microsoft have begun, and I look forward to announcing those in the future as we have demonstrable technology results.

Cheers,
Sam

PHP on Windows

hjanssen — Tue, 04 Mar 2008 13:50:00 GMT

I have this really odd feeling of Dejavu. The last blog I wrote (here), I started by saying that I have been very delinquent with writing blogs. So what do I do? I continue to be very delinquent with my blogs!

If anything, I am consistent! :)

So, why this sudden re-emergence of myself on port25? (Besides the fact that I just realized that it has been months since I last blogged, that is) Well we have been keeping ourselves very busy in the last few months. A lot of that work relates directly to PHP, so I wanted to talk about some of these efforts.

We have significantly increased our work in this area. And my group continues to find itself at the middle of pretty much all of these efforts. The SQL Server driver for PHP is now in its second release (Get the latest bits here). And we continue to make improvements and enhancements going forward to this driver.

We have also been working very closely with the PHP community on improving PHP performance on Windows. This is an effort that will be ongoing and probably accelerating in the months to come.

With the release of Windows Server 2008, I wanted to take a moment and highlight some of the things that have been done to make Windows an excellent PHP Deployment platform.

First of all, Microsoft late last year released FastCGI for IIS6 and IIS7. IIS7 is integrated with Windows Server. IIS7 has as part of its deployment FastCGI included (not an optional download as is the case with IIS6) this in effect means that with IIS7, Microsoft has added out of the box Microsoft-supported software designed to run PHP on IIS.

Equally as interesting, we have been working with Zend to help them certify Zend Core for Windows Server 2008. I think this makes it one of the first PHP products to be certified for WS2008. Which makes Windows PHP ready :). Imagine that, PHP running on Windows Server, fully certified!

To this end we will continue to work with Zend very closely to continue to improve PHP on Windows.

Secondarily (although not directly related to Windows Server) we have also been working with Zend to provide Cardspace functionality in the Zend Framework. You can get it here, and read more about how to use it at the Zend website here. Which is another way in which Microsoft has added support to PHP for Microsoft technologies.

This continues the close relationship with Zend into the future, and I am expecting to work with them on a whole host of other PHP related efforts going forward.

As you could read in earlier blogs by Garrett Serrack we hosted the Apache Software Foundation guys here in Redmond last week. So look for a blog from me later this week on the wrap-up of that visit.