Port 25: The Open Source Community at Microsoft : Community, Networking

Team Microsoft Sings "Them Incubator Blues" at ApacheCon

Peter Galli — Thu, 12 Nov 2009 22:33:00 GMT

As you probably know, Microsoft was both a sponsor and active participant at ApacheCon 2009 in Oakland, California last week.

But what you might not know is that we also showed our lighter, more fun side, when we participated in the Lightning Talks, which were held on Thursday evening, November 5, during a reception with plenty of popcorn, beer and wine.

As my colleague Jas Sandhu noted in his blog about this - and where you can also find the song's lyrics - the talks are a lively, spontaneous ApacheCon tradition with speakers getting about 5 minutes to poke at each other, the projects, technology, community etc ... and have a bit of fun!

The singing and dancing Microsoft team was led by Kent Brown, the Product Manager for Windows Communication Foundation, who was not only the singer, songwriter and guitar player, but also the author of 'Them Incubator Blues,' which is very tongue-in-cheek and loosely based on his experience participating in the Stonehenge Project and working with the community.

He was joined by me, Microsoft's open source community manager; Tanya Young, our chief cat herder at the conference; and Jas Sandhu, a Senior Technical Evangelist. It was great fun and we hope you enjoy it!

Video courtesy of Mladen Turk from Red Hat.

Part 1: Lessons I Learned as a Project Manager Converting to Agile

saraford — Mon, 19 Oct 2009 19:16:00 GMT

Before I became the Program Manager for CodePlex.com, Microsoft's open source project hosting site, I worked on the Visual Studio team on four different product cycles. Since Visual Studio uses traditional Microsoft product lifecycle releases, I had to learn about Agile development alongside learning about open source development when I joined the CodePlex.com team. Making the switch from releasing every three years to every three weeks didn't happen overnight!

One of the things I discovered is that Agile is used a lot in open source communities. In fact, many of the people who I've worked with personally on learning Agile have strong roots in OSS. Also, the fact that Microsoft is starting to adopt Agile philosophies shows how the company is changing, becoming more transparent, finding more ways to connect with the community, and embracing other schools of thought. And this is why I am here, to be on the inside to push for these cultural changes within Microsoft.

Although Agile is the single greatest thing a team could do to significantly improve the user experience and quality of their website, I believe it can be quite challenging for anyone not in a developer role to get accustomed to. I hope that by sharing my experiences, I can help others in non-developer disciplines on an Agile team.

In this series of blog posts, you'll discover how I learned to program manage an Agile team after six years of waterfall (the traditional method of software development).

Three Major Takeaways

If I could go back in time, here are the three things I would tell myself about Agile.

1. Design and plan for the very next step.

It's not about reaching the moon, but getting out of your front door. Sure, you can design the perfect feature, but if it is going to take you six months to get there, it is useless to an Agile team. It's about designing the journey towards the perfect feature that matters.

2. Break down work into the smallest possible functional sets.

Adding work is fun and rewarding, but removing incomplete work due to a lack of development time is painful and risky. But, you can't deploy a half-written feature either. First, break down the work into the smallest pieces. Then, put together the smallest functional sets that have to be deployed together for the feature to make sense. Your development team will tell you how many sets they can do per release.

3. Design and plan only 80% of the way.

Not designing the full 100% is a true blessing in disguise. Since you have another release right around the corner, you have the time to collect user feedback and incorporate it into the next design. Not only does this solve the remaining 20% (getting you closer to the perfect design with less cost), but also allows your customers to be virtual members of your team.

The Program Manager Release Cycle

To begin, here's an Agile release cycle from the point of view of the Program Manager. For simplicity, this illustration only depicts a single release cycle, without any overlap of previous or future cycles.

Ratings and Reviews: An Example

To further illustrate, consider the ratings and reviews feature on CodePlex.com as an example. Users can rate a release and write a review for projects on CodePlex.

One quick aside about ratings and reviews: CodePlex users rate an individual release instead the entire project. For example, consider Stephen King as an author. What does it mean for me to rate Stephen King 4 out of 5 stars? I find some of his books to be awful, like the Tommyknockers. I want those 7 hours of my life back. 1 out of 5 stars. Yet, for me, some of his other books are incredible, like The Dark Tower. 5 out of 5 stars. Hence, we allow users to rate an individual release to provide more relevant information to potential downloaders of the project.

Let's explore the ratings and reviews feature step-by-step in the Program Manager shoes.

1. Design Phase Part 1: Limit the scope to designing the minimum to make the feature useful and meaningful. For ratings and reviews, the feature must have the following:

a. User can rate a release. User can view the rating.

b. User can write a review. User can read the review.

c. User can sort by highest rated releases in project directory.

2. Design Phase Part 2: Bucket into smallest deployable functional sets. For me, personally, I use sticky notes to illustrate the "must have" pieces for each set.

a. Sticky Note #1: Rate releases / View rating

b. Sticky Note #2: Write a review / read review

c. Sticky Note #2: Sort by highest rated releases

3. Iteration Planning Meeting: To start the development cycle, meet with the development team to discuss costing.

a. In the case of ratings and reviews, my devs said they could do Sticky Notes #1 and #2, but the project directory sorting feature would have to wait for the next release.

4. Dev Cycle: Because the designs are closer to 1-page specifications rather than fully-documented implementations, questions will come up from the dev team. This is where you, the Program Manager, will:

a. Answer any questions about the missing 20% of the specifications / wireframes

b. "Course correct" (more on that later)

c. Add more feature work if time allows

5. Deployment: The release goes live. Now you can collect user feedback and incorporate it into the next development cycle.

a. The very first tweet I saw regarding the ratings and reviews feature was "Sara, is there a way to sort by highest rated?" Here, I was able to ask the user questions about how this feature should work to confirm our designs. Most of the time we don't inform users what's coming up next (we like surprises.) But in this case, it was pretty obvious.

Here's the visual representation of the ratings and reviews feature in the Program Management agile release cycle:

Conclusion of Part 1: The Agile Program Management Cycle

This concludes my first post on Program Managing an Agile team. Since I could go on endlessly writing about topics I'm passionate about (and those who know me will confirm this is not an exaggeration), I'm going to pause here to conclude this initial train of thought.

If you like what you see, let me know! And if you don't like what you see, please don't hesitate to let me know. Seriously, I love discussing my Love / Hate Relationship with Agile development, as depicted on my personal blog.

Up next: The concept of Course Correction.

GroundWork Open Source Joins Microsoft's System Center Alliance

Peter Galli — Thu, 24 Sep 2009 13:31:00 GMT

GroundWork Open Source, Inc., a commercial open source company that produces network management software, last week announced the availability of the GroundWork Connector for Microsoft System Center Operations Manager.

The company has also become a member of the System Center Alliance.

GroundWork Monitor, which already has more than 1,500 plugins available, integrates with System Center Operations Manager and extends monitoring and management coverage to non-Windows systems, applications and devices.

The new GroundWork Connector pulls information from System Center Operations Manager and displays it within GroundWork Monitor Enterprise, giving customers a deeper visibility into the availability and performance of all critical infrastructures on a single console. The connector gives insight into applications, databases, virtual machines and network devices that may be running on Linux, Unix, Windows or embedded operating systems.

I talked to David Dennis, the company's senior director of marketing and business development this week about the move, which he feels is a great follow-up to the release of the System Center Cross Platform extensions earlier this year.

That release broke new ground for using System Center in heterogeneous environments. "In the field, we have more and more users asking about how they can integrate the management of Windows with open source tools for managing network infrastructure, Unix, Linux, and the applications that run on top of them," he told me.

The dialog also no longer seems to be about choice between Windows or Open Source but rather "I want both - now how do I make them work together," he says. Even though GroundWork Open Source is an open source company, about half of the operating systems managed by GroundWork Monitor are running Windows.

"The combination of System Center Operations Manager and GroundWork Monitor provides a full-featured alternative to traditional systems management frameworks, but with greater openness and at a much lower price point," Dennis says.

Zend Launches Open Source Initiative to Drive Cloud Application Development

Peter Galli — Tue, 22 Sep 2009 14:00:00 GMT

Today, Zend Technologies announced the Simple API for Cloud Application Services project, which is a new open source initiative that allows developers to use common application services in the cloud, while enabling them to unlock value-added features available from individual providers.

This new project is designed to encourage widespread participation and contributions from the open source community, resulting in the availability of Simple Cloud API adapters for virtually all major cloud providers.

Zend, Microsoft, IBM, Nirvanix, Rackspace and GoGrid are all co-founding contributors to this community project, which aims to facilitate the development of cloud applications that can access services on all major cloud platforms and whose initial goal is to provide a set of programming interfaces for PHP developers to facilitate the development of applications that have basic cloud storage needs.

The first deliverables will include interfaces for file storage, document database, and simple queue services from platforms like Amazon Web Services, Windows Azure, Nirvanix Storage Delivery Network and Rackspace Cloud Files, allowing developers to deploy software applications to access services in these environments without making time consuming and expensive changes to their source code.

As Andi Gutmans, the CEO at Zend Technologies, notes in the press release announcing the project, "cloud computing offers irresistible value to enterprises of all sizes, but the lack of portability across cloud application services for even the most basic operations has been an impediment to broader adoption of cloud services."

An initial Simple Cloud API proposal and reference implementation is already available now for community review and participation, while a technology preview of the PHP client libraries for Windows Azure can be found here.

Microsoft is also contributing Simple Cloud API adapters, along with the official PHP client libraries for Windows Azure storage, to future versions of Zend Framework. These adapters will allow applications to take advantage of many Windows Azure features through the Simple Cloud API interface, while Microsoft's client libraries will put Windows Azure innovations, such as transaction and partial upload support, at the fingertips of cloud application developers.

"The Simple Cloud API is an example of Microsoft's continued investment in the openness and interoperability of its platform. We're excited to see how this project will foster adoption of cloud computing platforms by PHP developers and hope that many of these developers are encouraged to use Windows Azure," Doug Hauger, the General Manager for Windows Azure, notes in the press release.

Microsoft's involvement started a few months ago, through our work with Real Dolmen on a Windows Azure SDK for PHP developers. This SDK has been submitted to the Zend Framework, and it now forms the basis of Microsoft's contribution to the Simple Cloud API project.

As Vijay Rajagopalan, a Principal Architect at Microsoft, notes in his blog, the Zend Adapter for Windows Azure will leverage Microsoft's contribution. PHP developers will now be able to program against Windows Azure - in a way that is consistent with other cloud platforms - by tapping into the main features of Window Azure Storage.

Those PHP developers who need to use specific Windows Azure features not included in the scope of the Simple Cloup API (like transaction), will be able to combine the Zend Cloud Adapter with the dedicated Windows Azure SDK for PHP.

"This will allow developers to use common application services in the cloud, while enabling them to unlock value-added features available from individual providers. Simple API for Cloud also gives PHP developers more choices, and this is a great opportunity for them to think about using Windows Azure," he says.

The PHP Toolkit for ADO.NET Data Services

Peter Galli — Fri, 21 Aug 2009 11:00:00 GMT

Today, the DPE Interop team at Microsoft released a new open source project that bridges PHP and .NET using REST.

Specifically, the PHP Toolkit for ADO.NET Data Services was made available, which makes it easier for PHP developers to take advantage of the ADO.NET Data Services, a set of features recently added to the .NET Framework.

The PHP Toolkit for ADO.NET Data Services is an open source project funded by Microsoft and developed by Persistent Systems. It is available on CodePlex today.

These services, which were previously referred to by the codename Project Astoria, expose a wide range of data sources through a RESTful service interface.

There is full support for ADO.NET Data Services in Visual Studio 2008 SP1 as well as in the upcoming Visual Studio 2010, which includes direct support for both creating and consuming data services directly from the development environment.

Read more about all this on the Interoperability blog, and watch the Channel 9 video interview with Claudio Caldato, Senior Program Manager in the Interoperability Technical Strategy team, and Pablo Castro, software architect of ADO.NET Data Services.

This is yet another example of Microsoft's continued commitment to openness and interoperability as well as of its embrace of Web standards in its technologies.

More on the Hyper-V Linux Integration Components

hjanssen — Mon, 20 Jul 2009 20:24:00 GMT

Well, there is no easy way to say this, so I am simply going to start this blog with the following line.

Microsoft just submitted source code for the Hyper-V Linux Integration Components to the Linux Kernel Community Under GPL v2.

Well, there's a conversation starter! Are you still all sitting in your chairs???

Let me summarize:

Yes, our device driver code was released directly to the Linux Kernel
We released the code under GPL v2
We are working with Greg Kroah-Hartman so it is ready for the next release of the Linux Kernel, version 2.6.32
We will continue to update the driver code to enhance interoperability on an ongoing basis, but it's our hope that other developers in the community will find the code useful and worthy of collaboration.

Fallen off your chair yet?

Microsoft developed the Linux device drivers to enhance the performance of Linux when virtualized on Windows Server 2008 Hyper-V. My team and I were responsible for testing and validating the driver components that were contributed for this first release.

Now, my team and I will be responsible for further developing this code going forward. (Yes, that does mean that I have gone back to leverage my very early roots as a Kernel programmer. Let the world be warned!!!!). Haiyang Zhang has been working on this code with me, and he will continue to work with me on this going forward.

When I joined Microsoft three years ago, the primary reason was to put my money where my mouth was. You see complaining about something is easy, but it becomes a little more complicated when somebody offers you the opportunity to be part of helping change what you have complained about.

So, three years after taking the job that made me put my money where my mouth was (and still often is!), I for one am EXTREMELY happy to see one of the most significant fruits of our work here in the Microsoft Open Source Technology Center (OSTC). But I have to say, even I would have been hard-pressed to think three years ago that we would consider contributing to the Linux Kernel.

As you know, two years ago Microsoft announced a partnership with Novell, and Tom Hanrahan ran the lab on a day to day basis till about 9 months ago. Since then I have had the pleasure of running the technical side of the execution of that lab under Tom Hanrahan for the OSTC. One of the primary tasks for that lab is to make sure Windows runs well on top of XEN and Linux runs well on top of Hyper-V, and we do this in very close cooperation with Novell.

We do most of this work as an extension to Mike Neil's Hyper-V team.

As part of this, we were asked to help develop and maintain a crucial part of this work called the Linux Integration Components. This code is designed so that Linux can run in an "enlightened mode" on top of Hyper-V (enlightened mode is roughly the Hyper-V equivalent of "paravirtualized mode" for the Xen hypervisor). Without this driver code, Linux can run on top of Windows, but without the same high performance levels. It is this device driver code that we are releasing today, directly to the Linux Kernel.

We're not talking a few hundred lines of code here; we're talking about roughly 20,000 lines of code.

Is this a Dump and Run from Microsoft? Absolutely not! We plan to enhance the functionality of this code, and we will continue to work with the Linux Community to support the drivers and to ensure continued interoperability.

As you can imagine, this was the result of a lot of hard work: Hiyang Zhang, who has been co-writing this code; Hashir Abdi, who has been testing all this stuff; as well as Vijay Tewari and Mike Sterling from the Hyper-V team who have been taking care of the Hyper-V side.

And last, but certainly not least, Greg Kroah-Hartman, who has been helping me to make all this code land in the right area in the kernel. He has patiently worked to help me correct my obvious mistakes and to get the code contributed into the kernel.

So where are we today? Well, Greg Kroah-Hartman will make the code visible to the outside world today. (For those who want to get a head start, the code will sit under <your kernel tree>/drivers/staging/hv). After it becomes visible, I will write a few more blogs this week that should help you to understand, build and run them.

The titles I am thinking for these blogs are:

Where do the Linux ICs reside in the kernel tree and how do I build them?

And

How do I install, configure and run the Linux IC's?

I had almost forgotten how wrapped up you can be once you start writing code again. So I have not gotten much sleep this past week, but it has been a joy to get back into coding again!

Microsoft Releases Device Driver Code to the Linux Community

Peter Galli — Mon, 20 Jul 2009 16:48:00 GMT

In what many may see as a surprising move, Microsoft today released 20,000 lines of device driver code to the Linux community under the popular General Public Licence v2.

The code includes three Linux device drivers, and has been submitted to the Linux kernel community for inclusion in the Linux tree.

The drivers will be available to both the Linux community and customers, and will enhance the performance of the Linux operating system when virtualized on Windows Server 2008 Hyper-V or Windows Server 2008 R2 Hyper-V.

In an article posted to Microsoft's PressPass site, Tom Hanrahan, director of Microsoft's Open Source Technology Center, notes that this is a significant milestone because it's the first time the company has released code directly to the Linux community. "Additionally significant is that we are releasing the code under the GPLv2 license, which is the Linux community's preferred license," he said.

In the same article, Sam Ramji, senior director of Platform Strategy at Microsoft, points out that Microsoft communities and open source communities are growing together, which is ultimately of benefit to customers. An example of this is the Linux community, which has built a platform used by many customers. "So our strategy is to enhance interoperability between the Windows platform and many open source technologies, which includes Linux, to provide the choices our customers are asking for," he said.

Ramji also alluded to the fact that people are often surprised when they hear how much open source community and development work is happening across Microsoft, which is largely due to the fact that these collaborations focus more on getting the work done and engaging with the various communities on a one-to-one basis and less about promoting them.

One example of how Microsoft participates with, and contributes to, open source is its relationship with the PHP Community. The company's involvement includes contributing to the PHP Engine, optimizing PHP 5.3 to perform strongly on Windows, and working to improve the performance of numerous PHP applications on Windows. Then there is the ongoing participation in various Apache Software Foundation projects, such as Hadoop, Stonehenge and QPID.

"In short, we're focused on building sustainable business strategies for open source at Microsoft ... we see open source playing into three key areas, one of which is the use of 'inbound' open source and the open source development model to make our software development processes more efficient."

"Good examples of this include what we did recently with jQuery in Visual Studio 2008, the implementation of OpenPegasus connectors and adaptors into System Center Operations Manager, and work that the Microsoft High Performance Computing team did with the Argonne National Lab (ANL) to source its MPICH2 implementation, which is a portable implementation of the Message Passing Interface (MPI) used in cluster computing and super computers," Ramji said.

We'll be posting a number of other articles on the release of the device driver code to the Linux community over the week, several of which will be penned by Hank Janssen from Microsoft's Open Source Technology Center, so look out for those.

An Open Source Network File System Client for Windows

Peter Galli — Wed, 22 Apr 2009 18:37:00 GMT

Microsoft is sponsoring research at the University of Michigan's Center for Information Technology Integration (CITI) to develop an open source Network File System client for Windows. This will enable Windows to better interoperate with this emerging Internet storage protocol for fast file sharing.

NFS is a commonly used protocol for sharing files among networked computers and storage hardware, particularly with UNIX and Linux-based software. NFSv4 is the latest version of this software and adds support for parallel access to file servers, object-storage, and storage area network infrastructures.

Bob Muglia, the president of Microsoft's Server and Tools Business and a University of Michigan alumnus, expressed excitement about the project, saying that NFSv4.1 is an important standard for accessing parallel file systems in the high-performance computing market, where access to vast amounts of data is critical in areas like scientific or technical computing systems.

"We believe that customers want to be able to choose the technologies that best meet their needs and that also interoperate with existing systems. Ultimately, CITI's work will help change the way customers can combine their systems by enabling computers running Windows to directly and easily access NFS file shares on servers running Linux, Solaris, and AIX operating systems."

CITI, which is a research unit in the College of Engineering, developed the open source Linux-based reference implementation of NFSv4 that is already included in all Linux distributions. However, Peter Honeyman, a research professor in the division of Computer Science and Engineering and principal investigator of this project, notes that Windows is a critical component in the University's research cyber-infrastructure, responsible for the control of instruments in laboratories across the university, in medicine, engineering, geosciences, bioinformatics, and many other disciplines.

"So this project is especially important in helping university scientists and engineers fill a gap in the storage fabric. This partnership also shows how the university can serve as a living laboratory for the development of interoperable enterprise scale systems that meet the needs of industry and academia," he said.

SMB2: a Complete Redesign of the Main Remote File Protocol for Windows

Peter Galli — Mon, 08 Dec 2008 16:24:00 GMT

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's.

Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time.

Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

In this blog Jose Barreto, a senior technical evangelist in Microsoft's Storage Solutions Division, explains some of the history behind the protocol and outlines important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Introduction

SMB (Server Message Block) is a remote file protocol commonly used by Microsoft Windows clients and servers that dates back to 1980's. Back when it was first used, LANs speeds were typically 10Mbps or less, WAN use was very limited and there were no Wireless LANs. Network security concerns like preventing man-in-the-middle attacks were non-existent at that time. Obviously, things have changed a lot since then. SMB did evolve over time, but it did so incrementally and with great care for keeping backward compatibility. It was only with SMB2 in 2007 that we had the first major redesign.

A History of SMB and CIFS

When it was first introduced to the public, the remote file protocol was called SMB (Server Message Block). SMB was used, for instance, by Microsoft LAN Manager in 1987 and by Windows for Workgroups in 1992. Later, a draft specification was submitted to the IETF under the name Common Internet File System (CIFS). The CIFS specification is a description of the protocol as it was implemented in 1996 as part of Microsoft Windows NT 4.0. A preliminary draft of the IETF CIFS 1.0 specification was published in 1997. Later, extensions were made to address other scenarios like domains, Kerberos, shadow copy, server to server copy and SMB signing. Windows 2000 (released in 2000) included those extensions. At that time, some people went back to calling the protocol SMB once again. CIFS/SMB has also been implemented on Unix, Linux and many other operating systems (either as part of the OS or as a server suite like Samba). A few times, those communities also extended the CIFS/SMB protocol to address their own specific requirements.

One important limitation of SMB was its "chattiness" and lack of concern for network latency. It would take a series of synchronous round trips to accomplish many of the most common tasks. The protocol was not created with WAN or high-latency networks in mind and there was limited use of compounding (combining multiple commands in a single network packet) or pipelining (sending additional commands before the answer to a previous command arrives). This even led to products created to address the specific issues around SMB WAN acceleration. There were also limitations regarding the number of open files, shares and users. Due to the large number of commands and subcommands, the protocol was also difficult to extend, maintain and secure.

Introducing SMB2

The first major redesign of SMB happened with the release of SMB2 by Microsoft. SMB2 was introduced with Windows Vista in 2007 and updated with the release of Windows Server 2008 and Windows Vista SP1 in 2008.

SMB2 brought a number of improvements, including but not limited to:

Reduced complexity, going from over 100 commands and subcommands to just 19 (see details below)
General mechanisms for data pipelining and credit-based flow control (see details below)
Request compounding, which allows multiple SMB requests to be sent as a single network request(see details below)
Larger reads and writes make better use of faster networks, even with high latency
Caching of folder and file properties, where clients keeps local copy of information on folders and files
Durable handles allow an SMB2 connection to transparently reconnect to the server if there is a temporary loss of network connectivity
Message signing improved (HMAC SHA-256 replaces MD5 as hashing algorithm) and configuration/interoperability issues simplified
Improved scalability for file sharing (number of users, shares and open files per server greatly increased)
Protocol works well with Network Address Translation (VC count is gone)
Extension mechanism (for instance, create context or variable offsets)
Support for symbolic links

It is important to highlight that, to ensure interoperability, SMB2 uses the existing SMB1 connection setup mechanisms, and then advertises that it is capable of a new version of the protocol. Because of that, if the opposite end does not support SMB2, SMB1 will be used.

The SMB2 protocol specification was published publicly by Microsoft and you can find the link at the end of this post.

Reduced Complexity

One of the ways to showcase the reduced complexity in SMB2 is to make a comparison to the commands and subcommands in the old version.

Here is the complete list of the 19 opcodes (or commands) used by SMB2 in the message exchanges between the client and the server, grouped in three categories:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP, LOGOFF, TREE_CONNECT, TREE_DISCONNECT)
File, directory and volume access (CANCEL, CHANGE_NOTIFY, CLOSE, CREATE, FLUSH, IOCTL, LOCK, QUERY_DIRECTORY, QUERY_INFO, READ, SET_INFO, WRITE)
Other (ECHO, OPLOCK_BREAK)

When you try to get a similar list for the old SMB, things get a little more complex. I tried to make a list of all commands and subcommands using only the documents linked below and came up with over 100:

Protocol negotiation, user authentication and share access (NEGOTIATE, SESSION_SETUP_ANDX, TRANS2_SESSION_SETUP, LOGOFF_ANDX, PROCESS_EXIT, TREE_CONNECT, TREE_CONNECT_ANDX, TREE_DISCONNECT)
File, directory and volume access (CHECK_DIRECTORY, CLOSE, CLOSE_PRINT_FILE, COPY, CREATE, CREATE_DIRECTORY, CREATE_NEW, CREATE_TEMPORARY, DELETE, DELETE_DIRECTORY, FIND_CLOSE, FIND_CLOSE2, FIND_UNIQUE, FLUSH, GET_PRINT_QUEUE, IOCTL, IOCTL_SECONDARY, LOCK_AND_READ, LOCK_BYTE_RANGE, LOCKING_ANDX, MOVE, NT_CANCEL, NT_CREATE_ANDX, NT_RENAME, NT_TRANSACT, NT_TRANSACT_CREATE, NT_TRANSACT_IOCTL, NT_TRANSACT_NOTIFY_CHANGE, NT_TRANSACT_QUERY_QUOTA, NT_TRANSACT_QUERY_SECURITY_DESC, NT_TRANSACT_RENAME, NT_TRANSACT_SECONDARY, NT_TRANSACT_SET_QUOTA, NT_TRANSACT_SET_SECURITY_DESC, OPEN, OPEN_ANDX, OPEN_PRINT_FILE, QUERY_INFORMATION, QUERY_INFORMATION_DISK, QUERY_INFORMATION2, READ, READ_ANDX, READ_BULK, READ_MPX, READ_RAW, RENAME, SEARCH, SEEK, SET_INFORMATION, SET_INFORMATION2, TRANS2_CREATE_DIRECTORY, TRANS2_FIND_FIRST2, TRANS2_FIND_NEXT2, TRANS2_FIND_NOTIFY_FIRST, TRANS2_FIND_NOTIFY_NEXT, TRANS2_FSCTL , TRANS2_GET_DFS_REFERRAL, TRANS2_IOCTL2, TRANS2_OPEN2, TRANS2_QUERY_FILE_INFORMATION, TRANS2_QUERY_FS_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_QUERY_PATH_INFORMATION, TRANS2_REPORT_DFS_INCONSISTENCY, TRANS2_SET_FILE_INFORMATION, TRANS2_SET_FS_INFORMATION, TRANS2_SET_PATH_INFORMATION, TRANSACTION, TRANSACTION_SECONDARY, TRANSACTION2, TRANSACTION2_SECONDARY, UNLOCK_BYTE_RANGE, WRITE, WRITE_AND_CLOSE, WRITE_AND_UNLOCK, WRITE_ANDX, WRITE_BULK, WRITE_BULK_DATA, WRITE_COMPLETE, WRITE_MPX, WRITE_MPX_SECONDARY, WRITE_PRINT_FILE, WRITE_RAW)
Other (ECHO, TRANS_CALL_NMPIPE, TRANS_MAILSLOT_WRITE, TRANS_PEEK_NMPIPE, TRANS_QUERY_NMPIPE_INFO, TRANS_QUERY_NMPIPE_STATE, TRANS_RAW_READ_NMPIPE, TRANS_RAW_WRITE_NMPIPE, TRANS_READ_NMPIPE, TRANS_SET_NMPIPE_STATE, TRANS_TRANSACT_NMPIPE, TRANS_WAIT_NMPIPE, TRANS_WRITE_NMPIPE)

I make no claim that the list above for SMB is exact or complete, but it does make a point. As an interesting exercise, check the lists above to verify that, while SMB2 has a single WRITE operation, there are 14 distinct WRITE operations in the old protocol.

SMB2 also requires TCP as a transport. SMB2 no longer supports NetBIOS over IPX, NetBIOS over UDP or NetBEUI (as SMB version 1 did).

Pipelining

A key improvement in SMB2 is the way it makes it easy for clients to send a number of outstanding requests to a server. This allows the client to build a pipeline of requests instead of waiting for a response before sending the next request. This is especially relevant when using a high latency network.

SMB2 uses a credit based flow control, which allows the server to control a client's behavior. The server will start with a small number of credits and automatically scale up as needed. With this, the protocol can keep more data "in flight" and better utilize the available bandwidth.

This is key to make a large transfer go from hours (in SMB) to minutes (in SMB2) in a "long and fat pipe" (high bandwidth, high latency network).

For an example of how pipelining in SMB2 can improve performance, check out this blog post.

Compounding

When you look at the command set for the new SMB2 protocol, you notice that they are all simple operations. The old SMB1 protocol had some complex commands and subcommands that combined a set of simple operations as required in specific scenarios.

One of the important changes in SMB2 is the ability to send an arbitrary set of commands in a single request (single network round trip). This is called compounding and it can be use to mimic the old complex operations in SMB1 without the added complexity of a larger command set.

For instance, an old SMB1 RENAME command can be replaced by a single request in SMB2 that combines three commands: CREATE (which can create a new file or open an existing file), SET_INFO and CLOSE. The same can be done for many other complex SMB1 commands and subcommands like LOCK_AND_READ and WRITE_AND_UNLOCK.

This compounding ability in SMB2 is very flexible and the chain of commands can be unrelated (executed separately, potentially in parallel) or related (executed in sequence, with the output of one command available to the next). The responses can also be compounded or sent separately.

This new compounding feature in SMB2 can be used to perform a specific task in less time due to the reduced number of network round trips.

Conclusion

I hope this post has helped you understand some of the important improvements in SMB2, particularly in regards to reduced complexity, pipelining and compounding.

Reference

Below is a list of important links that document SMB2, SMB and CIFS, including the latest protocol specifications published by Microsoft:

ApacheCon and the Stonehenge Proposal

Kamaljit Bath — Mon, 10 Nov 2008 21:24:00 GMT

This is Kamaljit Bath, and I am in the Big Easy to experience my first Apache Conference! I am a Principal Program Manager in the Interoperability Technical Strategy Team at Microsoft. We have been doing a lot of great interoperability work and have done quite a few Open Source projects to build bridging solutions etc., but this is the first time I am attending ApacheCon. This is a learning experience for me.

Microsoft is certainly interested in expanding interoperability between Open Source solutions and Microsoft technologies, and is working with individuals and communities for that purpose. I think this is great because it will enable choice of solutions and create strong partnerships to promote growth for everyone in the industry.

ApacheCon has been quite an experience for me so far. I have seen the energy and high quality decision making. It is amazing how people from many different backgrounds can come together and accomplish so much in so little time.

I have also met some very interesting people and some that I have wanted to meet for a long time. Meeting motivated and driven people is what I like the most about conferences. I have learnt a lot from this experience and I will take back great memories from this trip.

In his keynote today Sam Ramji, the Senior Director for Platform Strategy at Microsoft, gave an update on the many interoperability and Open Source projects that Microsoft is engaged in. I am sure that some of this was news to many of the attendees, but hopefully it gave them an idea of the breadth of work that Microsoft is doing in this area.

Sam covered a lot of things, including our participation in Apache QPID project; the release of the 'Oslo-M' language under the Open Source Promise; participation in the Apache HBase project; and support for the new Stonehenge proposal by WSO2.

Sam also covered many other open source projects that Microsoft has used to build bridging technologies, while my manager, Jean Paoli, has covered these in detail in his blog.

Sam is a well known figure in these avenues and needs no introduction - he has been representing Microsoft at many of these conferences.

But it is also important to have a more grass-root level developer and architecture presence from Microsoft, and we are now moving in that direction. Hopefully, we will see an increased Microsoft presence at such events.

Microsoft is also supportive of the new Apache incubation proposal - Stonehenge - that was proposed by WSO2. It will focus on building a set of sample applications based on approved W3C and OASIS standard protocols with goal of proving interoperability between different implementations on various platforms.

I think these sample applications will provide developers a great starting place for their tasks by providing best practice guidelines and reference implementations on various platforms. They will also help find potential interoperability problems and hopefully develop into a great community to discuss the architecture of multi-tier SOA apps. We look forward to working with WS02 on the scope of this project, and having discussions with the community.

These are exciting times for the software industry and we are seeing the co-existence of commercial and open source software and coming together of various forces to create solutions for the new heterogeneous IT environment.

Onwards, with great faith and hope!

Network Monitor to Open Source Parsers on CodePlex

Paul Long — Fri, 07 Nov 2008 16:49:00 GMT

Today I am excited to announce that the development of protocol parsers for Microsoft Network Monitor is moving into an open source model, hosted on CodePlex. This site will host development of parsers for public protocols and for protocols described in our Open Protocol Specifications for Windows.

Network Monitor is a free protocol analyzer and network sniffer. It allows you to capture and view network traffic in a format that is easier for humans to read. It is often used as a troubleshooting/development tool or to validate that a protocol is behaving as you expect.

We've started the ball rolling by releasing an updated parser package and creating a source tree on Codeplex. While this process will take time, we hope to have all bugs filed on the site within a few months, as well as having all parser development taking place directly on CodePlex.

Every month we'll post a new installer package that Netmon users will be able to install, so as to benefit from the latest changes.

As we continue to evolve, we hope that the community will get involved by filing bugs and suggestions, contributing code and new parsers, and helping us improve how information like summaries and field descriptions are displayed.

At some point we also anticipate that some of our committed users will manage subsets of the parsers.

We are also really excited about the impact of making our parsers open source. The force of the community should help us keep up with the quickly changing world of new protocols and updated documentation.

If you want to contribute, please join the community on CodePlex and start giving us feedback.

For historic background, Network Monitor 3 was a complete re-write of the Network Monitor program that previously shipped in Windows Server and SMS.

One of the major design changes is that the parsers - code that describes how network packets are decoded - are written in a custom language and included with the product.

Most industry protocol analyzers include parsers as static compiled code, or DLLs, which make them harder to update and maintain. By contrast, because the Netmon parsers are run within our execution environment, they can provide a layer of protection against overruns and therefore help protect the user from poorly written code which can expose security vulnerabilities.

At the same time that Network Monitor 3 was being developed in 2004, the product teams were in the process of creating the documents for the Microsoft Open Protocols.

So, at that point, we were able to partner with our document writers and get parsers written for these open protocol specifications for Windows. Not only did this help us verify the documentation, but it also provides a strong base of parsers for Network Monitor that makes troubleshooting network traffic very transparent.

For the latest version of Network Monitor, visit our download page.

For more information and tips on using Network Monitor go to our blog.

Thanks and enjoy!

struct.new("future", :open, :microsoft)

Sam Ramji — Thu, 06 Nov 2008 08:49:00 GMT

I delivered the keynote at ApacheCon in New Orleans today, where I talked about some of the new milestones we have chalked up on the journey inside Microsoft towards greater participation and growth with open source communities, and our strategy of "architecting for participation."

This strategy focuses on four significant themes: community; contribution; partnerships; and choice. Microsoft believes that the next ten years of software will be a time of growth and change where both open source and Microsoft communities will grow together.

We also believe that in an increasingly interconnected world, where more people have a greater opportunity to use more technology to do more things than ever before. We support those choices and are expanding interoperability between open source technologies and Microsoft technologies.

So, on the interoperability front, we have been working with the WS02 since our TechEd 2007 Conference, to demonstrate interoperability using our StockTrader reference application. Today, the WS02 announced they would build an open source version of the sample application under "Project Stonehenge," which hs been proposed as a new Apache incubation project.

WS02 will use the project to set up sample applications that demonstrate seamless interoperability across multiple underlying platform technologies, using currently defined W3C and OASIS standard protocols.

My team has been working closely with that of Jean Paoli, the General Manager of Interoperability Strategy at Microsoft, whose team is driving much of this interoperability work. You can read more about all this in Jean's blog post.

Microsoft has also decided to move the development of protocol parsers for Microsoft Network Monitor - a free protocol analyzer and network sniffer - to an open source model, on CodePlex, which will host the development of parsers for public protocols and for protocols described in our Open Protocol Specifications for Windows.

An updated parser package has been released and a source tree created on Codeplex. We want Netmon to be the best-of-breed tool for network monitoring at Microsoft, not just for Windows.

Microsoft also recently joined the AMQP Working Group as a participant, with the goal of contributing towards the development of the specification and to enable greater customer choice in the marketplace.

At the request of community members, we have now committed to participate in the Apache Qpid project, a widely adopted open source implementation of the AMQP specification that addresses the customer need for choice and improved messaging interoperability.

Our customers are telling us that they would like to see the Apache Qpid project extended to interoperate with Windows, so the next few months of participation will be focused on understanding the community's effort to build Windows based AMQP software.

Participation will give us the opportunity to learn from other project participants, so that we can be in a position to consider making a valuable contribution. But it is important to note that the Apache Qpid project is just one of many AMQP specification implementations, and we are open to supporting additional projects.

You can read an interesting technical research paper from Ohio State University analyzing the performance of the Qpid implementation of AMQP here.

Microsoft also announced, at PDC 2008, our commitment to include "Oslo" - an upcoming set of technologies for modeling - in the Open Specification Promise. This will ensure that the "Oslo" declarative modeling language, codenamed "M", is interoperable with prominent industry standards such as WS* specifications, XML formats, industry protocols, and security standards.

Two of the core focuses for Oslo are integration and interoperability. As such, it will integrate with next-gen Microsoft technologies, including System Center, Visual Studio and BizTalk Sever. We also plan to work with partners and the industry, so as to make Oslo interoperable with important standards and industry protocols.

One of the key ways we think customers will achieve customization for their platforms is through the use of textual and visual DSLs, which can be written uniquely by the developer for vertical industries and specific domains, or they can use pre-existing DSLs in these same scenarios.

The hope is that we will establish a broad and open ecosystem around "M" that will enable customers to bring the power of model-driven applications and systems to their heterogeneous environments.

Finally, on the Live Search front, the Powerset team recently resumed its participation with HBase, which is elated to infrastructural storage technology enabling large scale data processing.

The HBase project receives significant lift from the active community that supports the project, and Powerset's continued participation on HBase could allow us to accelerate the integration of Powerset's technology into Live Search, resulting in improvements to the end-user experience.

So, stay posted. There's a lot more to come!

Infrastructure Management and Strategic Design: Part 4 – Service Management Frameworks

jcannon — Tue, 01 Aug 2006 16:51:00 GMT

Today, the IT departments offering and managing various IT Services might find themselves in what I would call a “pressure-cooker”. They are faced with a multitude of tasks and added pressure to maintain daily operations while driving efficacy, managing the growing complexity of Service Offerings and most importantly, doing so while keeping pace with the industry best practices. This has been one of the most explosive areas of growth and re-examination for the past few years. Back in my Ops days, I trained under ITIL i.e. IT Infrastructure Library and MOF i.e. Microsoft Operations Fundamentals to get a first hand look at some of the best Service Management practices in the industry. No matter how good I thought our Service Management practices might have been, I could not help but to think in terms of the maturity level of the Services that can be achieved by applying these principles. When you get down to it, you realize that the heart and soul of effective Service Management lies in how mature the offering and support model is. I have learnt a lot from the ITIL Service Management Essentials course, which I attribute to research and practices that have gone into developing these models. I’d like to share w/ you what made sense to me:

Bridge Concept: As described in various ITIL formats, Service Management can be referred to as the “bridge layer” between Business and Technology. It is through the conduit of Service Management that core business needs as well as core technologies find their match with each other. This is attained by aligning the Business Needs and Goals of the organization with the various technologies and IT functions that can map to these overall goals
Think Framework: Once you have scoped out, what may be a void in the service hierarchy of your organization, make a commitment to implementing a Service Management and Delivery framework that is suitable for your org. One size DOES NOT fit all and although I am only talking about ITIL concepts here, there are several methodologies you can explore before making the plunge.
Benefits of implementing a Service Management Framework: In just a few minutes after sitting in the ITIL Essentials Training class I was able to get a very crisp idea of what the benefits of implementing a Service Management Framework are. To name a few –

Raising the bar on Service Delivery Quality
More accurate alignment w/ Business Needs
Enhanced relationship between service provider and consumer
Deeper visibility into service complexity
Driving efficiencies with optimal resource utilization

Goals for implementing a Service Management Framework: if and when you do make the commitment to implementing a Service Management Framework, here’s a quick set of goals you can set when you proceed with the implementation of Service Management:

Put an SLA ( Service Level Agreement) or an SLO (Service Level Objective) around the critical and non-critical services you offer, respectively
Manage and monitor the implementation and practice of these SLA’s and SLO’s
Create and publish a service catalog describing the services offered by your IT department/division
Monetize the service management offerings, even if your customers are internal. This will help quantify the service effectiveness and bring measurability across the board

Additional Resources: the itSMF or IT Service Management Forum (www.itsmf.com ) is an independent, non-profit, user group distributed all across the world that has dedicated itself to exploring and promoting IT Service Management concepts and practices.

I am very eager to hear back from those of you that are an integral part of the Service Management Lifecycle. Please share your experiences, challenges and learning with us.

Kindest Regards and have a great week ahead!

Infrastructure Management and Strategic Design: Part 3

jcannon — Tue, 11 Jul 2006 20:17:00 GMT

Part 3 – Adaptation and simulation of Heterogeneous environments under lab conditions

A simple question that has always perplexed me is how software and hardware OEM’s across the world simulate heterogeneous environments under lab conditions. I have witnessed several different approaches, practices and stages of this adaptation and each one of them is unique and correct in its right and merit. I guess, that leaves the “big” question which remains unanswered i.e., how do you bring a “real-life” scenario and manifest it under lab conditions. This is even more challenging because the average test lab for a medium to large organization is no match to the size and complexity of its elder sibling, the Enterprise Data Center, running its production systems, applications and operations. So why squeeze all that complexity into a smaller scale ? Is there one perfect method?– of course not, depends on what heterogeneity means to you/your business. Let’s look at this and why it’s necessary and also share some techniques that may be helpful.

Start with why it’s necessary to represent if not an equivalent amount of heterogeneity within a lab but a comparable one. Start with simple logic – why do we need a lab in the first place ? In most cases it’s an environment we can turn to and run processes, tests and simulations which we dare not try in a Production Environment. However, the caveat here is that if we do want to test a tool or an app that we’re about to roll into a production environment, our best bet is to test it in the lab with conditions mirroring as closely to the production environment as possible. It’s also a place where we can develop workarounds, fixes, documentation, implementation practices and as much supplementary support mechanism as we’d like before we bite the bullet and push the tool or app into production. The expectation we keep in mind when we do that is that results from the lab and production rollout should bear a resemblance like that of the “Partridge Family” and hopefully not of the “Manson Family”. Okay, bad joke but you get the point.

Now on to “Tips and Tricks” to help with the process of adaptation and simulation of a lab environment that mimics your production one. Here’s what I found useful:

Deployment Methods: Using similar deployments tools, techniques and methods in the lab that are already in use in the production environments makes one aware of “delivery mechanisms” and the path, process the deployment cycle will take when released
Configuration Management: Extreme familiarity and knowledge of the configuration options of not just the delivery mechanism/s but also of the tool/s or app/s is something as valuable as having that Swiss knife in your pocket – you just never know when you’re going to need it
What Business Scale ?: Never hesitate to walk out of the lab and have a conversation with decision makers who chose the tool/app. Find out more about what their expectations out of this application are (by now I know some of you may be cringing in your chairs but I am dead-serious on this one). This is the best way to learn if the application should be tuned towards business scales such as Reliability, TCO, Scalability, Performance, High availability or whatever
Manageability: My personal favorite – always have a lifeboat handy i.e. when the fit hits the shan, will you still be able to recover the system, do a roll-back, connect remotely and most importantly, keep the service/s up and available
Driving Efficiencies: Most IT departments have to squeeze every efficiency they can out of their budgets, and labs are a luxury when they have to deliver results to CTOs. So what’s the best way to accomplish testing, or simulation, on a budget. How does someone with no extra money support such an effort. There’s some creative resource utilization that can be implemented such as:

Rotation of production hardware coming up for decommissioning and reallocating such resources to the lab
Making use of evaluation copies and licensing i.e. since most lab testing scenarios only extend to short periods to drive testing
Using down-time to allocate personnel to testing efforts i.e. if there’s lag time between two projects, using that time and headcount effectively to drive testing

And finally a small anecdote to help put things in perspective. In my past life, I remember several years ago when I was still on the east coast, I worked on implementing an asset tracking tool for desktops spread through the environment. We tested the tool on individual desktops and did not care about running the entire scenario using network connectivity across the simulation. We were told by the vendor that the tool uses less than 1% of CPU as negligible amount of memory. After random tests, we rolled out the tool and the purpose of the tool was to run a script and send the results back across the network. However, due to ACL’s in place, which we forgot to account for, and lack of validation of packet delivery, the desktops stopped responding. This was an expensive lesson in why we should test the waters to the best possible extent before setting sail.

Just a few thoughts and hope it triggers some more for everyone out there. As always, please do let me know if that has been useful and/or if you have a specific topic in mind you’d like us to write about.

-Kishi

Infrastructure Management and Strategic Design: Part 2 – Driving Network Efficiencies

admin — Fri, 16 Jun 2006 15:21:00 GMT

Infrastructure Management and Strategic Design: Part 2 – Driving Network Efficiencies

Every computing device in existence today lives and breathes on some sort of a Network. Doesn’t matter if your Home PC is connected to a Cable Modem or if you’re office laptop is part of an extended WAN, your device is persistently living and breathing on the Network. From the minute you turn on your device, one of the very first drivers to be loaded are Networking drivers. Everything from DNS, DHCP to the simple sharing of email and office documents depends on the very basic function of the Infrastructure, the Network. So why is it that the same Network which is considered a Category-A asset in an organization is also sometimes at the very root of serious headaches for a Network Administrator. Thoughts about the faith and reliance we put towards Networking Infrastructure is simple staggering. Let’s examine why

If you’re a soccer fan like me, I am sure you have been glued to the World Cup coverage. As I sat with some friends and discussed our loss to the Czech Republic, I realized that in that group, it didn’t matter if you use Firefox or IE to get the game stats, you JUST wanted to know the score. The thought that followed was that maybe “Tools” shouldn’t be as hyped-up as they are sometimes as compared to the stream of “networking” that sustains them. Following that my thoughts turned to the topic of Net Neutrality and how Networks are ideally designed to forward packets regardless of their size, purpose and content. Hmm…this is interesting, I thought. My entire life as an Infrastructure Architect, I tried to come up with creative ways to manage and optimize network / infrastructure performance towards a better outcome for end users.

You see, some of the most perplexing problems for Network Admins is to control and manage “chatty protocols, broadcasts and bandwidth hogs”. How can we do that effectively? Let’s look at this closely: When I read a blog about Web 2.0 or something similar, some of the questions that pop in my head are “Have they considered what effect the implementation would have on their Network Performance” ? How far does the implementation of a new model go towards Application Bandwidth Testing and how inherently and intrinsically dependent we are on Networking.

Networking and the simple availability of bandwidth, wired or wireless, has become as much of an expectation as running water. A few months ago I remember someone saying “Network is like electricity – you don’t call up and thank the power company when you turn on the light, do you, nor do you ask them how much power you can use for your house”. Intriguing thought, nevertheless, for those Layer 2 and 3 warriors, the term “Port Saturation” should definitely ring a bell. Key contributors to reaching port density are instances when there just isn’t any more bandwidth to go around. To avoid choking up the network, I found some of the following tips helpful if you’re managing a small to medium sized network:

Auditing protocols in use: A good place to start overhauling your network can be drawing a good Physical and Logical representation of your company’s network. This will compel you to examine how majority of the devices are connected to each other as well as the medium, physical as well as protocols in place. If you have access to a Sniffer or a Fluke, you might also want to do some packet analysis and see where a majority of the “chatter” is coming from. That in turn will expose what some of the “bandwidth hogs” on your Infrastructure are, be it Application, Devices or Servers
Auditing ACL’s and Policies: If its been a while since you reviewed your network security policies, it may be a good time to do that. The ACL’s you put in place on various switches across your network were driven by the security policies drafted formally. An ACL audit will help provide fresh transparency into the security elements within your network

And the tricky one “finding alternatives to chatter-heavy Tools and Apps”: Once you have identified the bandwidth hogs on your Network, a good bet would be to limit the broadcasts and chatter they generate. Segmentation and or isolation of these tools into separate VLAN’s may be a good starting point. If an application or tool in question is culprit, investigate how many users or mission-critical LOB’s are dependent on it. If the priority is fairly low, it may be time to migrate to something less chatty or simply more efficient

Alright, that’s it. I am logging off and heading to the movies to watch the Da Vinci Code. Will be back here next week and Thank You for tuning into Port25.