Port 25: The Open Source Community at Microsoft : Software Testing

Releasing the Source Code for the .NET Framework Libraries

jcannon — Wed, 03 Oct 2007 21:47:00 GMT

Some news from Scott Guthrie's blog this morning - we'll get more from the .NET and Shared Source team over the next few months:

"One of the things my team has been working to enable has been the ability for .NET developers to download and browse the source code of the .NET Framework libraries, and to easily enable debugging support in them.

Today I'm excited to announce that we'll be providing this with the .NET 3.5 and VS 2008 release later this year.

We'll begin by offering the source code (with source file comments included) for the .NET Base Class Libraries (System, System.IO, System.Collections, System.Configuration, System.Threading, System.Net, System.Security, System.Runtime, System.Text, etc), ASP.NET (System.Web), Windows Forms (System.Windows.Forms), ADO.NET (System.Data), XML (System.Xml), and WPF (System.Windows). We'll then be adding more libraries in the months ahead (including WCF, Workflow, and LINQ). The source code will be released under the Microsoft Reference License (MS-RL).

You'll be able to download the .NET Framework source libraries via a standalone install (allowing you to use any text editor to browse it locally). We will also provide integrated debugging support of it within VS 2008.

Integrated Visual Studio 2008 Debugging Support

The final release of VS 2008 will support the ability to configure the debugger to dynamically download the .NET Framework debugger symbols (and corresponding source code) from a web server hosted by Microsoft. You'll be able to configure the .NET Framework symbols to be downloaded all in one shot, or manually retrieved on demand:

Read the rest of the post at Scott Guthrie's Blog....

Systems Manageability Part 4: Systems Configuration

kishi — Fri, 25 May 2007 18:18:00 GMT

Background: This is Part 4, continuation of the series of 8 blogs I’m doing on Systems Manageability. In this specific blog, I will focus on and explain the second part of the “ontology” which is “Systems Configuration”

Level-Set: System Configuration and Management encompasses all tasks related to the configuration of a host in a standardized and (when possible) centralized way. Many projects in this category provide a common configuration interface, either command-line or GUI-based, designed to ease typical administrative tasks. Other projects, specifically Cfengine, provide a higher level policy-based system to provide consistent configuration and state management for a set of systems. Again, in this case there’s lots of different tools out there that can be used but we have focused on the most popular ones such as Webmin, YaST, SSH, VNC and Cfengine. In the paragraphs to follow, we have attempted to lay out our understanding of these tools after using them in the OSSL:

I. WEBMIN: "Webmin is a web-based interface for system administration for Unix. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on." Webmin is very modular in design, allowing third-party developers to add support for a particular service or task relatively easily. Many of the tasks involve easing or automating system administration tasks, or editing a configuration file using a specific syntax. Webmin is currently supported by OpenCountry, a company that sells Linux management solutions. The OpenCountry website includes information about Webmin, including two variations of the system that they support.

Webmin Plus: Webmin Plus, is a version of Webmin supported by OpenCountry. According to the website, the Webmin Plus version has been tested by the company and includes new features, specifically support for the back-up and restore application called Bacula (http://www.bacula.org/). Webmin Plus is still freely distributed under a “Mozilla-type” license, and is downloadable via Sourceforge.
Webmin Pro: Last I checked, Webmin Pro had not yet been released, but the following is an excerpt about the product from the OpenCountry website: “...a commercially supported comprehensive product for the entire data center enabling centralized systems administration of mixed distributions, Linux and Windows!”

II. YAST: YaST (Yet another Setup Tool) is an OS installation and configuration utility used primarily in SUSE-based systems. YaST typically serves as the primary control panel interface in, and can be used for a number of configuration tasks – such as adding and removing software, patch management, user management, device configuration and for configuring individual services and daemons. Other common administration tasks such as obtaining system information and reading server logs is also possible via the YaST interface. All of the aforementioned YaST features are implemented as modules, each of which provide a specific functionality or perform certain tasks. These tasks typically involve editing one or more text configuration files on the system in a specific format to configure a specific service or daemon. On other Linux or UNIX-like systems, these tasks are typically performed manually via the command-line.

The YaST utility is very modular in its design, allowing Novell or other third-party providers to add modules into the YaST interface to configure a particular device or service. Many of these modules work independently of each other, and as such are often packaged as individual RPM packages that may be added or removed depending on the software and devices that are installed on a system. YaST modules are written using a scripting language specific to YaST called YCP. Other scripts, such as Perl or shell scripts can also be utilized via a YaST module to perform a particular task. A CIM module for YaST is also distributed with SLES10, which provides a client interface for CIMOM (Common Information Object Manager) to other YaST2 modules. It seems the most common administration task for which YaST is used involves setting up individual package repositories (discussed further in the Patch Management and Maintenance), adding or removing software packages and configuring or initiating online updates. YaST is capable of searching for and locating software on remote repositories, retrieving the software packages, resolving package dependencies, checking the cryptographic signature of the package (if available) and then installing the software on the system. Multiple repositories can be configured. Repositories can be located on a hard disk or CD/DVD, or on a remote system obtainable via HTTP(s), FTP, NFS or CIFS. Once a repository is configured it can then be indexed for later searching. The software search functionality is very powerful, allowing one to search for appropriate software packages using many of the attributes available in the RPM package header – such as the description or contents of the package. Besides software management, the quality and completeness of many YaST modules varies. Many modules (such as the log viewing modules) offer minimal functionality, and only work well enough to provide a few basic configuration options. Complex server configurations will therefore still require one to edit text-based configuration files by hand, or use another configuration engine for the task, such as Webmin. However, many other common tasks, such as configuring display settings or a printer, can be done entirely via YaST.

III. SSH/SCP/SFTP: SSH (Secure SHell) is likely the most widely used remote administration tool for Linux and UNIX-based systems. The typical SSH toolset includes the SSH client and server, as well as the SCP and SFTP client applications for copying files, both of which simply utilize the ssh binary on the backend. The following excerpt is from the OpenSSH project home page: "OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions. Since most any task can be performed via the command-line, the OpenSSH utilities are likely the most critical component for a Linux administrator to have available. The remote copy and command execution options allow one to build, deploy and run a script on a number of machines relatively quickly and securely. OpenSSH is typically installed by default in most Linux-based distributions, although in some distributions the server may by default be disabled or blocked by the firewall. "Most common uses of SSH are:

Remote Command Execution: One of the most common uses of the ssh utility is to run commands in remote machines
Authentication: In order to log into, copy files or run commands on a remote host via ssh, one must first authenticate to the remote machine. Local password authentication may be used without any particular configuration on the server. Linux systems that support PAM (Pluggable Authentication Modules) may also use password authentication to authenticate using a number of mechanisms, including NIS, LDAP, Kerberos, or AD (Active Directory).Rather than requiring a user to manually type in a password, a more common authentication option is to use public key authentication. A user’s public key must be inserted into a file on the remote host called authorized_keys before they are allowed to authenticate to that server. This can often be done via the post installation procedures built into kickstart or autoyast.
Tunneling other Applications: Another common application is to use SSH for tunneling other protocols. This is often used to tunnel protocols that are not typically encrypted such as NFS or X11. By using the –Y or –X switches with the ssh client application, one can “turn on” X11 tunneling, allowing graphical applications run on the remote machine to display locally on the administrators workstation. For example, if one were to SSH into a SUSE-based system and run “yast2” via the command line, the YaST display would be tunneled via SSH and displayed on the local system – even though the actual application is running on the remote system.
Cluster Management: When managing a number of Linux servers, such as a HPC cluster, it is typical to have a large number of systems with identical configurations. To ease administration of these systems there have been a number of SSH-based utilities that allow one to run commands on multiple systems, or copy a file to multiple systems, in parallel. The following utilities are all licensed under the GNU GPL.

ClusterSSH: ClusterSSH allows an administrator run to open a remote SSH session to a number of systems at once via a single terminal window, and run commands or alter configuration files on all the systems simultaneously.
PCP: PCP is a tool designed to copy files in parallel to multiple nodes in a cluster or server farm.
GEXEC: GEXEC is a tool that is somewhat complimentary to PCP that provides a parallel remote command execution system for large clusters. The system includes a client and server, as well as a library that allows integration into third-party applications.
PSSH: PSSH is distributed as a suite of utilities that perform many of same functions as ClusterSSH, PCP and GEXEC.
pssh – Parallel SSH, similar to ClusterSSH or GEXEC.
pscp – Parallel SCP, allows one to copy files to multiples nodes in manner a similar to PCP.
prsync – The prsync utility automates running rsync on multiple nodes in parallel, essentially another method of copying files or entire directory trees to a number of remote nodes.
pnuke – The pnuke command can be used to kill a number of processes running on multiple nodes.
pslurp – The pslurp utility is similar to the pscp utility, except that it is designed to copy files from a set of hosts. This allows one to copy, for example, a log file that exists on multiple nodes, to a local directory tree.

IV. Cfengine: “Cfengine, or the configuration engine is an autonomous agent and a middle to high level policy language and agent for building expert systems to administrate and configure large computer networks. Cfengine is designed to be a part of a computer immune system. It is ideal for cluster management and has been adopted for use all over the world in small and huge organizations alike.” Cfengine consists of a userspace application called cfagent and a host of other utilities that reads and parses a series of text configuration files and performs tasks on the host system based on the configuration. The configuration syntax of Cfengine is actually a high-level policy language that allows cfagent to test the system’s configuration and perform corrective actions based on those tests. For example, cfagent may test to assure that a certain line of text exists within a configuration file, and if not it will add the text and restart the associated service. The cfagent utility is typically run on an hourly (or so) basis via cron, a task-scheduling application. This assures that mis-configurations will be found and corrected within a reasonable time frame.

The policy simply tests to make sure an entry for user root exists within the /etc/shadow file, and also checks to make sure the password matches. This assures that all systems have the same password for the root user. The configuration of Cfengine can become very complex, which would likely not surprise those who have had experience with the tool. The structure of the policy language eases this dilemma a bit, as platform definitions can be made and inherited by other blocks to help determine the appropriate action to take. The configuration is essentially a high-level policy language, and thus the various tests must be built and scripted manually. The toolset is, however, enormously powerful when implemented correctly. But as with many open-source technologies, the learning curve can be quite steep, and one must study the complexities of the tool before it can be competently used in a production environment. A version of Cfengine has been ported to the Windows platform to run under Cygwin.

editfiles:
                    # We have different passwords for lab systems and workstations.
       linux.shadowpasswords.md5passwords.(!workstations)::
{ /etc/shadow
    SetLine "root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::"
    AppendIfNoLineMatching '^root:.*'
    LocateLineMatching '^root:.*'
    ReplaceLineWith “root:$1$383J33RL$ XXXXXXXXXXXXXXXXXXXXXX:12984:0:99999:7:::”
}
       linux.shadowpasswords.md5passwords.workstations::
{ /etc/shadow
    SetLine “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
    AppendIfNoLineMatching ‘^root:.*’
    LocateLineMatching ‘^root:.*’
    ReplaceLineWith “root:$1$gcGWA0qS$YYYYYYYYYYYYYYYYYYYYYY:13027:0:99999:7:::”
}
Example Cfengine policy to check the password for the root user.

The following example Cfengine policy checks for the existence and the contents of the /etc/cron.d/yast2-online-update file for SUSE systems. If necessary it creates the file, and writes a cron entry into the file to schedule a daily check for updates and patches. Upon completion, it then runs the command “/etc/init.d/cron restart” as defined in the suse.restartcrond definition.

editfiles:
        suse::
                { /etc/cron.d/yast2-online-update
                  DefineClasses "restartcrond"
                  Umask 077
                  AutoCreate
                  BeginGroupIfNoLineMatching "^.*[\s\t]+root[\s\t]+online_update"
                      AppendIfNoSuchLine "30 3 * * * root online_update"
                  EndGroup
                }
shellcommands:
        suse.restartcrond::
                "/etc/init.d/cron restart"

Example Cfengine policy to assure that SUSE systems check for updates daily.

And that does it for the “Systems Configuration” section As always, please let us know if you found the above mentioned useful and any comments/feedback you may have. Thank you for tuning into Port25.

Systems Manageability - Part 1: Why Manageability Matters

kishi — Wed, 21 Mar 2007 20:09:00 GMT

Impetus: This is the 19th year I have spent in the Information Technology business, out of which more than 15 were spent designing and implementing IT environments of various scopes, platforms and sizes. Among several similarities and differences between each implementation, a few constants always emerged, and my favorite of all: Systems Manageability. This question always got asked during every high-profile design review meeting. You know the meeting I’m talking about, the one with the CIO and the IT Director are sitting across the table and asking how we’re planning on managing the environment. The main concern you could see in everyone’s expression was “what sort of manageability needs to be built around for operations and support ?”. So why does manageability matter ? Let’s start with what people imagine, when they think of Systems Manageability. It means different things to people in different roles: the Infrastructure folks imagine uptime and redundancy, the Developers imagine reliability, the Business Managers imagine efficiency. But the answer is so overwhelmingly obvious. Had it not been for systems management toolsets, apps and frameworks, ITPro’s and Admins would be worried sick about everything from uptime to reliability to scalability. So yes, manageability matters, a lot, because it’s the knowledge that “all’s well and running smoothly” that matters to everyone from a CIO to a Developer to a IT Admin.

Importance: Now that we have established the impetus behind why Systems Manageability matters, we should now address the importance tied to it, such as:

Increasingly complex and heterogeneous environments need increased attention: Pick any environment of your choice, whether it’s where you work or the one you hear about the most. It comes with its own share of challenges and oddities. What differentiates one environment from the other is the manner in which it is “run”. The more complex you hardware, software or network stack, the greater is the manageability tied to it. This means that with every component you add to your existing system, you’re only increasing the complexity even further. So is that a bad thing – NO. Because it’s the environment that should be designed with “adaptation” in mind and not the other way round. Thus, having a complex environment simply translates to keeping an eye on more things. Systems Manageability plays a key role in this scenario
Infrastructure sits at the “core” of IT: Here’s an exercise for any of you that may be interested – the next time you see or hear a technology professional being interviewed about a certain “application” or tool that they’ve developed – try to imagine an entire infrastructure that needs to support and run that “tool” or “App”. The exercise will make you think about what’s going on in the mind of an IT Admin who is responsible for running your environment implementation and why Infrastructure is a big deal. Systems Manageability plays a key role in this scenario
End-user productivity has a directly proportional relationship to Performance Tuning and Optimization: Growth comes in spurts and bursts and never an even pace as we all know. This means, various pieces of your hardware, software, toolsets are implemented in various growths and phases. Simply put, growth does not follow a pre-chartered course and timeline. This makes ongoing performance tuning and optimization a necessity. And it has its benefits - it allows you to see the app or toolkit from the eyes of the people who made it. It also gives you the “know how” to make the specific changes in thresholds and values that could mean a difference of night and day, in terms of efficiency. Systems Manageability plays a key role in this scenario
Striking a balance between Manageability and Flexibility is tough: Identity Management and Security issues are now a mainstream topic of discussion and more and more attention is being paid to system security, access and authentication framework. The single reason for putting these in place to ensure that “you are who you say you are”. On the flip side, overly managed and restrictive environments can limit the flexibility of what an end-user can do. That’s why it is always tough to strike a balance between a system that is secure / well managed and one which seems more “seamless”. Systems Manageability plays a key role in this scenario

In the next seven blogs or so to follow, you will get a detailed breakdown on the Systems Manageability project that we have completed in the lab. We will be covering the Project Methodology and Project Ontology in my next blog. As always, send us your comments and feedback and THANK YOU for tuning into Port25.

Who really needs to gather crash information and what do they need to do with it?

anandeep — Wed, 07 Mar 2007 04:06:00 GMT

I just got back from Cambridge (in the United Kingdom, not the one by the Charles river) from the Microsoft Research / Technische Universitat Darmstadt “Reliability Analysis of System Failure Data” conference.

Microsoft Research has a very nice lab near Cambridge University. This was my first visit to Cambridge and I was able to drink in (literally!) some of the local color. I went to the pub (called the “Eagle”) where Watson & Crick had their “aha!” moment about the double helix structure of DNA. I also visited the pub next to Queen’s College called the “Anchor” which Pink Floyd’s Syd Barrett used to frequent.

But it was not all pub-crawling, we had some serious stuff to deal with at the conference. The objective of the conference was to bring academia and industry together to deal with a problem in the field of reliability analysis. The problem is that industry (there were representatives from Sun, Cisco, IBM and of course Microsoft) had the failure data but NOT the models and techniques for solving the problem overall. Academia had the models and techniques but NOT the kind of failure data that it needs to solve the overall problem. This conference was an attempt to get the two sides together and find a solution to this conundrum.

After we had presented our position papers (our lab’s paper is here), we split into workshops on Data Collection, Data Repositories and Data Analysis. The idea was to come up with the “next step” in taking reliability analysis of system failure data forward.

In the Data Collection section, we were stuck for a bit. We were trying to look for compelling reasons for data collection by end-users (rather than by the software makers like Microsoft, Sun or IBM). What reason would an IT department have for implementing mechanisms to collect failure data?

At first the reasons seemed obvious – to monitor for failure and to correct defects of course. But then the representatives from the software makers spoke up and said that they did collect failure data and were using it exactly for the purpose of correcting defects. Vince Orgovan from Microsoft stated in his paper that almost 400 million PCs provide data to Microsoft. Not only was the data available but Windows XP supported corporate error reporting in exactly the same way that it supported error reporting to Microsoft. It just needs changing a few registry keys to do this. The Windows debugger “!analyze” can be used on the error data, much as it is used internally.

This took the wind out our collective sails. If shipping software was providing all these mechanisms what could we suggest as a next step that had compelling value? Most corporations would like to leave the job of correcting defects to the software makers (proprietary or open source) anyway! The software makers were in a much better position to look across many deployments and correct defects in the software.

The only compelling reason we could come up with to build a mechanism for data collection was to help with deployment on the user side. The data collection mechanism would collect failure data during the preliminary testing. This data would then be fed into a model that could be used to judge the maturity level of the deployment. Kind of a CMM (Capability Maturity Model) for reliability. We even suggested that we have an ITIL management practice around this. This would potentially allow the ITIL model to not only give good qualitative measures like it does today but to quantify the reliability of a deployment.

This in itself is a very useful thing to have, but I cannot believe that it is the only reason that we would collect failure data at the end user level. Let me know if you think of any others.

Open Source would have much the same issues but for the fact that there is not a central organization that collects all this failure data. The situation in Open Source may be the reverse of the situation for proprietary software makers in that the failure data is collected at the IT organization level and not centrally. How does this failure data really result in code defect corrections? I guess that it is either pre-analyzed and submitted as a bug or people patch their own instances of the source code. But my opinion is that eventually open source software systems will have to build central repositories of failure data in much the same way that commercial software vendors have built them.

Unsexy Development

anandeep — Fri, 27 Oct 2006 14:27:00 GMT

I loved doing development in a research and university environment. You got to write cool code, prove new ideas, break new ground and generally ended up with bragging rights to say “I did an image recognition algorithm on a multi-layer architecture implementing reactive and planning parallelism on an autonomous robot!” The code had to work on your workstation or maybe on a demo machine once. Once you wrote the code, the only people who touched the system were hapless graduate students implementing the next big idea. They had to come to you and you could then dazzle them with your insight! This was “sexy development”!

When I moved to industry and wrote software for day to day use – things changed. Now you had all those people with “manager” titles telling you what to do, and those people called “testers” who told you why your code sucked (you couldn’t logically argue your way out of that because the weasels usually had proof)!. Of course being consummate professionals you adapted. You got the religion of “bullet proof code” and worked on making sure the testers only had “fit and finish” bugs filed against you. Which the intern could work on. That was still fun - a different challenge maybe not as “pure” as designing a neat new algorithm but pretty good nevertheless!

You got past the testers but when they integrated the components that you had bullet-proofed to run end-to-end or user acceptance tests, unexpected stuff happened. Who would have thought that they would configure the machine that way or that another non-surface component could pass you null strings. Now you had to plan not only for the testers – but also for other developers and those pesky sys admin guys. How did they become sys admins? They couldn’t tell a polynomial solution from a log n solution anyway! But being nothing if not adaptable you adapted. You now built bullet proof AND idiot proof code. (My father, a military pilot and flight instructor, when teaching flight safety used to say “Nothing is foolproof because fools are so ingenious!”). It got a little boring at times but you still had the satisfaction of building something that was “engineered”.

I thought I had shipped the product but I found couldn’t sit back and relax. The support guys were making insinuations against my code. It didn’t work they said – and you hadn’t put in the right level of granularity in the logs for them to do a diagnosis. This had nothing to do with Computer Science – any bozo could write stuff to the log. Why didn’t the intern do it? What do you mean he can’t make sense of my code? Yeah, I do know my code best. I guess it’s the right thing to do. Certainly not as fun as designing, bullet proofing and idiot proofing new code but good supportability is “sine qua non” for a well done project!

Is that the end of it? No, further design and coding needs to be done for making software more manageable, to make the logs more systematic, to make sure that the product works when its deployed to multiple configurations, that it performs well and fails gracefully.

Unless you specialize in a certain aspect of manageability, reliability or diagnosis – this is not “sexy” development. I probably wouldn’t get as much satisfaction from designing event logs as I would from designing a new search algorithm.

I was getting paid to do all this (ok, so it was my own startup but I was getting paid in VC money!) and it was still very hard. We did do it but it took lots of coaxing of our developers to pay attention to this. They all preferred to work on the next release that had all the sexy features. Even though they knew that to make the startup successful and still have a job, the unsexy stuff needed to be done and done RIGHT!

When you are working for the “love of the game” and not money, like in Open Source – who coaxes you? Who does the unsexy stuff? Are there enough people who specialize in the esoteric aspects of event logs, that this is not a problem? Or do users who need the feature “just do it” and add the code to the community version? Or are things slipping through the cracks?

I did a sweep of the usual suspect Linux developer mailing lists and found that there is concern about whether unsexy stuff gets done. Here is a typical comment that I saw

“I think that the only issue with Open Source boils down to this:

The things that nobody wants to do, but somebody has to.

Nobody wants to think about documentation. Or user interfaces. These things are hard, tedious, and a hell of a lot more boring than actually coming up with stuff to "make things work".” (from here)

Documentation is famously one of those things that is considered “unsexy” (well, ok in commercial software too). There are efforts like Grokdoc to make documentation of Open Source projects sexy by making it a priority. But the “who does unsexy?” issue is a real concern in Open Source.

We ran into a similar issue with event logs. You know the text stuff you write so that you can find out later what happened. At the lab we just did an investigation of whether we could tell if one of our boxes had crashed from the syslog and from console messages. We were a little taken aback by how many times we couldn’t tell what states the machine had gone through.

On doing some investigation we found that the most influential project that was addressing this issue, the Evlog project (most supported by IBM) has been quiet since 2004. This code is used internally within IBM but was not mainstreamed into the Linux kernel.

How does one get unsexy stuff like this into the Linux kernel so that is comparable to UNIX/VMS/Windows?

I contend that it is critical to Open Source that attention be paid to the event logs. They are critical in making any operating systems reliable. VMS/UNIX/Windows all went through the process of making their event logs more meaningful – and this has helped make them much more reliable.

We will be addressing this further in the next couple of weeks – keep tuned!

Memtest

jcannon — Tue, 11 Jul 2006 18:32:00 GMT

*Updated* So, this weeks tech tip is about memtest, and yes, I am sure there are some that might scoff at this....But I think we have a tendency to loose sight of the basics. For instance, last week we had quite an interesting time debugging a problem that occurred intermittently and we where not able to find a way to consistently reproduce the problem. We ran through all kinds of things until we decided for grins and giggles to run memtest. And low and behold, it found memory errors. We replaced the memory and have not seen a recurrence of the problem.

We decided to pull this old but trusty tool back out of the stable. Special thanks goes to Kyle Adams and Stephen Zarkos who did a lot of the footwork on this one.

Memtest is a simple program that is designed for the x86 architecture. You would use it for things such as when hardware hangs or when your computer doesn’t boot at all. Either way, you could just grab memtest and throw it onto your computer.

Actually, there is not a hard and fast rule when to use it. There are two ways I would put it in the toolbox to use, and they have to do with methodology more than anything.

I have no clue what is wrong and I am completely out of ideas, I am just stabbing in the dark.
As a standard suite of checks and tests I do to debug a problem I will run memtest.

Honestly, I think number one is one that happens more in real life. A lot of people do not think that HW like memory will be causing any problems. They forget that often with memory it is not a black or white issue. It is not an all or nothing failure. It sometimes happens and sometimes it does not. I have never really seen a failure that I would say without question, that is memory!
You can get it here (Linux GPL, and windows version); http://www.memtest86.com/

(There is also a non GPL, but still free version for windows available here http://hcidesign.com/ I am not to familiar with how it works, but the web page gives you a lot on information)

One thing to note, this can run for a very long time, several hours in some cases.

GENERAL FEATURES
Memtest gives a user the ability to access the memory in an effort to pinpoint a problem in the memory itself. It uses a set of algorithms to check for consistency and errors in the placing of memory. The algorithms that are used by memtest to test the memory are the following:

Address test, walking ones, no cache
a. Fills in the address space with ones in a sequential order
Address test, own address
a. Puts the address of the test address in itself
b. Test for addressing errors
Moving inversions, ones and zeros
a. Checks the addresses using a series of ones and zeros
Moving Inversions, 8 bit pattern
a. Uses an 8 bit wide pattern to test for errors on “wide” memory
Moving inversions, random pattern
a. Creates a set of random numbers and its compliment, writes to address.
Block Move, 64 moves
a. Memory is initialized with 8 byte inverting patterns.
b. Moved every 4 MB
Moving inversion, 32 bit patterns
a. Shifts data patterns one bit for each successive address
Random number sequence
a. Writes a set of random numbers into memory
b. Checks the memory for consistency on the next pass
Modulo 20, ones and zeros
a. Uses the Modulo-X algorithm to check for errors not detected by inversions because of buffering
Bit fade test, 90 minute, 2 patterns
a. Initializes memory and then sleeps for 90 minutes
b. Checks memory after the 90 minutes is up

The point: applications still need error-free memory to execute correctly, especially today with application complexity increasing all the time. How do you replicate problems in your lab environment with such diverse environments across your network, or even more importantly, separate hardware from software failure?As always, comments/suggestions etc appreciated.

Hank Janssen

Testing software on multiple platforms - do we trust the tools or rely on methodology?

admin — Wed, 28 Jun 2006 20:13:00 GMT

Consider this scenario – the CIO’s office just called. They have decided to follow your recommendation and are going to use JBoss (substitute favorite open source software package here) as their application server (substitute database server/web server/whatever server here). They also decided to make it standard on both your Windows servers and your Linux/NetBSD (substitute favorite flavor of open source operating system here). Now, they ask you – will it handle the loads on both operating systems? They don’t want any of the servers to be idle after all that investment they made.

You have a reputation as a “data driven” dude, i.e. you look for empirical evidence in making decisions like this. No hand wavy – “it will work …. maybe” for you. You know your workloads cold. You know you will run test scripts to simulate those workloads.

The million dollar questions are (drum roll please):

Will you use a standard tool or testing package (such as Grinder with JBoss) that works on BOTH Windows and Linux ?
Will you trust your test scripting skills and your methodologies and write your own scripts – using whatever tools make sense on Windows and Linux?

Why are these questions important? The objective is to have consistent and comparable tests on each platform – and it is important to make sure that the testing process does not introduce any bias one way or the other.

So which approach - trusting the tools or trusting your methodology allows for consistent and comparable tests on each platform?

Lets examine some pros and cons of each approach:

Damn! The laws of physics strike again. Looks like there is no easy answer since the observer effect will always be present i.e. observing the software using testing software impacts the software being observed so that a precise observation can’t be guaranteed.

So what do you do in such a case?

My friend (and I think he works with me too) Hank Janssen had the following to say (I will not point you again to his handsome visage like I did in my last blog or people will start talking!)

If the same tools are available on both platforms, then use the same tools.
Stay with industry standard, and or OSS accepted methods of testing.
Have a list of ‘acceptable’ tools to pick from
Publish a list of methodologies/guidelines you are using (so that people can make their own judgments)

I am not so married to the tool as I am married to a consistent and approved process. There will be many different tools for different languages/applications/infra structures/protocols and the like.

I wanted to stay away from testing methodologies that are so different between different platforms that I would have a hard time interpreting the results. For example, using Loadrunner or Winrunner on Windows and than Grinder on Linux. (In case of Java loads) This example would result in apples to not so apples comparison. Most OSS tools (jMeter and Grinder) are available on both platforms.

Advice from the testers in Microsoft

I spoke to a few people who do testing for a living within Microsoft and here is some advice they had to offer

Do think about configuration parity of hardware. For instance you may need more RAM for Windows to run the application and a higher capacity network card for Linux. Consider if your goal is to actually establish the configuration parity or is it to find which OS scales under a given configuration?
If you do end up trusting the tool – define a sample case. This is a test case for which you roughly know the performance parameters on each platform. Run it under the testing tool for both platforms to make sure there are no significant differences in performance due to the tool itself. Of course defining a sample case may not be a trivial task – but use one that tests the most critical parameter for your application – memory, CPU or I/O for instance.
Consider running the software being tested on its own server – without running any additional testing software on the servers. Provision a remote server with the testing tool (choose one), and install agents for that tool on each server with the software being tested. Measure network latency and I/O latency and run the tests simultaneously from the remote machine to each server. This works if the software being tested requires only simple parametric commands to run or is entirely run from a UI. Doesn’t work very well if complex scripting is required to run on each server.
Do understand the effects of running tests through the entire system. There is at least one war story that I can vouch for, it entailed testing of a web retailers production systems under peak load conditions, using test scripts on client systems. Unfortunately the test team forgot to turn off the connection from the web systems to the warehouses that carried the inventory because someone missed that in the end to end testing plan! Some poor soul written into a test case got umpteen gizmos from the retailer. The system worked as advertised, which was not the intent of the test team (at least for the test)!

But enough about us, what approach do YOU take, dear readers? Please do let us know - we are looking for war stories, dispatches from the data centers, tales from the trenches and any other feedback on testing disparate platforms that you have to share!