Consistency and Standards – an IT Pro’s best bet in crisis - Port 25: The Open Source Community at Microsoft
< Back to Blogs
Consistency and Standards – an IT Pro’s best bet in crisis by admin on April 27, 2006 05:27PM

Seventeenth Century Philosopher and Author Voltaire wrote “I disapprove of what you say, but I will defend to the death your right to say it." This resonates very much with my own thoughts after going through the comments and feedback from my previous blog. I want to thank everyone for providing valuable feedback, regardless of whether I agree with it or not. The principle here is not to choose a side but to put a process in place that allows for an open and honest dialogue and I am exhilarated at the results of this endeavor.

Moving on to “Consistency and Standards” – the short theme for today’s blog. One of the questions put to me recently was to share something that I might have benefited from, during my past life in IT Operations. As I had mentioned previously, I have been involved with IT Operations since 1989 in some shape or form. From a student who worked at the Academic Computing Services in Syracuse University managing Macintosh and Sun Sparc clusters all the way to the past three years when I was managing 7x24 support of Class-A Production Services like AD, DNS, WINS etc. for MSN Operations, I can vouch for the fact that consistency in implementation standards have saved the day countless number of times. The point here is, no matter what platform, toolset, operating system or application you may choose, developing standards towards consistent implementation of these will always reap into rewards towards a lower “TCO” or Total Cost of Ownership in terms of supportability.

I remember a few years ago we were battling the spread of one of the malicious worms across the Internet. We were in the middle of taking inventory of what the configuration of our Production servers providing these mission-critical Class-A services, spread all across the world looked like. We all realized that by adhering to a common toolset, standard SKU’s for Hardware as well as for the OS versions helped us reduce the deployment cycle of a patch from what had seemed like days to a matter of hours. You may ask – “Hey – how does that matter” ? Well, imagine writing a different script for each type of configuration and multiply that by the hundreds of servers spread across the world in eight different datacenters. That’s quite a chore, especially when time is not on your side, you’re facing a crisis and you only have limited number of resources you can muster up for support. You’ll also need to track success and failure of applying the patch across datacenters and monitor where additional attention is needed.

So, what does that mean?  That complex environments require standards to work “for and with” IT Administrators. Admittedly,“one-off” or “out-of-standards” configs are very easy to do if we’re trying to please a group or mend fences with a specific customer. But in reality, we’re doing them (our customers) a dis-service but putting their environment in harm’s way and increasing their risks. Why - because supportability of their environment is ultimately our responsibility. So…the more colors we put on the fence, the more painters we will need and the longer it will take……

Comments RSS
  1. However, you leave off the weakness of absolute consistency. When you have a perfectly homogenous network, any malware that gets past your protection can propagate as fast as the network and the hardware will allow. You have no firebreaks, no protection whatsoever until a patch can be released. The cases where huge networks had to be manually updated due to the speed of propagation of Code Red/Nimda are examples of the dangers of homogenous networks. The dangers of a genetic "stick" instead of an actual family tree have been proven over and over in farming, animal husbandry, and human medicine, yet networking people seem to think that these lessons don't apply to their domain, but the evidence is overwhelming that they do.

    A heterogenous network is FAR better able to withstand attack than a homogenous one, regardless of which OS. A totally homogenous network, whether it be Windows, OS X, Linux, or Solaris is, in essence a massive single point of failure. Once you get into the network, you only have to repeat the same attack. With a properly designed heterogeneous network structure, no attack or malware can run rampant. A windows attack will fail against !Windows OS's, a Linux attack will fail against windows and so on. A heterogeneous network increases the amount of work any attacker has to do by a large amount.

    The mythical cost advantage of a homogeneous network comes out of two things: Training and Management. If you don't like paying for training, then there is a cost increase to a heterogeneous network. If paying for training isn't a problem, then you can get training on any platform for the same cost.

    Managing a heterogeneous network is more complex, but not inherently more expensive. There are tools that will manage multiple platforms just as well as the tools that will manage only one, and the cost difference is not significant.

    As well, the increased flexibility a heterogeneous network gives you in terms of more ways to solve a problem, more ways to serve your customers, more directions to take *more* than makes up for any initial cost differential.

    What I find is that when you have people griping about the cost of managing multiple platforms is that you have one platform that's well-managed and the others are left in the wind. Usually, Windows is well-managed, and the others are not, so, due to not even trying to manage them properly, they cost more. That's not due to having multiple platforms in any way, that's just due to being lazy.

    The idea that a homogeneous network is anything but bad design needs to be purged from common thought. It doesn't work anywhere, not even in networks.

    However, I can see where Microsoft likes the idea. The concept that "Windows is all you ever need" has made them a ton of money. It's also the reason that windows interop still consists of begging Microsoft for APIs, reverse engineering protocols, paying huge amounts of money in licensing fees, then doing 100% of the work yourself, because when it comes to working with other platforms, Microsoft is going to do little more than the APIs. (SFU has always been about migrating to Windows, and SFM is so old and craptacular that it should get killed. I mean, it only supports AFP 2.2, which is over a decade old. The last improvement was in Windows 2000, when they supported AFP/IP. If you're going to suck that bad, yank the code, it's an embarassment.)

    Of course, when you guys are pushing homogeneity, (and we KNOW you ain't talkin' 'bout Linux), it kind of kills your whole "This isn't just a tool to push windows" line. Homogeneity and Interoperability are mutually exclusive lines of thought. Which one are you about?

    posted at 07:15PM 04/27/2006
  2. This can really be boiled down to one thing...what is the best technology for the application.  I used to say "It's all about Linux" yet any more, it isn't all about Linux.  It is about getting the job done quickly.  It is about getting the job done right, the first time.  

    Here is one thing I have always asked myself...if migration to Linux is so easy, why does Microsoft have example after example of companies who migrated from Linux to Windows?  From a cost-of-product point of view, Microsoft licenses are, from my experience, more expensive.  However, when you take the money out of the mix, there has to be a technical reason for that transfer of technology.

    VERY FEW of my customers are all Windows shops.  Most run about 80 percent Windows and 20 percent Linux / Solaris.  Honestly, unless I am in a totally different world, I see more Windows migrations than I do Linux migrations.  Why is that?  Microsoft technology works, and it works well.  The fact is that Windows IS all you ever need.  Ask JetBlue...100% Windows 2000 / 2003 Server.  That is just one example, but I just don't see Linux growth being what it once was.

    What is the best enabling techology?  If you have to fight with interoperability, then what is the point.  In the world of doing more with less, who has the time or the money to mess around.  Nobody I know does.

    Just my two cents.

    posted at 08:00PM 04/27/2006
  3. dan said:

    Hey, haven't posted before.  Probably won't again.  Just one thing.  I'm a huge Linux and BSD user.  Not much of a windows fan these days, and not a fan of MS these days.  Or so I thought.  Till now I've been silently subscribed to Port 25's RSS.  It's been showing me what I've been seeing some other people have mentioned in other various articles, mainly they MS is huge, kind of like IBM huge.  They aren't all bad.  There's lots of divisions.  Some are better than others.  I'm not a fan of the windows division, and the lawyers and PR people, but there's some good stuff still coming out of your company.  Visual Studios has always been IMHO one of your better products, and I am a big supporter of the Free Visual Studios Express and the PR competition for it, even if not a user. I think it's good.  And now Port 25.  Stories from the inside from people more like me than many others at Redmond.  It's good I think.  So I'm quietly paying attention.  Which is my point.  Do not get disheartened by a never ending massive flow of hate messages.  Angry people are more vocal.  You just have to learn to filter and ignore.  Look at the slashdot moderation system.  It's pretty decent at handling the inevitable mass of crap.  But most importantly, do not give up.  I can't be the only passive silent viewer you have out there.  I don't post because I don't exactly care too much, I'm not driven by rage to post angry pointless comments.  But don't mistake me, I am listening.  And I'm seeing another side of your company.  And I like it. It gives me hope.  And I bet there are a lot of other silent viewers out there that you are getting a message out to.  People aren't too often likely to speak up and say 'Um, ok cool I guess', but they feel it.  I think this site is important stuff. So don't stop. :)

    posted at 08:16PM 04/27/2006
  4. What happens to Jet Blue when a fast - spreading Windows attack gets in to their system?

    It's dead or it's dark, until they can make patches. You can get by on a homogeneous network, but like a corn field that has only one strain, the first time you have something bad happen, you're screwed.

    As well, there are ways you can integrate non-windows platforms into a primarily windows network for little to no extra effort and no extra cost whatsoever. File servers don't have to be Windows to integrate pretty much perfectly. Printer management. if you factor your web applications, you can still have all the logic and brute force running on IIS/etc, but have the servers on the 'front lines' as it were be Apache running on Linux, OS X, Solaris, etc. If you want hard core reliability with fewer machines, and less HVAC cost than a Windows or Linux server farm, IBM's iSeries and zSeries systems are a good choice for quite a few tasks that don't require Windows.

    Heck, my company's code repositories for our actuarial modeling applications are on an Xserve. They only needed CVS, and since SSH on Mac OS X Server  is kerberized, they can access CVS over SSH in a Single - Signon environment. They don't ever know the difference, as thanks to Apple's Active Directory plugin, they just get access to the repository. Thanks to Apple implementing NTFS Semantics in their ACLs for  Mac OS X 10.4 Server, they get the same security model they're used to. If I don't tell them it's not running on Windows, they'd never know.

    They get the functionality they need, I can give them a secure connection to their repositories, and we get a firebreak in case of an upatched malware outbreak. It's win/win, and it's just as easily maintained as our Windows servers, which I do as well.

    The idea that homogeneity is a requirement for efficiency and optimal performance of network and staff is at best nonsensical, and at worst, blatantly stupid.

    posted at 09:08PM 04/27/2006
  5. einhverfr said:

    In general, I agree with your main point that general consistancy in process is important is important both for quality and efficiency reasons.  I didn't see you advocating a homogeneous network on this post though I can see why people might have assumed you were.  Instead, I see your post as advocating absolute consistancy in the areas of IT process and here I largely agree.

    The only disagreement I would make is that TCO is only a small part of the equation.  In my experience, open source deployments tend to stress return on investment far more than TCO, and thus although the TCO is often higher (due to consulting labor, etc), the solutions are often better matched to the customer.  I close reading of the third party studies (particularly that of the IDC) on your Get the Facts site shows that they are not inconsistant with my experience, though perhaps they miss out on the main reason why people choose open source solutions (but hey, you wouldn't want to advertise THAT, would you)...

    This brings me to my point-- in flexible deployments where an open framework is often more important than a specific set of well-defined functions, although consistancy and standards are needed, so is flexibility in implementation.  Flexibility and agility are often worth the additional costs that they incur.   THus in these cases, there is a balacing act that occurs between absolute consistancy on one hand and flexibility and agility on the other.  Pushing one at the expense of the other often means sacrificing real business advantage.

    Best Wishes,
    Chris Travers
    Metatron Technology Consulting

    posted at 01:07PM 04/28/2006
  6. remdotc said:

    I am not pro microsoft or pro linux, realy I'm pro "go home and not do work". Whilst you can have 100 different studies show how Microsoft or Linux is better in terms of TCO, or security or reliablity, I am a realist.  

    Standards are Microsofts biigest downfall in every product they produce. Microsoft is not RFC complient on most of its products, this includes the TCP/IP stack.  So in terms of pure standards, Microsoft fails in that department. In terms of ease of use, Yes, many parts of the GUI tools are great, in terms of inter-operablity , ugh comes to mind.  Microsofts buiness products, such as Great Plains (now Dynamics) has a product called Buiness Portal which is a good example of non standards compliance.  Any non microsoft browser attempting to view elements inside Business Portal do not return key elements, such as navigation menus, dispite the fact the navigation menu is written in DHTML and the client supports it. Only upon changing the user agent string for the brower returns the information. Another Example is LDAP versus Active Directory. While AD uses LDAP, windows 2000 is the only AD server that will recognise LDAP servers.  Another example is Microsofts next gerneration browser, that completely ignores the host file (yet another RFC violation).  Now if you want to talk about not panicing dispite the fact your network just went down because some laptop user plugged in and uploaded some network crashing virus because you or your staff do not filter your network traffic, then maybe we can talk about standards

    posted at 03:11PM 04/28/2006
  7. Standards!  Consistency!  Perhaps the best example I can think of, isn't actually a standard as such, but it illustrates "consistency" quite well.  In the late seventies/early eighties in New Zealand in one of the various Government departments there was someone in charge of an out-of-the-way office, which a fixed budget.  When he got to the end of the financial year he realized he hadn't spent his entire budget allocation for that year.  And so it would roll over to the next year, and he wouldn't get the full budget.

    So he asked himself what he should do.  His answer was to check through the financial records and find out the things he hadn't spent on for the last few years.  "Desks" was the answer.  So despite being adequately set up with desks, he went out and ordered a massive order of desks.  Demountable, so they could be stored, in a warehouse he had specially built.  The desk order was so massive it distorted the national furniture market for a few years following.

    How does this relate to your point?  Well, when the stated problem is interoperability and standards, and one has to solve that, and one of the known risks with Microsoft is that of Win32 monoculture, one runs the risk of distorting the market by pushing the wrong solution.  One should rather help solve the interoperability issue between Microsoft and other software, rather than push for a Microsoft-only solution that distorts the market and damages it by opening it up to malware risks.

    posted at 07:28AM 04/29/2006
  8. einhverfr said:

    Anything can be mindlessly done.  Consistency is no substitute for intelligence ;-)

    Anyway, I read Kishi's statement about the heterogeneous school network as one example where thy were also consistent in their implementation. So I am not sure where the consistency == homogeneity argument comes from.

    Also there are times when a monoculture (or at least something like it) is useful.  One example is where one might wish to build a very tightly integrated network (think of MIT's Athena project)-- yes, you can have different CPU architectures and even operating systems involved, but often times it is far less expensive to standardize on systems that can run the same software (because of the shared storage layer).  Such shared-everything clusters are very difficult to run effectively if they are internally heterogeneous.  Not that such is even practical on Windows...

    Best Wishes,
    Chris Travers
    Metatron Technology Consulting

    posted at 04:59PM 05/02/2006
  9. Having worked at MIT, I can tell you that the biggest headache for Athena to date has been Windows & forcing AD to play nice with their environment. OS X has been much less problematic for them.

    posted at 08:03PM 05/02/2006
  10. einhverfr said:

    jwelch wrote:

    "Having worked at MIT, I can tell you that the biggest headache for Athena to date has been Windows & forcing AD to play nice with their environment. OS X has been much less problematic for them."

    I don't doubt it.  But why would you want to?  I mean you can authenticate WIndows systems against MIT Kerberos.  Sure you lose some of the directory services possibilities, but you can minimize the headache here with careful planning.

    This is what I was getting at regarding Samba-- the best way to do this is to build a solid UNIX/Linux network and integrate the WIndows workstations to the extent you need them into the peripheral aspects of the network.  Running a parallel infrastructure adds no value to anyone except the makers of headache medicines....

    Best Wishes,
    Chris Travers
    Metatron Technology Consulting

    posted at 02:35AM 05/03/2006
Post a Comment
*
*