Draft paper: version 0.4

Microsoft versus Linux

I think it's a great accomplishment of the Linux community that Microsoft has recently started to publish white papers about this operating system. They're even so worried that they `bench-marked' NT Server against Linux. They published their results. Read this as: they were pleased with them or able to make them pleased to them.

I think this paper is very interesting. Not for its results, but for the way Microsoft presents information. As many of you know from bitter experience, Microsoft systems and software are not noted for their reliability. Still Microsoft sells. I believe that an important reason for this is, that Microsoft is very good in communicating to management. So good that the You can't be fired for buying Microsoft thought is firmly entrenched there.

How does Microsoft convince management? I think Microsoft masters four arts that the Open Source community doesn't practice. Using the above mentioned NT against Linux benchmark paper I've identified these arts as:

  1. The art of comparing: you should win.
  2. The art of ballooning: making yourself big.
  3. The art of statistics: stats will do anything.
  4. The art of deception: or should we say lying?

In the following four sections we will closely study those arts. My purpose is not so you can imitate it or become a master of these arts yourself. As we have "free" in the sense of "free speech", Open Source also has to maintain its open in the sense of "I will be open with you, I will be honest." This paper also isn't written, because I believe there is something wrong with the benchmarks. That case is firmly settled now: Currently, NT performs better, in almost any case, as Linux.

This paper is about how Microsoft uses that results to present its case. My purpose is to give you something to say to your management when they will point to Microsoft's white paper. As this is a draft paper, comments, suggestions and additions are eagerly sought.

The art of comparing

In case some manager doesn't have much time, it helps to distill the most important findings. So a manager can quickly scan the paper and still get a good understanding of its contents. It goes without saying that this distillation should be honest. However, Microsoft defines distillation as: publish your best benchmarks when compared to the worst benchmark of your competitor. We will highlight two examples of Microsoft distillation:

  1. NT server performs up to 680 percent better as a web server.
  2. NT server provides up 96 percent better file server price/performance.
Note that Microsoft skillfully started bullets after the "up to". The actual figures stand out so, the reservation does not, so anyone just skimping the article won't miss the point they wanted to make.

NT server performs up to 680% better as a web server?

When you compare two benchmarks, they should be comparable. What did Microsoft compare in its statement that NT server performs up to 680 percent better as a web server? This statement was based on an article published in PC Magazine. On the NT side we have:
  1. The latest Microsoft NT Server edition, including all the latest service packs. I infer this because PC Magazine used Microsoft Transaction Server and Microsoft Message Queue, needing at least Service Pack 4 with Option Pack 4.
  2. The latest Microsoft Internet Information Server (IIS) version.
  3. The application on IIS was a DLL. That means it was precompiled and loaded just once. It used the IIS optimized API, not CGI.

On the Linux side we have:

  1. PC Magazine used a Linux 2.0 kernel, in no way optimized for SMP architectures.
  2. PC Magazine used an older version of Apache (before 1.2.4). This version of Apache didn't use threads, but processes. The overhead of processes is much larger. This version wasn't written nor optimized for multi-processor architectures.
  3. The application on Apache seems to be a Perl script (loaded and recompiled every time) using CGI. There was no hint in PC Magazine's test that they used something like PHP or embedded Perl.

So take NT's best performing case and put it against Linux lowest performing case and you're there. When the comparison was a bit more fair, comparing web serving with CGI applications, PC Magazine wrote: "On our CGI tests, Stronghold (Apache) held its own against the competition."

Note, in this case only the CGI variable was changed. They still compared compared NT, according to Microsoft optimized for SMP architectures, against a Linux kernel, not optimized for SMP and a web server not written for SMP kernels. PC Week, in a first-of-its-kind open benchmark, compared Linux with NT on a single processor architecture. The difference was much smaller in this case.

NT server provides up to 96% better file server price/performance?

What did Microsoft compare in its statement that NT server provides up to 96 percent better file server price/performance? This statement is based on a recent test by Mindcraft. Interestingly, Mindcraft did not publish results using NT Workstation clients, but only for Windows 95 clients.

As previous tests have shown that, using NT Workstations, Linux does outperform NT. But if you have only Windows 95 clients, NT performs better. Mindcraft was payed by Microsoft to do this comparison. As Mindcraft's results were not verifiable in PC Week's test, we don't know how much better Windows 95 clients perform against NT Server. And if you have NT workstations in your network, this benchmark doesn't help you to decide.

In the same graph, Microsoft also used testsresults done by PC Week. When PC Week confirmed again that using NT Workstations Linux did outperform NT, Microsoft came up with some new tuning parameters so the NTFS filesystem would perform better. You might expect that the coming Linux 2.3 kernel will provide better performance too.

Another question is how Microsoft calculated the cost of $2807 for it's server. You get NT Server 4.0 with 25 client licenses for $1609. The remaining $1200 is probably more client licenses. That will by you some 40 licenses. So Microsoft compared $2807 with a limit of 65 clients to $0 with no client limit. What would you choose?

Comparing is an art, you've just seen that. And it's not just this article in which Microsoft deploys this art. It's a common theme. See for example there Meeting Enterprise Security Needs:Microsoft Windows NT and UNIX white paper. To which Unix do they compare??

The art of ballooning

Microsoft is really good in putting out papers and emails like "NT breaks new performance barrier", "NT this" and "NT that". Don't mention problems or your competitors. Well, that could be business practice, but let us examine some facts. Microsoft has the desktop. But is Microsoft that overwhelming on the Internet yet?
  1. A well-respected company that gathers Internet statistics is Netcraft. In their June 1999 Web Server Survey, they confirm that Apache has 61% of market share. And it's growing much faster as it competitors.
  2. PC Magazine favored NT and IIS as their editorial choice. But why are they running Apache on Solaris? And so does PC Week.
  3. PC Magazine favored NT. But it doesn't agree that you got more money for your bucks as Microsoft wants you to believe. See their just released "The latest version of Linux gives Windows NT a run for its money -- for free." article.
  4. Sometime ago Microsoft acquired HotMail, a ten million user system. We all know NT is enterprise ready, can scale, has the hottest TPC-X benchmarks, whatever. Why does Microsoft use FreeBSD?
  5. And yet another Microsoft owned company uses FreeBSD.
  6. Kirch is with you. See his Microsoft Windows NT Server 4.0 versus Unix

The art of statistics

Microsoft's excellent understanding of statistics, can be clearly seen in their claim that Windows NT Server 4.0 Provides better Price/Performance than Linux. For web servers even up to - we've seen the `up to' before - 623%. In this section I take a look at how Microsoft calculated the file server price/performance statistics. Microsoft calculates the price/performance ration by dividing the costs of the server hardware and software by its peak Megabyte per second throughput. If the hardware costs $10000, the server $1000, and you have a throughput of 200MB per second, the price/performance ratio is $55: it costs you $55 per MB. Microsoft makes the following claims:

  1. According to Mindcraft, Microsoft costs $93 per Mbs.
  2. According to PC Week, Microsoft costs $114 per Mbs.
  3. The costs for Linux are about $182.

When people learn that NT Server has a better price/performance ration as a free OS, people react really surprised. How can something expensive have a better price/performance than a free OS? Learn how you do this when we entangle these figures.

As discussed above, Microsoft has paid Mindcraft to do a NT versus Linux benchmark. This benchmark was very favorable for NT. Later on PC Week was not able to verify the enormous differences Mindcraft found.

Anyway, what should one do to compete against a free OS? To get a higher price/performance ratio, you have to make the costs of the software insignificant to the costs of hardware. Microsoft does this by:

  1. Choosing really expensive servers. Do you spend $20,851 on a file server? Or the even more ridiculous $35868 server?
  2. Only counting the most basic costs of its Operating System. Add email to a Linux server. Costs you nothing. Add Microsoft Exchange Server to an NT Server and you got a monster, oops, I meant something expensive.

PC week commented that they got Linux most favorable results on a single processor server with 256MB of RAM. If we assume a $5000 file server (probably some 27GB disk, 256MB, and a Pentium II 400MHz or better), we get a $47 price/performance ratio for NT and $46 for Linux. Assuming the new NT NTFS tuning parameters of course, so NT outperforms Linux on file serving. How realisticly these NTSF tuning parametesr are, is still under debate.

The art of deception

In its article Microsoft argues that performance is not all. There's more than performance. How about user satisfaction? Microsoft doesn't mention that, but makes numerous false and misleading claims in this section. I've picked sentences from this section and gave my comments. When a sentence appears under Linux column I start it with Linux, when it appears under the Windows NT Server 4.0 column, I start it with NT. As this is a draft paper, I expect this section to expand a lot.

"NT: Lack of extensive testing to guarantee compatibility across components and applications."

Does Microsoft tests all of the 4000 applications that run on Windows NT Server? Or the 8000 Windows NT compatible applications?

"NT: Lack of an enterprise clustering system for service and application availability."

With Beowulf clustering, using Extreme Linux, you can put together dozens of Linux machines. They have been used as rendering machines, supercomputers, and lots of other applications.

"NT: Support for high availability application clustering and TCP/IP-based load balancing."

If you don't know what TCP/IP-based load balancing has to do with load balancing, I don't know either. It's probably means about the same as that Windows-CE is a real-time OS, or using another Microword: a soft real-time OS.

"Linux: Lack of a journaling file system (JFS), which means file system may not recover after unplanned down time."

Microsoft makes available two files systems: FAT and NTFS. The FAT filesystem isn't a journaling filesystem, NTFS is. You can use either with NT.

Linux' standard file system isn't a JFS. But you can install a JFS. SGI has made available a journaling file system for Linux. It's Irix renowned XFS technology.

A few weeks ago I followed a discussion about `soft updates' versus journalling. Soft updates should do for a standard Unix file system what a journalling file system does in respect to file system health.

"Linux: Supports 4 GB of RAM by default (2 GB kernel and 2 GB User/Application)."

Linux supports normally supports 2GB. Anyone even touched a machine with 2GB of memory lately? And if 2GB is not enough, Linux runs on lots more hardware, so use hardware that supports more memory.

The 2GB limit is due more to the mainstream Intel architecture than to Linux.

Lately I've read that NT applications could not allocate more as some 500MB of memory. I couldn't find the reference however anymore. If someone remembers this, please let me know.

"Linux: Inherits the security flaws of UNIX (for example it's easy to gain root access via poorly written applications)."

With this point, Microsoft probably seems to refer to buffer overflows. As almost daily large security holes are detected in NT, you have to be daring to write this down. And Microsoft software is immune to this of course. A heavily publicised buffer overflow was recently detected by eEye.

"Linux: No centralized security, which means that users must manually synchronize user accounts across servers and clients."

Unbelievable that someone could write this down. Microsoft refers here to the fact that an NT Server can function as a primary domain controller (PDC). Windows 95 clients and NT Workstations can connect to this server. User accounts and login scripts are stored on this server. Windows 95 and NT Workstations can also have there own user account database.

Unix machines have known the same principles for years, in fact far longer that Microsoft had something similar. You can setup a NIS/NIS+ server to get the same amount of functionality. The account database is stored on a central server and workstations are authenticated through this service.

Donovan Rebbechi wrote me that NIS isn't exactly known for its security. This was certainly true in the past. I'm no NIS expert, so I'm not sure if this holds for the current versions. Also NIS+ is far more secure than NIS.

Also you can use Kerberos authentication. Kerberos is one of the most secure authentication systems available. Kerberos support is only expected in Windows 2000, with Linux you can have it now.

"Linux: Hundreds of available applications."

I suspect this should be thousands. And every month large vendors promise to bring out their products for Linux too. Corel has made available WordPerfect for Linux. And it has promised to port more applications. All the main database vendors have made Linux versions. The more well known are DB/2, Oracle, Informix, Sybase and Interbase. And these vendors support it too.

"Linux: Not optimized for high-end servers."

Everyone chooses NT for there high-end servers, right? Let's leave high-end computing to the high-end vendors.

"NT: Dedicated support network"

Microsoft compares this to Linux' assumedly only peer-to-peer support. This is simply not true. You can buy commercial support for Linux from a fast growing list of vendors.

Besides that, what's wrong with peer-to-peer support? I've worked a lot with Microsoft support and its a mixed blessing. Sometimes they can only suggest to reinstall everything or try the latest service pack, bug fix, hot fix, etc. And sometimes their support really helps. Just the same experiences I have with peer-to-peer support. Only I need a lot less support for my Unix systems than for my NT and Windows 9x systems.

If you're really stuck, you have the source and go out and fix it yourself. This is not an option with closed source. You are depend on someone else.

"NT: Consistent GUI-based tools."

People say that GUI-based tools make tasks simpler. Sometimes this is the case. But for system administration, the difficulty is not GUI-based tools versus editing text tools, but it's far more conceptual.

You have to know about networks, IP addresses, hardware, etc. to be able to run a system. Point and click is only marginal. And when you have dozens of GUI programs each with dozens of screens and settings, finding information and where something can be done is just as hard as knowing which text file to edit.

For larger systems, text files have two advantages that closed GUI systems don't have:

  1. You can put comments in it, clarifying things for someone else. Why have you done things.
  2. Most of the time it's easy to parse text files, so you can write scripts reading and maintaining them. Text files are the most open database available.

"Linux: Administrators are required to re-link and reload kernel to add features to operating system. "

Not always true. And the Linux community could well make a kernel that just had everything in it, just like the Microsoft bloat.

With kernel compilation you can achieve two things:

  1. A lower footprint, meaning you can run on machines with less memory, and give more memory to applications.
  2. Leaving out things you don't need, so enhancing performance, stability and security.

"Linux: Open questions about internationalization, access by people with disabilities, and Year 2000 testing and compliance."

Text files and consoles are quite good to use for people with visual handicaps. GUI tools are much harder.

Do you mean we have choice to choose web servers, databases and applications, and that they can integrate? Great, that's just what we want.

"Linux: Provides source code to allow developers to deviate from standard distribution."

Exactly.

"NT: Broad language support including Java."

Which language is not available for Linux, that is available for NT? Best of all, most are free!

"Linux: No long term deployment road map - features are added based on OSS developer willingness to implement them."

Ever got Microsoft as far as adding a feature in NT that you really wanted?

"Deployment Road Map - Allows customers to plan future deployments."

As far as I understood planning, it had to do something with time. Both in features and in arrival time, depending on Microsoft's road map will make you really digitally nervous.

Conclusion

Don't think I'm a big Linux fan. I too believe that Linux is not up to NT right now. PC Week's tests confirm this. I think this test is very dependable, probing the technical arguments why NT performs better than Linux, even on a single processor architecture.

I'm a believer in FreeBSD on the server side and BeOS on the desktop. But I also believe in choice.

Choice, to be free to develop on a platform one likes. Not to be dictated to use Microsoft Visual Basic, Microsoft Transaction Server, Microsoft SQL Server, Microsoft Advanced Server, Microsoft Message Queue Server, or Microsoft whatever.

Choice, so information will remain free. When information only can be viewed with Microsoft Internet Explorer, when information is only served by Microsoft Internet Information Server, there is no choice. I use Netscape Communicator. It has a tendency to crash when I hit www.microsoft.com. Could be bad luck, but it remains remarkable.

Choice, so we can remain free. When Morris' worm hit the Internet in 1988, it's due to Unix diversity that not everyone was hit.

Choice, not one sizes fits all. Choice, even to choose not Linux.

References

A number of documents played a role in the nt versus linux discussion. I've listed the most important ones below.

Berend de Boer
1999-June-30

Site Meter

Changes in version 0.4:

Changes in version 0.3:

Changes in version 0.2: