Open Benchmark Invitation

From Bruce Weiner, Mindcraft, Inc.

May 18, 1999

(See the change list for updates to the original proposal made May 4, 1999. The purpose of these changes is to make the Open Benchmark as fair as possible.)

Rationale and Invitation

Linus Torvalds has been quoted in a few stories (ABCnews.com ,Linux Today, and Salon) as expressing concern about a new benchmark that Mindcraft was running. Mindcraft's honesty and name have been impugned in these stories and others as well as in great deal of newsgroup postings and email we have received. To get the other side of the story, please look at our rebuttals to the ABCnews.com, Linux Today, and Salon articles. 


We've Got A Time and Place

John Taschek  from PC Week, a Ziff-Davis publication, has offered the Ziff-Davis Labs in Foster City, California for the Open Benchmark. It looks like the fastest that he can make the lab available consistent with existing commitments of some of the key Linux, Samba, and Apache players is the week of June 14. So we have a date!


We will be glad to conduct this Open Benchmark at any mutually agreeable test site. Bob Young, President of Red Hat Software, said in a Salon article  published yesterday that if an organization such as Ziff-Davis were brought into the benchmark "...then absolutely we would be thrilled to participate." Well, PC Week has offered their lab as the site for the Open Benchmark. Mindcraft believes that the PC Week lab would be an ideal location and we hope that Red Hat and the Linux, Samba, and Apache experts will agree to using it.

Some people have asked why we want to use the same Dell server we used for our second test. There are two reasons why:

  1. We want a witnessed, scientific test showing if the results we got were accurate. This is the kind of test that Jeremy Allison, a vocal Linux and Samba proponent, has asked for. Changing the server would be unscientific.
  2. We want to verify the scalability and enterprise readiness of Linux. We believe that the best enterprise-class server to test this on would be one from a major computer manufacturer that supports Linux.  The Dell server meets these criteria (look at the joint Red Hat and Dell press release ).

We have also seen that some people are concerned that we picked a multiprocessor configuration knowing that Linux would not perform as well as Windows NT Server. Red Hat's press release announcing its Linux 6.0 Server Operating System  should allay those concerns. For those who would still like to have a comparison on a uniprocessor configuration, we have included one in the Open Benchmark.

Mindcraft believes that the best way to continue to show our name as a credible source of information is to have an Open Benchmark. Therefore, we welcome the opportunity to perform a benchmark of Linux and Windows NT Server that is open to the best experts in the Linux community. Mindcraft will participate in this benchmark at its own expense.

Mindcraft has withheld the publication of our second Linux and Windows NT Server benchmark results (the test for which Linus and others provided some suggestions for tuning) pending the response to this Open Benchmark invitation.

We call on Linus Torvalds to invite anyone he chooses to tune Linux, Samba, and Apache. We also invite Red Hat to send anyone they choose to participate in the benchmarking as a Linux Expert. In addition, we invite Microsoft to tune Windows NT Server. The Linux Experts, Microsoft, PC Week, and Mindcraft will witness all tests.

Purposes

  1. To see if Mindcraft's second benchmark results are biased and not representative of Linux's true performance.
  2. To do a fair comparison of Linux and Windows NT Server 4.0 with Linux tuned by Linux Experts and with all testing witnessed by them.

Test Environment

The following test environment will duplicate as closely as possible in the PC Week lab the test environment Mindcraft used for its second test of Linux and Windows NT:

  1. Mindcraft will arrange to have available the same system that was used for the second Linux/Windows NT Server test.
  2. The tests will use 4 x 100Base-TX networks for all configurations.
  3. Samba and NT file sharing tests will be done using the NetBench Enterprise mix. The NetBench tests will use either 144 clients or 72 clients, depending upon client availability in the PC Week lab. The same client setup will be used for both Linux/Samba and Windows NT.
  4. Apache and IIS Web server tests will be done using the WebBench zd_static_v20.tst, modified only to account for the number of clients used and to adjust the number of threads to match the number Mindcraft used in its first and second tests, which was two threads on each of the 144 clients we used. Thus, for 144 clients the benchmark will use two threads per WebBench client and for 72 clients the benchmark will use four threads per WebBench client . The same client setup used for NetBench testing will be used for both Linux and Windows NT WebBench testing.

Test Procedure

The testing will be divided into three phases:

Phase Purpose
1 Reproduce the results of Mindcraft's second test
2 Linux experts use anything available at the time of Mindcraft's second test. This will show how much better Mindcraft could have done at the time
3 Get the best performance possible using today's software
 

General Procedures

Tunes, Patches and Debugging

The tunes allowed include:

Patches are defined as any changes to the operating system, Web server, or file server that alter the programming logic. Tunes are not considered patches. Patches can be in source code form or in binary form, such as Service Packs.

Debugging, tracing system calls, and other methods to help the participants understand how their products are behaving is allowed. Participants may apply what they learn to tuning and patching their products consistent with the phase of the Open Benchmark being conducted at the time.

Phase 1: Reproduce Mindcraft's Results

This phase addresses the concerns that Linus Torvalds, Alan Cox, Jeremy Allison and others have expressed about not being able to be in the lab when Mindcraft conducted its second test. We'll do it again with them present. Alan Cox has expressed concern that the Linux experts cannot use software that was not available at the time Mindcraft conducted its second test. They will be able to use the very latest software available during Phase 3.

The Linux experts will conduct this phase of the Open Benchmark using the following procedures:

  1. Mindcraft will disclose to the Linux experts both the Linux and Windows NT Server test results from its second test. These results are not for publication until the results of the Open Benchmark are published consistent with the Publishing the Outcome  section.
  2. Mindcraft will disclose in detail the hardware configuration, Linux tunes, configuration files, patches, and software versions it used for the second test. Mindcraft used Linux 2.2.6, Apache 1.3.6, Samba 2.0.3, and a TCP patch provided by David Miller.
  3. The Linux experts will format all disks and verify that the hardware is configured as Mindcraft disclosed.
  4. The Linux experts will install, configure and tune all software as Mindcraft disclosed.
  5. The Linux experts will run the following tests:

Phase 1 Linux Tests

Test ID #of CPUs RAM Test
1 1 256 MB NetBench
2 1 256 MB WebBench
3 4 1 GB NetBench
4 4 1 GB WebBench
 

Mindcraft will conduct this phase of the Open Benchmark using the following procedures:

  1. Mindcraft will disclose the hardware configuration, Windows NT Server tunes, configuration files, patches, and software versions it used for the second test. Mindcraft used Windows NT Server 4.0 with Service Pack 4 applied, IIS 4, and the Windows NT Option Pack.
  2. Mindcraft will format all disks and verify that the hardware is configured as Mindcraft disclosed.
  3. Mindcraft will install, configure and tune all software as Mindcraft disclosed.
  4. Mindcraft will run the following tests:

Phase 1 Windows NT ServerTests

Test ID #of CPUs RAM Test
5 1 256 MB NetBench
6 1 256 MB WebBench
7 4 1 GB NetBench
8 4 1 GB WebBench

Phase 2: Linux Experts Use and Tune The Best Software Available at the Time Mindcraft Did Its Second Test

The purpose of Phase 2 is to see if the Linux experts could have achieved higher Linux performance if they had been present when Mindcraft conducted its second test. It directly addresses concerns on lab accessibility raised by Linus Torvalds, Jeremy Allison, and Alan Cox.

In this phase the Linux experts can use any versions of the operating system, Apache, or Samba, including patches to any of them, that were available on generally accessible Web or ftp sites, that were available for sale in stores, or that were available to Mindcraft at the time it started its second test, April 20, 1999, The Linux experts may make any tunes, patches, and configuration changes they want; David Miller's TCP patch is allowed to be used.

The Linux experts will conduct this phase of the Open Benchmark using the following procedures:

  1. The Linux experts will install, configure and tune all software as they choose.
  2. The Linux experts will run the following tests:

Phase 2 Linux Tests

Test ID #of CPUs RAM Test
9 1 256 MB NetBench
10 1 256 MB WebBench
11 4 1 GB NetBench
12 4 1 GB WebBench

There are no Mindcraft tests to conduct in this phase.

Phase 3: Get the Best Performance Possible Using Today's Software

In this phase the Linux experts and Mindcraft can use any versions of the operating system, Web server, or file server, including patches to any of them, that are generally available on the Web, at ftp sites, in stores, or from the product vendor at the time the Open Benchmark is conducted. The Linux experts and Mindcraft may make any tunes, patches, and configuration changes they want, consistent with the general availability constraint.

The Linux experts will conduct this phase of the Open Benchmark using the following procedures:

  1. The Linux experts will install, configure and tune all software as they choose.
  2. The Linux experts will run the following tests:

Phase 3 Linux Tests

Test ID #of CPUs RAM Test
13 1 256 MB NetBench
14 1 256 MB WebBench
15 4 1 GB NetBench
16 4 1 GB WebBench
 

Mindcraft will conduct this phase of the Open Benchmark using the following procedures:

  1. Mindcraft will install, configure and tune all software as it chooses.
  2. Mindcraft will run the following tests:

Phase 3 Windows NT ServerTests

Test ID #of CPUs RAM Test
17 1 256 MB NetBench
18 1 256 MB WebBench
19 4 1 GB NetBench
20 4 1 GB WebBench

Publishing the Outcome

Mindcraft, any of the participating Linux Experts, Microsoft, and PC Week will receive the raw test results and will have unrestricted use of the test results.

  1. Mindcraft and any of the Linux Experts that want to participate will issue a joint press release describing the test results. The tone of the press release will include the test results, will be factual in tone, and will be positive about the opportunity to have an Open Benchmark with the Linux experts involved. There will be quotations from Linus Torvalds or his designee, Red Hat (if they want to participate), and Bruce Weiner (Mindcraft).
  2. No test results or press releases will be published until PC week has had the opportunity to publish a story about the Open Benchmark. If PC Week chooses not to publish a story within two weeks of the conclusion of the Open Benchmark. All participating parties are free to publish the results consistent with this "Publishing the Outcome" section.
  3. Mindcraft will issue a report on its Web site similar in structure to the one for its first report.
  4. Red Hat and any of the Linux experts may generate their own reports regarding the test.

Change List

Change List - May 7, 1999

We made the following changes to the original invitation made on May 4, 1999. These changes were made to reflect PC Week's offer to use their lab for the Open Benchmark and to clarify some items on which we have received feedback. The changes are summarized by section.

Rationale and Invitation

  1. Bob Young's offer to test at Ziff-Davis has been added.
  2. PC Week's offer has been added.
  3. Mindcraft's acceptance of the PC Week lab has been added.
  4. PC Week's access to the benchmark results has been added.

Test Environment

  1. The client system specification in item #3 was moved to the Test Procedures section. In addition, both Windows 9x and Windows NT clients will be used.
  2. The number of clients to be used is more clearly specified in item #4 and structured to be consistent with the number used in Mindcraft's second test and with the number of clients available in the PC Week lab.
  3. The number of threads per WebBench client is more exactly specified in item #5. It now is specified to duplicate the number of total threads making requests on the server and to duplicate the number Mindcraft used in its first and second tests.

Test Procedures

  1. There are extensive structural modifications to clarify the purpose of each test, to clarify unintended restrictions on Samba and Apache, and to address expressed by Jeremy Allison and others. The changes are too many to enumerate here. In particular, the client OS to be used has been expanded to include Windows NT and the tables of tests have been expanded and reorganized to reflect all changes.

Publishing the Outcome

  1. PC Week was added to the list of receipients of the raw test results.
  2. A publication embargo has been added to allow PC Week an exclusive.

Copyright © 1997-99. Mindcraft, Inc.