Linux VS Windows Web Server Benchmarks

Following on from my recent Linux web server benchmarks and Windows web server benchmarks, I noticed that in general IIS appeared to perform better than all Linux based web servers that I’d previously tested.

As my Linux results were completed in March last year, I’ve run some of the tests again with the most up to date versions of each web server to ensure that the best performance can be achieved.

So let’s find out how Linux and Windows based web servers compare against each other in a static workload speed test.

First I’ll discuss how the tests were set up and actually done before proceeding into the results.

The Benchmarking Software

Once again I made use of Weighttpd to perform the actual benchmark tests, as I’ve found that it works well and scales quite nicely with multiple threads.

I ran the benchmarking software in it’s own Linux based virtual machine. The problem with running weighttpd on the same server that is running the web server software, is that a significant amount of CPU resources end up being used by the test itself which are thereby taken away from the web server.

Weighttpd was run via the ab.c script, which essentially automates the testing process for me. Instead of running hundreds of weighttpd commands I use this script to step through different concurrency levels automatically.

weighttp -n 1000000 -c [0-1000 step:10 rounds:2] -t 8 -k "http://x.x.x.x:80/index.html"

Where -n is the number of requests, in this case we are sending 1,000,000 requests for every concurrency level of -c specified. Thanks to ab.c, we are able to step through from 0 simultaneous requests up to 1,000 in increments of 10 at a time. Each test is also performed 2 times as specified in the rounds, this way I am able to get a better average result as each concurrency level is run 2 times. In total every test at each core level therefore results in around 200,000,000 GET requests being sent to the web server at http://x.x.x.x:80/index.html.

I would normally run the tests more times to get a better average, however as IIS was responding so quickly I had to increase from my usual 100,000 requests at each concurrency level to 1,000,000 in order to get proper numbers, so I’d therefore consider the results a rough average than a more exact reproducible result.

The -t flag specifies the amount of threads to use, this was adjusted from 1, 2, 4 and 8 depending on the amount of CPU cores assigned to the web server for each test. The -k flag is also specified as we are making use of keep alive.

The Test Environment

In order to perform the tests I made use of four Windows and one CentOS Linux virtual machines that were running on top of VMware ESXi 5.5, so while an extremely small amount of additional performance could have been gained from running on bare metal, making use of virtual machines allowed me to easily modify the amount of CPU cores available when changing between the tests. I’m not trying to get the absolute best performance anyway, as long as each test is comparable to the others then it’s fine and I can adequately test what I am after.

The physical server itself that was running ESXi had two CPU sockets, each with an Intel Xeon L5639 CPU @ 2.13GHz (Turbo 2.7GHz), for a total of 12 CPU cores. The numbers provided in this post should only be compared here within this post rather than other benchmarks I’ve done previously, as everything here was run on the same hardware here.

All virtual machines were assigned 4GB of RAM and were using the same SSD’s for storage. The amount of CPU cores on these were adjusted to 1, 2, 4 and then 8 for each test. Only one virtual machine that was being tested was run at any one time. All virtual machines, whether Windows or Linux were therefore run on the exact same hypervisor and hardware, the only differences therefore are at the operating system and web server level.

Web Server Configuration

It is important to note that these web servers were only serving a static workload, which was an 8 byte index.html page containing the string “Testing.”. The goal of this test was to get an idea of how the different web servers performed with raw speed by serving out the same file. I would be interested in doing further testing in the future for dynamic content such as PHP, let me know if you’d be interested in seeing this.

Otherwise all web server settings were left completely default, no changes were made at all.

Server Configuration

No changes were made to the Windows or Linux operating systems for these tests. All servers were fully up to date with all available updates installed. Windows Datacenter edition with a GUI was used for all Windows IIS tests.

Below is a list of the operating system and web server versions that were used during this test.

  • Windows Server 2008 R2 – IIS 7.5
  • Windows Server 2012 – IIS 8.0
  • Windows Server 2012 R2 – IIS 8.5
  • Windows Server 2016 – IIS 10.0
  • CentOS 7.3 – Apache 2.4.6
  • CentOS 7.3 – Nginx 1.10.2
  • CentOS 7.3 – OpenLiteSpeed 1.4.24
  • CentOS 7.3 – Lighttpd 1.4.44

Benchmark Results

Now that all of that has been explained, here are the results of my benchmark tests as graphs.

Windows IIS vs Linux Web Servers Benchmark - 1 CPU Core

1 CPU Core – Click Image To Expand

Windows IIS vs Linux Web Servers Benchmark - 2 CPU Cores

2 CPU Cores – Click Image To Expand

Windows IIS vs Linux Web Servers Benchmark - 4 CPU Cores

4 CPU Cores – Click Image To Expand

Windows IIS vs Linux Web Servers Benchmark - 8 CPU Cores

8 CPU Cores – Click Image To Expand

Total Time Taken

This final graph simply displays how long the tests took to complete in the format of hours:minutes:seconds. Keep in mind as mentioned previously each test will result in approximately 200,000,000 GET requests for the index.html page. Apache total time was not displayed, as the results skewed the graph too much as it was so slow. The single core result took over 21 hours to complete.

Windows IIS vs Linux Web Servers Benchmark - Total Elapsed Time

Total Time Elapsed – Click Image To Expand

Conclusion

The results are what I expected based on the individual results of my previous Windows and Linux web server benchmarks, however I am still surprised when I see these results. I’m very interested to find out why the Windows IIS servers beat almost all the tested Linux web servers in all tests.

In all tests we can also see Lighttpd slowing down after the 600 concurrent request mark, while in the first three tests Nginx appears to increase at higher concurrency levels.

Only in the higher core levels do the Linux based web servers start to catch up, namely on the 8 core test where OpenLiteSpeed and Nginx are performing on par with IIS. Given the same hardware, the Windows platform with IIS appears to be faster at serving out this static workload with pure default operating system and web server settings. It may very well be worthwhile further testing Apache/Nginx on Windows to see how that compares to Linux.

Leave a comment ?

20 Comments.

  1. Thank you for those tests. I curious how Debian will take care of it, as I saw other benchmarks where it’s 50-80% quicker than Centos.

    • Interesting, I might do some further testing with Debian and see how we go. Back in my original tests in March 2016 I did test with Debian and got very similar results with at least Nginx so not sure how much it will have changed.

  2. Please do PHP!! :)

  3. Extremely interesting. Thanks so much for your work on this. Would love to see how Debian stacks up.

  4. One simple and possibly interesting variation is to serve different sizes of static responses e.g. 64KiB, 256KiB. That keeps the focus on the performance of the webserver itself, but exercises a slightly different part of it (since the response in the current tests is a fraction of a packet).

  5. Yes, do php/wordpress! Very important for real word use! More meaningful for the general population.

  6. I’m pretty sure your tests are wrong related to CentOS.

    Have you configured CentOS for best performance? Please check https://adrhc.go.ro/wordpress/boost-or-at-least-unlock-the-website-performance/ for some settings, most important limits. Check elsewhere for networking related performance configuration. Linux is flexible – use this otherwise is as if testing best running man performance with one of his legs broken.

    I also tested Nginx on Ubuntu Desktop with limits configured and a 4 cores 1.6 GHz CPU vs IIS (don’t remember the version) with a 4 cores 2.4 GHz CPU and Nginx was way faster. I used ab from Ubuntu for both tests.

    On a N3150 CPU (1.6 GHz with 2.08 GHz Burst Frequency) for a 1.6K file I get 4600 req/second with ab run from the tested Ubuntu computer.

    How many req/second have you got on your tests? (should be much more considering that Xeon L5639 CPU)

    • No, I try to leave things fairly vanilla these days because if I change things for “optimal performance” in one specific workload, someone will copy those settings for something else where it may not be as good.

      As for Apache Bench (AB), this is what I read some time ago and why I don’t test with that: http://gwan.com/en_apachebench_httperf.html

      The requests per second are displayed in the graphs, 4 CPU cores in my Nginx tests shows it getting a maximum of around 150,000 requests per second, however just remember the file size I am testing with is static and much smaller (8 byte page).

  7. Windows Server 2016 is currently the most robust, both in security and performance.
    The problem is that people are biased with Windows Server and do not even seek to know how it works and their security levels.

    As the site gets more robust and full of content, it tends to not respond well and this can happen on Linux and Windows.

    But in general use and security levels I would go from Windows Server 2016, for those who want to venture out, will have to study 5 times more content than in Linux.

  8. You did not modify any of the settings on webservers… so basically in your test you are doing analysis of default settings and limits of webservers… Great. You should change the title of blogpost, so it’s more obvious that this is the case.

    • Correct, I’ve found there’s no point trying to tweak them as everyone will just argue about the best settings to use, and in many production workloads default settings are often used. This is still useful data in a large number of use cases.

  9. 200K response per second fair enough for such systems and it is fair enough for “most of the applications” in the world.

  10. I’d like to see this with lwan, as it is actually optimized for speed: https://lwan.ws/

    I’m a linux developer, but I’m not actually surprised by these results; when used right (see IOCP, https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores), windows is *scary* good at async io

  11. Looking at the Apache/nginx line are more or less on the same level all the time (until you see that the VM with the weighttp starts to affect the test when using more cores), which seems you are running the default settings, which ain’t made for really high number of connections.
    I would just say that this test seems to just test the default settings, not what the different web daemons are capable of doing.

    When you look at the line generated for the IIS, it looks like an earth quake, which tends to show how badly it handles threads.

    Configure the apache/nginx for high load and put on more load (please run the weighttp on another machine than the web daemons, not just another VM on the same machine) and you will still see a quite even line for the Linux and the IIS starts to make even worse with it’s attempt to look like the worst ever earth quake ever.

  12. Good article, but colors are very bad – there’re too many blue!

  13. I wonder why IIS 8 is faster than IIS 10… Does anyone know how to configure Azure Web Apps to use IIS 8 instead of 10?

  14. Your benchmarks are very outdated. I just benchmarked Lighttpd & Nginx, which shows Lighttpd consistently about 19% faster on 4 cores across the board. I’ve tried to improve nginx performance, but so far nothing.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>