Following on from my recent Linux web server benchmarks and Windows web server benchmarks, I noticed that in general IIS appeared to perform better than all Linux based web servers that I’d previously tested.
As my Linux results were completed in March last year, I’ve run some of the tests again with the most up to date versions of each web server to ensure that the best performance can be achieved.
So let’s find out how Linux and Windows based web servers compare against each other in a static workload speed test.
First I’ll discuss how the tests were set up and actually done before proceeding into the results.
The Benchmarking Software
Once again I made use of Weighttpd to perform the actual benchmark tests, as I’ve found that it works well and scales quite nicely with multiple threads.
I ran the benchmarking software in it’s own Linux based virtual machine. The problem with running weighttpd on the same server that is running the web server software, is that a significant amount of CPU resources end up being used by the test itself which are thereby taken away from the web server.
Weighttpd was run via the ab.c script, which essentially automates the testing process for me. Instead of running hundreds of weighttpd commands I use this script to step through different concurrency levels automatically.
weighttp -n 1000000 -c [0-1000 step:10 rounds:2] -t 8 -k "http://x.x.x.x:80/index.html"
Where -n is the number of requests, in this case we are sending 1,000,000 requests for every concurrency level of -c specified. Thanks to ab.c, we are able to step through from 0 simultaneous requests up to 1,000 in increments of 10 at a time. Each test is also performed 2 times as specified in the rounds, this way I am able to get a better average result as each concurrency level is run 2 times. In total every test at each core level therefore results in around 200,000,000 GET requests being sent to the web server at http://x.x.x.x:80/index.html.
I would normally run the tests more times to get a better average, however as IIS was responding so quickly I had to increase from my usual 100,000 requests at each concurrency level to 1,000,000 in order to get proper numbers, so I’d therefore consider the results a rough average than a more exact reproducible result.
The -t flag specifies the amount of threads to use, this was adjusted from 1, 2, 4 and 8 depending on the amount of CPU cores assigned to the web server for each test. The -k flag is also specified as we are making use of keep alive.
The Test Environment
In order to perform the tests I made use of four Windows and one CentOS Linux virtual machines that were running on top of VMware ESXi 5.5, so while an extremely small amount of additional performance could have been gained from running on bare metal, making use of virtual machines allowed me to easily modify the amount of CPU cores available when changing between the tests. I’m not trying to get the absolute best performance anyway, as long as each test is comparable to the others then it’s fine and I can adequately test what I am after.
The physical server itself that was running ESXi had two CPU sockets, each with an Intel Xeon L5639 CPU @ 2.13GHz (Turbo 2.7GHz), for a total of 12 CPU cores. The numbers provided in this post should only be compared here within this post rather than other benchmarks I’ve done previously, as everything here was run on the same hardware here.
All virtual machines were assigned 4GB of RAM and were using the same SSD’s for storage. The amount of CPU cores on these were adjusted to 1, 2, 4 and then 8 for each test. Only one virtual machine that was being tested was run at any one time. All virtual machines, whether Windows or Linux were therefore run on the exact same hypervisor and hardware, the only differences therefore are at the operating system and web server level.
Web Server Configuration
It is important to note that these web servers were only serving a static workload, which was an 8 byte index.html page containing the string “Testing.”. The goal of this test was to get an idea of how the different web servers performed with raw speed by serving out the same file. I would be interested in doing further testing in the future for dynamic content such as PHP, let me know if you’d be interested in seeing this.
Otherwise all web server settings were left completely default, no changes were made at all.
No changes were made to the Windows or Linux operating systems for these tests. All servers were fully up to date with all available updates installed. Windows Datacenter edition with a GUI was used for all Windows IIS tests.
Below is a list of the operating system and web server versions that were used during this test.
- Windows Server 2008 R2 – IIS 7.5
- Windows Server 2012 – IIS 8.0
- Windows Server 2012 R2 – IIS 8.5
- Windows Server 2016 – IIS 10.0
- CentOS 7.3 – Apache 2.4.6
- CentOS 7.3 – Nginx 1.10.2
- CentOS 7.3 – OpenLiteSpeed 1.4.24
- CentOS 7.3 – Lighttpd 1.4.44
Now that all of that has been explained, here are the results of my benchmark tests as graphs.
Total Time Taken
This final graph simply displays how long the tests took to complete in the format of hours:minutes:seconds. Keep in mind as mentioned previously each test will result in approximately 200,000,000 GET requests for the index.html page. Apache total time was not displayed, as the results skewed the graph too much as it was so slow. The single core result took over 21 hours to complete.
The results are what I expected based on the individual results of my previous Windows and Linux web server benchmarks, however I am still surprised when I see these results. I’m very interested to find out why the Windows IIS servers beat almost all the tested Linux web servers in all tests.
In all tests we can also see Lighttpd slowing down after the 600 concurrent request mark, while in the first three tests Nginx appears to increase at higher concurrency levels.
Only in the higher core levels do the Linux based web servers start to catch up, namely on the 8 core test where OpenLiteSpeed and Nginx are performing on par with IIS. Given the same hardware, the Windows platform with IIS appears to be faster at serving out this static workload with pure default operating system and web server settings. It may very well be worthwhile further testing Apache/Nginx on Windows to see how that compares to Linux.