![\"aliens\"](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/aliens.jpg\")
<\/p>\nIt seems like we\u2019re entering the YouTube era of benchmarking.<\/p>\n
<\/p>\n
You know how you can go on YouTube and find someone who knows almost nothing about the subject they are posting a video about? Well, benchmarking these days is kind of like that\u2026<\/p>\n
<\/p>\n
Just a quick word about the author. I worked as a senior engineer with Mercury Interactive. Mercury\u2019s LoadRunner tool was\/is the industry standard for performance testing\/benchmarking. I\u2019ve spent 17 years trying to make people smarter about how they test and tune their systems. I also worked extensively with the ActiveTest group within Mercury, and I\u2019m going to use some of the methods and procedures gleaned over thousands of customers and millions of dollars of R&D to develop the ActiveTest products.<\/p>\n
I have to say, I am dismayed at some of the nonsense that I see people posting.<\/p>\n
There really isn\u2019t a polite way to say it. Some of these folks don\u2019t even know the basics, so I thought it might make sense to write this post that explains Benchmarking 101. The goal of the post is to hopefully instill the rigor that we\u2019ve seemingly lost and get back to doing intelligent investigation of data and system performance.<\/p>\n
I will admit that I am biased towards LoadRunner since it is the tool that I know. Any number of tools can generate load in competent hands. For the purposes of this article, I will refer to four components of tooling using ideas from LoadRunner, but you should expand them to whatever tool you use. The four components are:<\/p>\n
I\u2019m just going to start with a simple statement. If your test tool is something that you run from a command line, your test results are pretty much worthless. If you run a simple google search, you can find dozens of options that help you produce more reliable results. Tools like Siege, weighttp, etc. tend to be used by very unsophisticated testers because they are free, but generally, more so because they are simple.<\/p>\n
I\u2019m sorry, but doing an accurate, meaningful benchmarking exercise isn\u2019t a single command line. If you are making any claims beyond some very basic pass\/fail stuff, this is useless. Let\u2019s face it, most of the benchmarks we see published are extremely grandiose in their claims. 13X performance? What does that mean?<\/p>\n Nothing. Nothing at all.<\/p>\n Before you begin, you need to have something you want to test, and a scientifically accurate way to measure it. 13X is a comparative statement. 13X what? How was X measured? Did you measure 13X using the same methodology? Was your test environment under control?<\/p>\n Looking back at the four components, let\u2019s investigate good testing practices.<\/p>\n It all starts with a script. What are you trying to emulate? If you script people browsing catalog pages, does that mean that the checkout process is 13X faster too? How data dependent is your script- are you passing multiple logins and passwords? You aren\u2019t if you run a command line\/online tool\u2026<\/p>\n Your controller environment is where you simulate the scenario behavior. Do all of my users arrive at one time (like a holiday sale?) or do they ramp up to help understand where my breaking points are? Are they all doing the same thing, or different things?<\/p>\n How you pair your script and scenario DEFINES what you are testing. This needs to match your goal for the test.<\/em><\/strong><\/p>\n I\u2019ll say that again. Those two components define the condition you are testing. If you see someone publish a benchmark without defining the test conditions or extrapolating it to things not defined in those components, the odds are pretty good that they don\u2019t know anything at all about benchmarking.<\/p>\n And, usually, they prove that they don\u2019t know by the next step\u2026<\/p>\n The load injector has finite capacity. If you see benchmark data that suggests that a load injector seems infinitely scalable, you should probably ignore the results they publish. A simple litmus test is that it takes as much or more hardware to generate load as it does to respond to it. The injector needs to receive all of the data sent by the web server and make sure that the request completes successfully…. And thus, should be on a separate platform from the web server! Good practice is to benchmark your injectors to know exactly how much load they are capable of generating without producing errors.<\/p>\n Lastly, analysis should be automated. In my experience, it is pretty easy to edit away data that suggests problems, so you need to remove that possibility if you are trying to provide a valid benchmark. All of the data should be used, and the data should not contain any error status codes. Vendors all want their products to shine, and that can lead to shenanigans.<\/p>\n So, what are good practices?<\/p>\n Start with a proof statement. Develop a test that exercises only that proof. Validate your test environment. Use automation to collect data and prepare results transparently. Explain anything that falls outside the model.<\/p>\n What is the anatomy of a poorly designed benchmark?<\/p>\n If you use a command line tool, or test without control over the scenario, or test without visibility into a scenario, or drive infinite load with your VM, or use Excel to report…you are practicing bad benchmarking. If you have data that you cannot explain, making something up.<\/p>\n Web performance data is a discipline ruled by mathematics.<\/p>\n There is no politically correct way to say this other than there are a lot of people doing garbage work these days. Aside from the bad testing practices, if you do not understand the data you collect, it is really difficult to explain what happened. It is even worse, if you just start fabricating conclusions. Many of these people are viewed as experts, but they lack a basic understanding of the fundamentals of web performance. It\u2019s time to expose.<\/p>\n Let\u2019s start with the basics. (Understanding relationships)<\/p>\n If I told you that I was going to drive from point A to point B as fast as I could, you would easily be able to calculate how long the trip should take through simple math (distance\/speed). A simple mathematical relationship defines time, distance, and speed as a functional relationship. Think of this as an analogy for baseline performance of a single web user.<\/p>\n However, we live in the real world, and we have to share the road. If the road has more lanes, it can handle more cars. If the cars are well coordinated, traffic flows well and our round trip time stays about the same. Think of this as an analogy for a healthy system.<\/p>\n But, what happens when you have more cars than road capacity?<\/p>\n Traffic causes congestion, or a slowing of the round trip time. Even though the road hasn\u2019t changed, its ability to handle traffic deteriorates. Any given road can handle some load of cars. Think of this as an analogy for hits\/requests per second. These are measures of performance.<\/p>\n Another physical limitation of a road is the physical size and width of the road. If you consider a Smart car or a VW Beetle vs. a double trailer semi truck, you can see that they will occupy a different footprint on the road. While you might have enough physical space for one of those trucks, it could probably fit 5-6 of the mini cars. Think of this idea as an analogy for throughput. Throughput is a measure of capacity.<\/p>\n We will use these concepts to describe benchmarking and how it models to the road analogy throughout the rest of this post.<\/p>\n In the following scenario, I have written a simple script that visits a web page. Every time the script finishes, it resets and starts over. Every 10 seconds, a new user joins the scenario, repeating the script in the same manner.<\/p>\n But\u2026 remember the first covenant of a benchmark?<\/p>\n What are you trying to show\/prove?<\/i><\/p>\n For this specific run, I will be illustrating one simple idea: what a healthy load looks like and the metrics that show the mathematical relationships between number of users, hits\/sec, connections, and throughput when your test systems are functioning well during a benchmarking exercise…and when you need to stop to investigate.<\/p>\n Using the traffic analogy, I am attempting to figure out how many cars I can add to the road before I start seeing traffic delays\u2026 sort of (we will discuss this in further detail later).<\/p>\n In this graph, you can see that the number of users is scaling slowly. By taking this approach, we can watch the other graphs in the tool and make sure that everything is performing up to expectations.<\/p>\n It seems pretty obvious to say that a single user generates a \u201chit\u201d (or a request for a resource from the web server via a connection), and that two users would generate twice as many, three users three times as many, etc.<\/p>\n There is a mathematical relationship between the number of users running and hits\/requests per second. They are linear, meaning that as one scales the other follows it. This relationship holds true from one user to any number of users.<\/p>\n Think of this as the cars on the road. As long as there is no bottleneck in traffic, you can add more cars without impacting commute time. As long as the server isn\u2019t overly busy, it creates the connection and sends the files.<\/p>\n In exactly the same way that users and hits correlate, so will throughput and connection requests in a well behaving benchmark<\/strong>. If you do not see this mathematical relationship between the numbers, stop. Something is broken. Yet another reason that command line\/online tools are useless- there is no visibility during the benchmark.<\/p>\n Graphs 1-4 ALWAYS move together linearly in systems that are performing well, independent of scale.<\/em><\/strong><\/p>\n Assuming that all of the mathematically bound graphs are moving well together, you need to validate that there were no errors being returned by the server during the test run. If you look at these graphs, you see nothing but passed transactions and 200 and 302 status codes (a normal redirect, not an error). Any 4XX or 5XX HTTP status codes mean that your web server was not able to handle a request (HTTP Status codes are only returned by a web server).<\/p>\n While the run didn\u2019t have a specific performance goal, it is pretty typical to watch transaction performance during a benchmark to get a sense of a few metrics:<\/p>\n 1.) What is my no load performance? Here, that occurs around 10s or so into the test and is about 2.7s.<\/p>\n 2.) What point does the number of users drive the performance above the threshold of acceptable performance? Let\u2019s pretend that it was 3s and that occurs around 2:30 into the test, which can be referenced back to 16 virtual users running.<\/p>\n 3.) What point does the system begin to fail? While this specific test was terminated prior to errors being present, it appears that response times were beginning to grow which indicates there is a problem.<\/p>\n So, I aborted this run\u2026 simply because I had the data I needed for this blog post.<\/p>\n I have a little confession to make. I know what the bottleneck in this test was. I was running the injector on my laptop, which also had the controller\u2026 and my e-mail and a bunch of other things open. My laptop began to run out of resources. At the end of the test run, you can see response times jump way up. That is a behavior that occurs consistently with the hard interruption\/termination of the tool I use, and has been observed dozens of times.<\/p>\n I thought you said not to use the load injector to monitor?<\/em><\/p>\n I did, and for this very reason\u2026 however, I know what the limits of my laptop are for this type of work, because I have benchmarked its capability as a load injector<\/strong>. However, in this case, I\u2019m also using it to illustrate what I call an \u201cillogical conclusion\u201d<\/em><\/strong>.<\/p>\n What is a garbage conclusion? Frankly, it is nothing more than a lazy guess. In some cases, it is even worse than that: it is a bad guess that goes completely against what the data is saying.<\/p>\n In my example, you might conclude that the web server was slowing down and about to break, because that is what the data looks like it is saying. In reality, the server was perfectly healthy and returning data to the injector as fast as the no load performance. It was the injector that was unable to process it in a timely manner.<\/p>\n If you overdrive your load injector, it will blow up your test results.<\/strong><\/p>\n Simply put, if you understand the mathematical relationships between the data in healthy systems, you need to stop and investigate what is going on when those relationships no longer work. It\u2019s not rocket science, but it is<\/em> science.<\/p>\n Simply opening a browser from an independent PC during a benchmark to inspect system behavior can show you if it is your web server or your load injector.<\/p>\n First, let me say that I have no other purpose in writing this article except to correct errors out in the community, which is filled with bad benchmarks and misunderstanding. It is a little bit self-serving, though, since I spend a LOT of time defending our products against bad benchmarking\/garbage conclusions. Hopefully, this blog post will clear up some of the bad practices and force people to adopt industry standard practices.<\/p>\nweighttp -n 100000 -c [0-1000 step:10 rounds:3] -t 8 -k \"http:\/\/x.x.x.x:80\/index.html\"<\/code><\/p>\n
\nA lot of online tools are available for use, but there are many issues with using them. First and foremost is the lack of control and visibility into conditions. Pingdom and webpagetest.org are good for spot checking performance, but not at all useful for a benchmark. Benchmarking needs to be done in a controlled environment to have any comparative value.<\/p>\nGood Benchmarking Practices<\/h3>\n
![\"To](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/infinite.jpg\")
<\/p>\nUnderstanding Data and Relationships<\/h3>\n
Let\u2019s benchmark!<\/h3>\n
Graph #1 (Running virtual users)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report0.png\")
<\/p>\nGraph #2 (Hits per second)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report1-lined.png\")
<\/p>\nGraph #4 (Connections)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report3.png\")
<\/p>\nGraph #3 (Throughput)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report2.png\")
<\/p>\nGraph #4 (Connections)<\/div>\n<\/p>\nGraph #5 (Passed transactions)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report4.png\")
<\/p>\nGraph #6 (HTTP status codes)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Report5.png\")
<\/p>\nGraph #7 (Transaction Response time- also, why I stopped this run!)<\/div>\n![\"Benchmarking](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/report6-lined.png\")
<\/p>\nGarbage in, Garbage Out<\/h3>\n
![\"meltdown\"](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/meltdown.jpg\")
<\/p>\nReal world examples of Bad Benchmarking and Illogical Conclusions<\/h3>\n
![\"Abraham\"](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Abraham.jpg\")
<\/p>\nExample #1<\/h4>\n
![\"Whitaker](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/whitaker.jpg\")
<\/p>\n![\"You](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/you-had-one-job.jpg\")
<\/p>\n![\"Replot\"](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/Replot.png\")
<\/p>\n![\"Deming\"](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/deming.jpg\")
<\/p>\n![\"6th](\"http:\/\/blog.litespeedtech.com\/wp-content\/uploads\/2016\/09\/school.png\")
<\/p>\n","protected":false},"excerpt":{"rendered":"