{"id":4537,"date":"2016-03-23T14:39:08","date_gmt":"2016-03-23T18:39:08","guid":{"rendered":"http:\/\/blog.litespeedtech.com\/?p=4537"},"modified":"2016-03-23T14:39:08","modified_gmt":"2016-03-23T18:39:08","slug":"how-we-benchmark-at-litespeed-technologies","status":"publish","type":"post","link":"https:\/\/lswp.store\/index.php\/2016\/03\/23\/how-we-benchmark-at-litespeed-technologies\/","title":{"rendered":"How We Benchmark at LiteSpeed Technologies"},"content":{"rendered":"<p>Permit me to introduce myself.<\/p>\n<p>My name is Steven Antonucci, and I am the head of Business Development for LiteSpeed Technologies. In essence, I own sales and marketing for LiteSpeed, but I have a very technical background. I\u2019ve been around the industry for 28 years, holding a degree in Electrical Engineering from Stevens Institute of Technology. For the past 17 years, I have worked for Mercury Interactive\/HP Software directly in various sales and marketing and TECHNICAL roles&#8230;including the NYC lead performance architect for LoadRunner.<\/p>\n<p><!--more--><\/p>\n<p>So, briefly\u2026 I know what a good benchmark looks like.<\/p>\n<p>When I joined LiteSpeed Technologies a few weeks ago, I immediately became aware that people doubted our benchmark results. And frankly, they were correct.<\/p>\n<p>It wasn\u2019t so much that the testing we did was rigged (it wasn\u2019t), but the way that we talked about it made no sense. So, being here now a full month, and having a much better understanding of the market, the players, and our process\u2026 I\u2019m fixing that.<\/p>\n<p>Right now.<\/p>\n<p><strong>The Four Covenants of a Benchmark<\/strong><\/p>\n<p>I have been working on this post now for a few days (mentally), including 3AM this morning. When you consider what it takes to produce a meaningful benchmark, I would offer that it boils down to Four Covenants:<\/p>\n<ul>\n<li>Ethics<\/li>\n<li>Tools<\/li>\n<li>Skills<\/li>\n<li>Methodology<\/li>\n<\/ul>\n<p>If you are missing any of these items, your benchmark is meaningless.<\/p>\n<p><strong>Ethics<\/strong><\/p>\n<p>Everything starts with ethics. If you cannot trust the people, you cannot trust the result.<\/p>\n<p>When we benchmark, we will always describe the environments. We know that our stuff works better than our competitors, and we want to be on a level playing field. If we are comparing A to B, we are comparing them on the same hardware, with all of the recommended basic installations. We are not \u201ctuning\u201d any of the environments to perform better or worse, nor in any way changing the loads to make one solution appear better.<\/p>\n<p>I am in charge of the ethics of the benchmarks we produce here at LiteSpeed. I simply will not tolerate any violations of this policy.<\/p>\n<p><strong>Tools<\/strong><\/p>\n<p>When I got here, most of our tooling was around Siege. Most of our benchmark data came from spreadsheets that were gathered \u201csomehow\u201d and conclusions were reached \u201csomehow\u201d.<\/p>\n<p>Being comfortable with LoadRunner (I am LoadRunner Certified), I immediately upgraded our tooling. If you are unaware of LoadRunner\u2019s position, it is the dominant load testing platform used by more than 75% of the market to assure mission critical applications and websites. It also provides visibility into benchmark data that we never had in the past, and automates A LOT of the manual data assembly and presentation stuff.<\/p>\n<p>LoadRunner works differently than Siege. LoadRunner proxies the browser to record a context sensitive script that can follow a business process like logging in, browsing a catalog, adding an item to a shopping cart, and checking out. This is particularly important in platforms like Magento, as shopping cart functionality is not something you can benchmark via Siege (or similar tools that just call URL\u2019s)<\/p>\n<p>We connect to the environments we are benchmarking with HP\u2019s SiteScope for LoadRunner tool. Looking into CPU, memory, and other metrics while we are applying test loads helps us explain why we are faster. It also gives us a demo environment for trade shows like WHD, Magento Imagine, and HostingCon.<\/p>\n<p>That\u2019s right. If you want to see it live, we\u2019ll let you put your hands on it. If you want to replicate our tooling, I\u2019m happy to help (santonucci@litespeedtech.com). We are using the Community Editions of both products, and there is zero cost.<\/p>\n<p><strong>Skills<\/strong><\/p>\n<p>In my 28 years in technology, I\u2019ve talked to A LOT of companies, and they all claim to have the best, smartest people working there. Statistically, we all know that this is impossible.<\/p>\n<p>Even though I am LoadRunner certified, I wouldn\u2019t hire me to do benchmarking. I have the luxury of a breadth of industry contacts to help me do my job. Today, I am having my friends at Foulk Consulting help me with our benchmarking. (http:\/\/foulkconsulting.com\/) I\u2019ve known Ryan and his team for 15 years, and I trust that they have the right people to help me. If the largest sports apparel manufacturer trusts Ryan\u2019s guys to fix their shopping cart issues&#8230;you should too.<\/p>\n<p><strong>Methodology<\/strong><\/p>\n<p>Being at Mercury during the first DotCom wave, I learned A LOT about methodology. I supported the ActiveTest products. We offered a 2X performance gain guarantee or we would refund the customer\u2019s money. My job was to clearly understand their environments and assess the risk with the engagement. Our technical team handled all of the heavy lifting for things like the Democratic National Convention and the NBA.com site when Yao Ming joined the Houston Rockets and suddenly brought one billion Chinese people to the site every morning&#8230;big stuff.<\/p>\n<p>The way you design a benchmark test really matters. When I arrived, our tests were very simplistic and our conclusions very grandiose. I am bringing way more rigour to the effort so that our benchmarks become more focused and our results will be significantly more concrete. You will begin to see more mixed traffic, scenario based benchmark data describing specific, validated data points from many more test runs.<\/p>\n<p>Which is what every world class engineering team at every company on the planet does.<\/p>\n<p><strong>Summary<\/strong><\/p>\n<p>Moving forward, you should expect to be blown away by our benchmarks. They will be far more detailed than anything available in the industry. We\u2019re going to \u201copen source\u201d everything we do around the benchmarks- meaning you\u2019ll be able download the same tools we use, use our scripts, our scenarios\u2026 and you\u2019ll get the same results.<\/p>\n<p>Finally, I would extend the offer to anyone that wants it\u2026 Feel free to reach out to me.<\/p>\n<p>santonucci@litespeedtech.com<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Permit me to introduce myself. My name is Steven Antonucci, and I am the head of Business Development for LiteSpeed Technologies. In essence, I own sales and marketing for LiteSpeed, but I have a very technical background. I\u2019ve been around the industry for 28 years, holding a degree in Electrical Engineering from Stevens Institute of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":4733,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[16,20],"tags":[],"class_list":["post-4537","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-benchmarks","category-performance"],"jetpack_featured_media_url":"https:\/\/lswp.store\/wp-content\/uploads\/2016\/03\/8-cores.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/4537","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/comments?post=4537"}],"version-history":[{"count":0,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/4537\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media\/4733"}],"wp:attachment":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media?parent=4537"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/categories?post=4537"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/tags?post=4537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}