{"id":9596,"date":"2018-04-23T06:00:23","date_gmt":"2018-04-23T10:00:23","guid":{"rendered":"https:\/\/blog.litespeedtech.com\/?p=9596"},"modified":"2018-04-23T06:00:23","modified_gmt":"2018-04-23T10:00:23","slug":"litemage-cache-warmup-for-large-stores","status":"publish","type":"post","link":"https:\/\/lswp.store\/index.php\/2018\/04\/23\/litemage-cache-warmup-for-large-stores\/","title":{"rendered":"MM: Cache Warmup for Large Stores"},"content":{"rendered":"<p><a href=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18.png\"><img fetchpriority=\"high\" decoding=\"async\" class=\"aligncenter size-full wp-image-9598\" src=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18.png\" alt=\"Magento Monday: LiteMage Cache Warmup for Large Stores\" width=\"640\" height=\"320\" srcset=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18.png 640w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18-300x150.png 300w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18-600x300.png 600w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p>Welcome to another installment of <a href=\"http:\/\/blog.litespeedtech.com\/tag\/magento-monday\">Magento Monday<\/a>!<br \/>\nToday\u2019s Topic: <strong>LiteMage Cache Warmup for Large Stores<\/strong><\/p>\n<p><em><strong>Disclaimer<\/strong>: The information contained in this post is accurate for LiteMage v1.3.7 [<a href=\"https:\/\/www.litespeedtech.com\/products\/cache-plugins\/magento-acceleration\/release-log\">release log<\/a>] and Magento 1.x. If you are using a newer version of the extension, some details may have changed. Please<a href=\"https:\/\/www.litespeedtech.com\/support\/wiki\/doku.php\/litespeed_wiki:cache:litemage\"> refer to our wiki<\/a> for the latest!<\/em><\/p>\n<p>LiteMage for Magento 1.x is equipped with a crawler function. The crawler travels its way throughout the backend, refreshing pages in the cache. The purpose is to keep the cache as fresh as possible while minimizing visitor exposure to uncached content.<\/p>\n<p><a href=\"https:\/\/www.litespeedtech.com\/support\/wiki\/doku.php\/litespeed_wiki:cache:litemage:troubleshooting:warmup#keeping_public_cache_warm_indefinitely\">Our wiki instructions for setting up the LiteMage crawler<\/a> are detailed and extensive, and if you&#8217;ve never enabled your site&#8217;s crawler before, we recommend you start there. If you own a large store, you may find the crawler&#8217;s duration to be problematic, so today we&#8217;re going to discuss <strong>how to configure the crawler for stores with many thousands of products<\/strong>.<\/p>\n<p>What works for small-to-medium sized stores may not work for larger stores. If you have thousands of pages for the crawler to traverse, you may need to adjust the crawler&#8217;s settings to make it less intrusive into your store&#8217;s daily operation.<\/p>\n<h2>How Long Does Your Crawler Take?<\/h2>\n<p>Before you can begin to determine the optimal configuration for your store\u2019s crawler, you will need to know how long it actually takes to complete.<\/p>\n<p><a href=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-9600\" src=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19.png\" alt=\"LiteMage Cache Warmup for Large Stores: LiteMage Crawler Warmup Status\" width=\"1000\" height=\"177\" srcset=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19.png 1000w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19-300x53.png 300w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19-768x136.png 768w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-19-600x106.png 600w\" sizes=\"(max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<p>You can find that out, along with other useful crawler statistics, in the <strong>LiteMage Crawler Warm-up Status<\/strong> section. From the Magento Admin Panel, navigate to <strong>System &gt; Cache Management<\/strong>. It should be at the bottom of that page.<\/p>\n<h2>An Example Scenario<\/h2>\n<ul>\n<li><strong>Shop configuration<\/strong>: A book shop with 15 thousand products<\/li>\n<li><strong>Public Cache TTL (Time to Live)<\/strong>: 86400 (24 hours)<\/li>\n<li><strong>Crawler Warmup Interval<\/strong>: 57600 (16 hours)<\/li>\n<li><strong>Crawler Duration<\/strong>: 8 hours<\/li>\n<\/ul>\n<p>Fifteen thousand product pages are set to expire in the cache after 24 hours, so every 16 hours the crawler begins refreshing the pages. This process takes about 8 hours to complete, and consumes enough resources to slow the site down considerably.<\/p>\n<p>Crawling the site takes 8 hours, and it runs every 16 hours. Essentially <em>half of the time that this store is up, it is experiencing crawler-induced lag<\/em>.<\/p>\n<p>This is not ideal. However, you don\u2019t want to give up the warm cache, so what can you do?<\/p>\n<h2>A Possible Solution<\/h2>\n<p><a href=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8.jpeg\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-9602\" src=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8.jpeg\" alt=\"LiteMage Cache Warmup for Large Stores: Crowded Bookshelves\" width=\"1000\" height=\"698\" srcset=\"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8.jpeg 1000w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8-300x209.jpeg 300w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8-768x536.jpeg 768w, https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-8-600x419.jpeg 600w\" sizes=\"(max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<p>LiteMage\u2019s crawler function is flexible, and with a few adjustments, you can tune it to work well for your store.<\/p>\n<p>The first thing to consider is the TTL. How often do your products change? If it\u2019s less frequently than \u201cevery day,\u201d you don\u2019t need a TTL of 24 hours. How about a week? If it\u2019s reasonable to assume that your products are not likely to change within seven days, you can set your TTL to 604800 seconds (1 week).<\/p>\n<p>Once your TTL is changed, you can adjust your warmup interval. We have a handy rule-of-thumb for that:<\/p>\n<p><strong>Warmup Interval &lt;= Public Cache TTL &#8211; 2 X Warmup Duration<\/strong><\/p>\n<p>Let\u2019s say the TTL is set to 604800 (7 days).<\/p>\n<p>We know our warmup duration is about 8 hours, or \u2153 day.<\/p>\n<p>The Warmup Interval we choose should be less than or equal to 7 days minus 2 times \u2153 day<\/p>\n<p>In other words, the Warmup interval should be less than 6 \u2153 days.<\/p>\n<p>Let\u2019s set the Warmup Interval to 6 days (or 518400 seconds). This means that instead of your system slowing down every day for 8 hours, it will slow down every <em>six<\/em> days for 8 hours. That\u2019s a big improvement.<\/p>\n<p>You can also enable the delta crawler. The delta crawler will allow you to confidently set a one-week TTL, knowing that if a product changes during that time, it will be recrawled within 5 minutes.<\/p>\n<h2>In Conclusion<\/h2>\n<p>As you can see, with the LiteMage crawler, you have options. If your products are fairly static, you might even choose to set a one <em>month<\/em> TTL, or longer!<\/p>\n<p>If you have any questions about this, leave a comment. We\u2019ll be happy to help!<\/p>\n<p>P.S. Perhaps your large store could benefit from our ADC solution. Check out <a href=\"https:\/\/www.litespeedtech.com\/products\/litespeed-web-adc\">LiteSpeed Web ADC<\/a> for details.<\/p>\n<p>&#8212;<\/p>\n<p>Have some of your own ideas for future <a href=\"http:\/\/blog.litespeedtech.com\/tag\/magento-monday\">Magento Monday<\/a> topics? Leave us a comment!<\/p>\n<p>If you enjoyed this, here are a few other things you can do:<\/p>\n<ul>\n<li>Subscribe to the <a href=\"http:\/\/blog.litespeedtech.com\/tag\/magento-monday\/feed\">Magento Monday RSS feed<\/a><\/li>\n<li>Download <a href=\"https:\/\/www.litespeedtech.com\/support\/wiki\/doku.php\/litespeed_wiki:cache:litemage:installation\">LiteMage Cache for Magento 1<\/a><\/li>\n<li>Download <a href=\"https:\/\/github.com\/litespeedtech\/magento2-LiteSpeed_LiteMage\">LiteMage Cache for Magento 2<\/a> from the GitHub Repository<\/li>\n<li>Learn more about the plugin <a href=\"https:\/\/www.litespeedtech.com\/products\/cache-plugins\/magento-acceleration\">on our website<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>What works for small-to-medium sized stores may not work for stores with thousands of products.<\/p>\n","protected":false},"author":1,"featured_media":9598,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[25],"tags":[81,175,178],"class_list":["post-9596","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-litemage","tag-crawler","tag-magento","tag-magento-monday"],"jetpack_featured_media_url":"https:\/\/lswp.store\/wp-content\/uploads\/2018\/04\/null-18.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/9596","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/comments?post=9596"}],"version-history":[{"count":0,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/9596\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media\/9598"}],"wp:attachment":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media?parent=9596"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/categories?post=9596"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/tags?post=9596"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}