{"id":7203,"date":"2017-06-14T06:00:00","date_gmt":"2017-06-14T10:00:00","guid":{"rendered":"http:\/\/blog.litespeedtech.com\/?p=7203"},"modified":"2017-06-14T06:00:00","modified_gmt":"2017-06-14T10:00:00","slug":"wpw-crawl-your-site-make-it-fly","status":"publish","type":"post","link":"https:\/\/lswp.store\/index.php\/2017\/06\/14\/wpw-crawl-your-site-make-it-fly\/","title":{"rendered":"WpW: Crawl Your Site; Make it Fly!"},"content":{"rendered":"

\"WordPress<\/a><\/p>\n

Welcome to another installment of WordPress Wednesday<\/a>!
\nToday’s Topic:\u00a0LiteSpeed’s WordPress Cache Crawler<\/strong><\/p>\n

We\u2019re going to talk about the concept of \u201ccrawling\u201d and why it has the potential to make your WordPress site even faster<\/b>.<\/p>\n

What does a crawler do?<\/h2>\n

Our LiteSpeed Cache for WordPress plugin<\/a> includes crawler functionality. What does this mean?<\/p>\n

LSCache\u2019s crawler, travels its way throughout the backend, refreshing pages that have expired in the cache. The purpose is to keep the cache as fresh as possible while minimizing visitor exposure to uncached content.<\/p>\n

IMPORTANT NOTE: <\/strong>Crawler functionality is enabled or disabled at the server level, and its availability is controlled by your hosting provider. We\u2019ll talk about that more below, or you can see our docs on how to enable the crawler<\/a>.<\/span><\/p>\n

\"WordPress<\/p>\n

Why would we want to do this?<\/h2>\n

Well, first let\u2019s look at how pages are cached without <\/i>a crawler. The whole process is initiated by user request. The cache is empty until users start sending requests to the backend. The first time a page is visited, the request hits the backend, WordPress\u2019s PHP code is invoked to generate the page, the page is served to the user, and then stored in cache for next time.<\/p>\n

That\u2019s a fairly time-consuming (and bandwidth-consuming!) process for the server.<\/p>\n

Now, let\u2019s look at what happens when the cache is built by a crawler. When the crawler requests a page, the request hits the backend, WordPress\u2019s PHP code is invoked to generate the page, but because of the special header that lets LSWS know that this is a crawler that initiated the request, the full page doesn\u2019t need to be served. It is simply stored in cache.<\/p>\n

This saves significant bandwidth.<\/p>\n

Additionally, with the crawler refreshing expired pages at regular intervals, the chances that a user will encounter an uncached page is significantly diminished. This makes for a faster site.<\/p>\n

Let\u2019s look at some settings<\/h2>\n

You want the crawler to be effective in its mission, but not at the expense of your system\u2019s performance. It\u2019s helpful to know what some of the settings mean, so you can control just how many resources you want to give to the crawling process.<\/p>\n

Note<\/strong>: We’ve selected default settings that should be reasonable for most sites, so if you don’t want to be messing around with these settings, it’s fine to leave them as-is.<\/p>\n

Navigate to WP-Admin > LiteSpeed Cache > Crawler > General Settings<\/b> and take a look at the following settings:<\/p>\n

\"LSCache<\/p>\n

Crawler<\/h3>\n

This setting allows you to enable (ON<\/code>) or disable (OFF<\/code>) the crawler. It defaults to OFF<\/code>.<\/p>\n

Delay<\/h3>\n

The crawler sends requests to the backend, one page after another, as it traverses your site. This can put a heavy load on your server if there is no pause between requests. Set the Delay<\/b> to let LSCache know how often to send a new request to the server. The default is 500<\/code> microseconds (or 0.0005 seconds). You can increase this amount to lessen the load on the server, just be aware that will make the entire crawling process take longer.<\/p>\n

Run Duration<\/h3>\n

In order to keep your server from getting bogged-down with behind-the-scenes crawling, you can put limits on the crawling duration. For example, if we set Run Duration<\/b> to 400<\/code> seconds, then the crawler will run for 6.66667 minutes before taking a break. After the break is over, the crawler will start back up exactly where it left off and run for another 400 seconds. This will continue until LiteSpeed crawls the entire site.<\/p>\n

Interval Between Runs<\/h3>\n

This setting determines the length of the break mentioned above. In that same example, if we set Interval Between Runs<\/b> to 600<\/code> seconds, then the crawler would pause for 10 minutes after every 1-minute run.<\/p>\n

\"LSCache<\/p>\n

Crawl Interval<\/h3>\n

How often do you want to re-initiate the crawling process? This depends on how long it takes to crawl your site. The best way to figure this out is to run the crawler a couple of times and keep track of the elapsed time. Once you\u2019ve got that amount, set the interval to slightly more than that. For example, if your crawler routinely takes 4 hours to complete a run, you could set the interval to 5 hours (or 18000<\/code> seconds).<\/p>\n

Threads<\/h3>\n

When Threads<\/b> is set to 3<\/code>, then there are 3 separate crawling processes happening concurrently. The higher the number, the faster LiteSpeed crawls your site, but also the more load that the crawler puts on your server.<\/p>\n

Timeout<\/h3>\n

Use this setting to specify how long you are willing to take processing a single URL. The crawler has this many seconds to crawl a page before moving on to the next page. The default is 30<\/code>, but you can adjust that up or down according to your needs.<\/p>\n

Server Load Limit<\/h3>\n

This setting is a way to keep the crawler from monopolizing system resources. Once the server’s load average reaches this limit, the crawler will be terminated, rather than allowing it to compromise server performance.<\/p>\n

For example, to terminate the crawler once half of your server resources are being consumed, set Server Load Limit<\/strong>\u00a0to\u00a00.5<\/code>\u00a0for a one-core server,\u00a01<\/code>\u00a0for a two-core server,\u00a02<\/code>\u00a0for a four-core server, and so on.<\/p>\n

Protecting your server from overload<\/h2>\n

We designed all of the settings above to work together to protect your server from overload. The first four parameters (Delay<\/strong>, Run Duration<\/strong>, Interval Between Runs<\/strong>, and Crawl Interval<\/strong>) control how often and for how long to allow the crawler to run. You can set these values to give your crawler as many or as few resources as your system can afford.<\/p>\n

Timeout<\/strong> keeps the crawler from getting stuck for too long on any one page.<\/p>\n

Threads<\/strong> and Server Load Limit<\/strong> are two settings that work together to automatically terminate the crawler if it tries to get too greedy with system resources. Here\u2019s an example:<\/p>\n

Let\u2019s say we have the following values set before the crawler starts:<\/p>\n

Server Load Limit<\/strong> = 5<\/code>
\nThreads <\/strong>= 4<\/code><\/p>\n

The current server load is at 2, and the crawler begins. It\u2019s crawling 4 urls at a time (due to the Threads<\/strong> setting), but this has caused the server load to jump up above our limit of 5. In response, the crawler drops the number of threads to 3 and keeps going.<\/p>\n

If the server load hits our limit of 5 again, the crawler will drop the number of threads to 2 and go on. This process repeats until we are down to a single thread.<\/p>\n

If the server load is still too high with only one crawler thread, the crawling process terminates.<\/p>\n

On the other hand, if the server is doing just fine with one thread, the crawler will increase the number of threads one at a time, until it has either reached the Thread Limit<\/strong> we set (4 in this case) or our Server Load Limit<\/strong>.<\/p>\n

Watching the crawler do its thing<\/h2>\n

If you\u2019re the kind of person who likes to sit in front of a terminal and watch a process making progress (and, really, who isn\u2019t?) you\u2019ll enjoy the Watch Crawler Status<\/b> section.<\/p>\n

Navigate to WP-Admin > LiteSpeed Cache > Crawler > Summary<\/b>.<\/p>\n

\"LSCache<\/p>\n

If the crawler isn\u2019t actually running right now, make sure you set Crawler<\/b> to ON<\/code> on the previous screen, and that one or more of the crawlers listed on this page have Activate<\/strong> set to ON<\/code>. If a crawler is not currently running, you can press the Manually run<\/b> button.<\/p>\n

\"LSCWP<\/p>\n

Look at it go!<\/p>\n

Impacts on Shared Hosting<\/h2>\n

Site owners love the crawler functionality, but if you are a shared hosting provider, chances are your first thought goes to the impact all of this crawling could have on your servers. Despite the fact that we\u2019ve provided several features designed to minimize a crawler\u2019s impact on the server, it’s a valid concern when you are hosting thousands of installations. This is why we have ultimately put crawler control into the hands of the hosts.<\/p>\n

LiteSpeed Web Server disables crawling by default<\/b>. This will keep usage restricted to those who truly want to use it. As host, it is your choice whether to enable crawler functionality at all. You can keep it permanently-disabled on a server-wide basis, if you do not want crawling on your system.<\/p>\n

When crawling is disabled, site administrators will see the following message in WP Admin > LiteSpeed Cache -> Crawler<\/b><\/p>\n

Warning: The crawler feature is not enabled on the LiteSpeed server. Please consult your server admin or hosting provider.<\/pre>\n
\"LSCWP<\/p>\n
How to Enable the Crawler<\/h3>\n
NOTE: we do not recommend crawler usage for shared hosting setups unless the server has enough capacity to handle it!<\/b><\/p>\n
There are a few different approaches you can take to crawling on your LiteSpeed Web Server:<\/p>\n