{"id":9158,"date":"2018-02-14T02:00:43","date_gmt":"2018-02-14T07:00:43","guid":{"rendered":"https:\/\/blog.litespeedtech.com\/?p=9158"},"modified":"2018-02-14T02:00:43","modified_gmt":"2018-02-14T07:00:43","slug":"limiting-the-litespeed-cache-crawler-for-wordpress","status":"publish","type":"post","link":"https:\/\/lswp.store\/index.php\/2018\/02\/14\/limiting-the-litespeed-cache-crawler-for-wordpress\/","title":{"rendered":"WpW: Limiting the LSCache Crawler"},"content":{"rendered":"

\"WordPress<\/a><\/p>\n

Welcome to another installment of WordPress Wednesday<\/a>!
\nToday\u2019s topic is: Limiting the LiteSpeed Cache Crawler<\/strong><\/p>\n

Disclaimer<\/strong>: The information contained in this post is accurate for LSCWP v1.9 [release log<\/a>]. If you are using a newer version of the plugin, some details may have changed. Please refer to our wiki<\/a> for the latest!<\/em><\/p>\n

When we introduced the LiteSpeed Cache Crawler<\/strong> in our WordPress plugin last May, we told you<\/a> that the crawler has the potential to consume significant system resources, if not configured with efficiency in mind. It was for this reason that we put the on\/off switch in the hands of the hosting providers.<\/p>\n

Some hosts wanted the ability to allow their clients the use of a LSCWP crawler, but only if the providers had the ability to set some limits.<\/p>\n

We thought that was a great idea, and so today we\u2019re going to talk about the new crawler-specific server variables that we created in response to this request.<\/p>\n

Why Limit the Crawler?<\/h2>\n

 <\/p>\n

\"Limiting<\/a><\/p>\n

The settings in LSCWP\u2019s Crawler tab have always been designed to minimize server impact. Site admins are encouraged to configure their crawler to be conservative in its usage of resources. That said, users are free to configure it however they like. Site admins on shared hosting could conceivably use crawler settings that bring a server to its knees if they wanted to, negatively impacting every other site on that server.<\/p>\n

With no safeguards in place to protect against the rogue site owner in the second scenario, many shared hosting providers have decided not to take a chance on enabling the crawler.<\/p>\n

But now, with these new server variables, we are giving hosting providers the ability to set some boundaries for the site owners<\/strong>. The idea being, if they can guarantee that no client can hog all of the resources, then there should be enough to go around.<\/p>\n

Exactly What Do These New Variables Limit?<\/h2>\n

Currently hosting providers can potentially limit two of the crawler settings that are found in LiteSpeed Cache > Settings > Crawler<\/strong>.<\/p>\n

Delay<\/h3>\n

This setting lets LSCache know how often to send a new request to the server. The longer the delay, the less load there is on the server, but the longer it takes to crawl. LSCache defaults Delay<\/strong> to 500 microseconds, but site admins may change the number to whatever they like.<\/p>\n

The CRAWLER_USLEEP<\/code> variable puts a minimum allowed value on the Delay<\/strong> field. You can specify a large number, if you want to slow down all of the crawlers on your server and minimize their impact.<\/p>\n

For example, if you set CRAWLER_USLEEP<\/code> to 1000<\/code> microseconds, then your site admins may set their own crawler delays to any number greater than or equal to 1000. All site crawlers across the server will never be allowed to run faster than 1000 microseconds between requests.<\/p>\n

Server Load Limit<\/h3>\n

Server Load Limit keeps the crawler from monopolizing system resources. Once the load reaches this limit, the crawler will be terminated rather than allowing it to compromise server performance. LSCache defaults Server Load Limit<\/strong> to 1<\/code>, which is fairly conservative, but site admins have the ability to crank it up as high as they like.<\/p>\n

With the following server variables, hosting providers can prevent site owners from setting the limit too high.<\/p>\n