{"id":7417,"date":"2017-06-21T06:00:21","date_gmt":"2017-06-21T10:00:21","guid":{"rendered":"https:\/\/blog.litespeedtech.com\/?p=7417"},"modified":"2017-06-21T06:00:21","modified_gmt":"2017-06-21T10:00:21","slug":"wpw-more-about-crawling","status":"publish","type":"post","link":"https:\/\/lswp.store\/index.php\/2017\/06\/21\/wpw-more-about-crawling\/","title":{"rendered":"WpW: More About Crawling"},"content":{"rendered":"

\"\"<\/p>\n

Welcome to another installment of WordPress Wednesday<\/a>!<\/p>\n

Last week<\/a> we talked about the new crawler functionality in our LiteSpeed Cache for WordPress plugin<\/a>, and since then there have been some questions about enabling the crawler, and a couple of new fields added to the Settings page<\/b>. Today, we thought it would be a good idea to talk about these things and get everyone up to speed and on the same page.<\/p>\n

Enabling the Crawler<\/h2>\n

This has been the area where we\u2019ve had the most requests for help, so let\u2019s see if we can clear up some of the confusion.<\/p>\n

The first thing to know is that the crawler must be enabled at the server or virtual host level<\/b>. If you don\u2019t have access to these areas, you will need to ask your web host to do it for you.<\/p>\n

Additionally, this functionality requires the latest build<\/b> of our server software, so if you\u2019re running 5.1.16 and you are still having difficulty getting the crawler enabled, chances are you are not running the latest build (as of this writing, that is LSWS version 5.1.16 build 3). Force a reinstall, and it should work.<\/p>\n

Enabling the crawler requires you to add a few lines to a server config file or a virtual host include file. These are those lines:<\/p>\n

<IfModule Litespeed>\n CacheEngine on crawler\n<\/IfModule>\n<\/pre>\n
Due to the fact that every control panel stores its config files in its own directory structure, we can\u2019t just say, \u201cupdate filexyz<\/code>.\u201d We have, however, written up a detailed wiki article<\/a> that explains where you need to look for your particular control panel (or for no control panel at all). We recommend that you refer to the wiki<\/a><\/b> to be sure you\u2019re putting the code in the proper location.<\/p>\n
New Fields<\/h2>\n
There are two new fields on the WP-Admin > LiteSpeed Cache > Settings > Crawler<\/b> tab. Site IP<\/b> allows the crawler to go directly to your server to retrieve content, without having to get the DNS involved. Custom Sitemap<\/b> allows the crawler to use a sitemap you already have, rather than building its own.<\/p>\n
Site IP<\/h3>\n
As of the plugin\u2019s version 1.1.1, you can enter your Site\u2019s IP address to simplify the crawling process and eliminate the overhead involved in Domain Name Server (DNS) and Content Delivery Network (CDN) lookups.<\/p>\n
Note: a server may have multiple IPs. Use the IP address that was set for your domain in the server\u2019s configuration.<\/i><\/p>\n
Conventionally, the crawler retrieves a URL to crawl from the sitemap, and looks to the DNS to get the URL\u2019s IP address. If you\u2019re using a CDN, the IP that the DNS returns is actually that of the CDN<\/i>. The CDN only caches static content, so when the crawler asks the CDN for the dynamically-generated content it needs, the CDN must go back to your server to retrieve it and then<\/i> return it to the crawler. This is so inefficient! The crawler, which lives on the same server as your content, needs to travel all over the internet before it can get that content!<\/p>\n
On the other hand, if you tell LiteSpeed\u2019s crawler ahead of time what your domain\u2019s IP address is, it doesn\u2019t need to involve the DNS or CDN at all. The content can be retrieved directly from the server. Nice!<\/p>\n
To understand this better, let\u2019s look at this diagram:<\/p>\n
\"\"<\/p>\n
In other words…<\/p>\n
If you are using a CDN<\/h4>\n