Google has introduced locale-aware crawl configurations for Googlebot for pages that are detected by Google thus adapting to the content they serve based on the request’s language and perceived location of the user.
These locale-adaptive pages are designed in such a way that their content reflects the user’s language or his alleged physical location. By default, Googlebot ask for pages without setting an Accept-Language HTTP request header and employs IP addresses that materialize to be located in the USA.
Below are the configurations introduced by Google for Googlebot for pages:
- Geo-distributed crawling: this facility is where Googlebot begins to make use of its IP addresses which normally appear from countries outside the USA, in addition to the current IP addresses that appear to be from the USA that Googlebot currently uses.
- Language-dependent crawling: this facility allows Googlebot to start crawling with an Accept-Language HTTP header in the request.
Since these new crawling configurations are automatically enabled for pages, Google detect to be locale-adaptive thus users may realize changes on how it crawls and show users’ site in Google search results without altering their CMS or server settings.
Worth noting is that new configurations do not change Googlebot’s recommendation to use separate URLs for each locale.
