Crawling and Indexing of Locale-Adaptive Pages by Googlebot

In case of websites that don’t have separate URLs for each locale, Google uses different crawl settings.

IMPORTANT: We continue to support and recommend using separate locale URL configurations and annotating them with rel=alternate hreflang annotations.


Crawling and Indexing of Locale-Adaptive Pages by Googlebot

If your website showing different content based on the perceived country or preferred language of the visitor (i.e., you have locale-adaptive pages), Google might not crawl, index, or rank all of your locale-adaptive content. Since, by default, Googlebot requests pages without setting an Accept-Language HTTP request header and uses IP addresses that appear to be located in the USA, not all content variants of locale-adaptive pages may be indexed completely.

In order to address crawling and indexing of locale-adaptive content, we use locale-aware crawling to better surface your content for searchers around the world. Locale-aware crawling occurs when Googlebot crawls with one or both of the following configurations:

  • Geo-distributed crawling: Googlebot appears to be using IP addresses based outside the USA, in addition to the longstanding IP addresses Googlebot uses that appear to be based in the USA.
  • Language-dependent crawling: Googlebot crawls with an Accept-Language field set in the HTTP header.

Confirm that your website configuration supports locale-aware crawling

After automatically enabled these new crawling configurations for pages we detect to be locale-adaptive, without any changes in your CMS or Server settings you can see changes how we crawl and show your site in Google search results

Note that these new configurations do not alter our recommendation to use separate URLs with rel=alternate hreflang annotations for each locale. We continue to support and recommend using separate URLs as they are still the best way for users to interact and share your content, and also to maximize indexing and better ranking of all variants of your content.

As always, if you have any questions or feedback, please tell us

Girish Kapoor

SEO & Digital Marketing Analyst at Soft System Solution. I love social media. Almost as much as I love SEO. When the two come together, it’s a pretty exciting thing in the search marketing world.

You may also like...

Leave a Reply