Crawling to a Standstill?

Can being crawled by Googlebot Desktop lead to indexing delays..?

Google’s John Mueller addressed this very question in a recent SEO office-hours hangout; can getting crawled by the desktop iteration of Googlebot go some way to explaining the frustrating (and sometimes even costly) problem of delayed indexing?

Googlebot desktop of course works in unison with Googlebot mobile and John was keen to offer insight into the two, how they work and whether it actually matters to indexing efficiency if one, the other or both play a role in crawling a page.

The question was posed by someone who was clearly concerned about falling foul of this potential issue after some confusion over which did the crawling and which did the (worryingly slow) indexing.

Was Googlebot Desktop to blame?

Googlebot Mobile and Desktop

Google has an array of website crawlers which sniff out links and follow them to discover new pages in need of crawling and indexing. A closer look at traffic logs might see any of the following Google or Google-related crawlers that may visit a website:

• Googlebot Desktop
• Googlebot Image/Video
• Googlebot Smartphone
• Google-Read-Aloud
• Adbots/Feedfetcher-Google
• Mediapartners-Google (Mobile AdSense)

The question John Mueller addressed was primarily concerned with Googlebot Desktop and Smartphone and what impact (if any) the Desktop version may have:

“Our website’s posts are being indexed after one week. The site is old and mobile-friendly.

When I submit URLs in Inspection Tools, it shows it’s crawled by Google Smartphone.

But in the ‘About’ section, the indexing crawler is shown as Googlebot Desktop.

Does this mean the Smartphone bot is crawling the URL and the Desktop bot is indexing this URL?

Is this the reason behind the indexing delay?”

So, is the concern a valid one, does a certain Googlebot crawler cause delays in indexing over which users appear to have little control?

Here is John’s reassuring response:

“This would not be a reason for any kind of indexing issues.

I don’t know offhand why you might see this mix of desktop and mobile crawlers there.

It might just be that the Search Console is essentially saying well, by default, the Inspection Tool will just use the mobile Googlebot.

In practice, we always crawl with both of these crawlers.

It’s just a matter of how much we crawl and index with any of these crawlers.

So, usually you’ll see something like 80% is coming from mobile Googlebot and 20% maybe from desktop Googlebot.

And that kind of ratio, that is essentially normal and doesn’t mean that things are slower than anything else.” 

So, it would appear that multiple versions of Googlebot are busily crawling the web, which is completely normal  and sites will probably be crawled by both Googlebot Desktop and Googlebot mobile (Smartphone) with no impact on indexing times.

John’s insight is reassuring for the rest of us; however it may still leave the person posing the question with more investigative work to do.

You can find useful documentation including an overview of Googlebot and an in-depth overview of how crawlers access your site over on Google Search Central.

See below video from Google SEO Office Hours - John Mueller talks on this at @34:30