NEW LIST: The most active crawlers on the web

It's been a while since we looked into the most active web crawlers and bots (2016) on the web. For our latest dive into the data, we looked at the numbers for H1 2018, January to June inclusive.

The data is sourced from thousands of websites built and hosted on the goMobi web publishing platform. Below is the breakdown of all the bots we saw.

The most active crawlers are page preview bots

Unsurprisingly, given its omnipotence, Facebook's externalhit bot is a regular visitor. We saw 7 different types of Facebook crawler in our data.

With over 8.7% of the total visits counted, "Facebookexternalhit" has been busy crawling pages that people share on the the platform, but not quite as busy as the first three months of the year, when its share was 12% of the overall hit count.

When a link is pasted, Facebook crawl the target page and pull information such as title, description and preview/featured images, as below:

The Facebook crawler UAs seen most often are:

Although not seen in our data this time, "Facebot" is their advertising crawler, which may make an appearance on sites that use paid promotion on the platform.

You can read more about Facebook's crawlers and how to handle them here.

Bing more active than Google

The most active individual bot on our network was BingPreview. Combined, the 7 Bing bots account for 32.2% of all visits on our network.

BingPreview, "used to generate page snapshots", was closely followed by the standard crawler Bingbot. Interestingly, Bing's mobile crawler announces itself as an iPhone - the very first version, released way back in 2007.

You can read about all crawlers used by Bing, and exactly what each of them does here. Below are the user agents for the main three.

Googlebot - less active, or just more efficient?

Given their dominance of all things search, it may be surprising to see Google so far down the list for 2018. However, if you've developed the most intelligent, efficient web-crawling and indexing system the planet has ever seen, as well as the most popular free analytics package, you might not need to crawl every site quite as often as your competitors.

That said, we did spot 146 variations of Google crawlers and bots in our data, up from 102 in the first three months of 2018. Given Google's integration with the wider web - think AMP, Structured Data etc - and their move to offer Google services (A.K.A. ads) on KaiOS earlier this year, there are numerous reasons why a Googlebot would visit your site.

Below are the most common Google crawlers, and their UAs, that we saw in the first half of 2018.

Yahoo!

One of the earliest internet pioneers, Yahoo has undergone major changes in recent years. In 1994 they were called "Jerry and David's guide to the World Wide Web", and simply listed other websites. There was no search offered, but in the year 2000 they integrated Google's product. By 2004, they had developed their own search functionality.

What we see in our data is similar, but on a lesser scale to Google. There is one main crawler - Yahoo! Slurp - and 8 variations, all with different jobs to perform as they navigate the internet.

Unlike Google, Yahoo doesn't appear to have a dedicated mobile web crawler, certainly not one operating at the same scale. Given Google's recent push to the mobile-first index, it's not surprising.

Yahoo's crawlers account for 1.5% of the total bot traffic included, down from 2.6% in Q1 2008. Here are the main crawlers, and their respective UAs.

Others to note

So we've looked at the "Big Four" and their crawlers and bots, and provided the most common UserAgents we see across our network. What else in the data is interesting?

With 4% of the overall traffic, Sogou Spider is the 6th busiest bot on the list. It's the web crawler for Beijing-based search technology provider, Sogou.com.

ScoutJet, the web crawler for IBM's Watson, clocks in with 1.6% of the total, while another Chinese search engine, so.com, is visible at 1.7% with its 360Spider bot.

With a DeviceAtlas Cloud Standard, Premium or Enterprise account, you can identify non-human traffic (robots, crawlers, checkers, download agents, spam harvesters and feed readers) in real-time. You can then decide how to act on this information, whether to block all undesired bots at the door, or just treat them in a different way to legitimate human visitors.

Read more about bot detection and how it can help your business to:

• Limit Ad-fraud
• Protect your site