crawlers list

Choose a category below to quickly discover all our device intelligence articles.


crawlers list
Device Detection

The most active crawlers and bots on the web

Allowing web crawlers to scan your site is vital if you want your web pages to appear in Google, Bing...

Tomas Trnka - 02 Mar 2023
15 min read
Device Detection

Unveiling the most active bots and crawlers on the web in 2018

For our latest dive into web traffic data, we looked at the numbers for H1 2018 to investigate the most...

James Kielty - 15 Aug 2018
7 min read
User-Agent Parsing

List of most active web crawlers User-Agent strings in 2016

To help you understand web crawlers, bots and spiders visiting your site, here's a list of the top 20...

Pawel Piejko - 28 Apr 2016
5 min read