web crawlers

Choose a category below to quickly discover all our device intelligence articles.


web crawlers
Device Detection

The most active crawlers and bots on the web

Allowing web crawlers to scan your site is vital if you want your web pages to appear in Google, Bing...

Tomas Trnka - 02 Mar 2023
15 min read
Device Detection

Dark Traffic and Misrepresentation - Analysing the Web Analysers (Part 2)

In part 1 of our Web Analytics Comparison: Analysing the Web Analysers post, we presented research findings...

Isabel Hughes - 19 May 2022
6 min read
Device Detection

Website Analytics Comparison - Analysing the Web Analysers (Part 1)

Analysing visitor traffic is important to determine the health of a website. In this article, we present...

Isabel Hughes - 19 May 2022
9 min read
Device Detection

Bot Analytics Series Part 2: The Importance of Monitoring Bot Traffic

Analysing your bot traffic can help you understand how your site is being used (for example by preview...

Regina De Burca - 26 Aug 2021
4 min read
Device Detection

Introduction to Bot Traffic — Part One of our Bot Analytics Series

As much as 40% of internet traffic is from non-human sources. This traffic ranges from legitimate bots...

Regina De Burca - 27 Jul 2021
9 min read
Device Detection

Unveiling the most active bots and crawlers on the web in 2018

For our latest dive into web traffic data, we looked at the numbers for H1 2018 to investigate the most...

James Kielty - 15 Aug 2018
7 min read
Device Detection

6 not so obvious ways companies can use device detection

With so many devices connected to the internet most companies have understood the need for a reliable...

Pawel Piejko - 26 Sep 2017
7 min read
Device Detection

How DeviceAtlas can protect your site against bots

Non-human traffic is larger than most business owners are aware of. This is mainly due to the fact that...

dcarlier - 11 Jul 2016
7 min read
Reports

Bots and Crawlers May Represent up to 50% of Web Traffic

Dublin – May 12, 2016 – DeviceAtlas, a leading supplier of intelligence on mobile devices, today released...

Martin Clancy - 13 May 2016
2 min read
User-Agent Parsing

List of most active web crawlers User-Agent strings in 2016

To help you understand web crawlers, bots and spiders visiting your site, here's a list of the top 20...

Pawel Piejko - 28 Apr 2016
5 min read