How bots are kept out

Two layers

A lot of web traffic isn’t people: search engine crawlers, uptime monitors, scrapers, link previews, test tools. Crustat keeps them out of your numbers in two steps.

1. Obvious bots are turned away at the door. They’re never stored and never shown.
2. Likely bots are set aside. They’re stored with a reason, left out of every number on your dashboard, and listed separately so you can look at them.

There’s nothing to set up, and the rules are the same for every site.

Turned away: obvious bots

Every browser and program sends a short description of itself. If that description is empty, or says it’s a bot, crawler, spider, scraper, headless browser, monitor or a common programming tool, the visit is dropped on arrival. That covers search engine crawlers, Lighthouse and other speed tests that say who they are, uptime checkers, link previews like Facebook’s, and scripts written with tools such as curl or Python.

These never show up anywhere, not even in the list below.

Set aside: likely bots

Some visits look like a normal browser but aren’t a person. Crustat sets them aside for one of three reasons:

• From data centres (cloud servers). The visit comes from a rented server rather than a home, an office or a phone. Crustat checks against about 30 hosting networks, like Amazon Web Services, Google Cloud, Microsoft Azure, DigitalOcean, Hetzner and OVH. Cloudflare, Akamai, Fastly and Apple are left off on purpose, so people using iCloud Private Relay or similar privacy services still appear in your stats.
• Automated browsers. The browser says it’s being run by a program, or names a testing or monitoring tool like PageSpeed or GTmetrix.
• Copies of your site on other addresses. Your page loaded on an address that isn’t your site, such as a translation service or someone’s copy of your site. Your own domain, its subdomains and any domains you added under Site settings → General → Domains are fine. See Add and manage your sites.

Where to see them

On your dashboard, under the chart, a note says how many were left out, like 12 bot visits left out. It’s hidden when there are none.

Click it to see the total for each reason. From there, More info opens a panel with one tab per reason: Data centres, Automated and Copies. Each tab shows:

• the pages they opened,
• where they connect from (the network, with a link to its public record),
• countries, browsers and where they came from,
• the most recent visits.

The Copies tab also lists where the copies are. Those addresses can be harmful, so Crustat doesn’t link to them. Copy one and check it with a scanner like urlscan.io before opening it.

You can download the list as a CSV file with Download CSV.

What to keep in mind

• The note only shows set-aside visits. Obvious bots turned away at the door never appear in it, so the real amount of bot traffic is higher.
• The rules can’t be changed per site. There’s no list to edit or allow.
• No filter is perfect. A well-disguised bot on a home connection can slip through, and a real person on a work VPN that runs through a cloud server can be set aside.

For what each visit sends, see What we collect.