The Expansel bot
You found expansel-monitor in your server logs. This page explains who sends it, why it visited your site, and how to control it.
How to recognise it
Every request we make identifies itself with this User-Agent:
expansel-monitor/1.0 (+https://expansel.com/robot)
The name to use in a robots.txt file: expansel-monitor
Anyone can borrow a User-Agent. If a request using our name behaves differently from what this page describes, send us the IP address and the time: we will tell you whether it was us.
Why it visited your site
Expansel monitors websites and checks links on behalf of its customers. Our bot visits you for one of these reasons:
- a customer monitors their own site: at regular intervals we check that it responds, how fast, and that its certificate is valid;
- a customer crawls their site to find broken links, redirects and errors: we read its pages by following its links, as a visitor would;
- a site we crawl links to yours: we check that this link works. That means one or two requests to that single address, never a crawl of your site;
- a customer checks a backlink: we read the page of your site that is supposed to link to theirs;
- someone used one of our free tools on an address of your site;
- we are offering you a free audit: we sometimes analyse a public site to send its owner a report on its broken links. These audits always follow robots.txt.
What it does, and what it never does
- It reads, and nothing else: GET and HEAD requests. It never fills in a form, never logs in, never creates an account and never buys anything.
- It spaces out its requests to the same server, slows down on its own when the server slows down, and waits as long as asked when it receives a 429 or a 503 with a Retry-After header.
- It does not download your files: a link to a PDF or an archive is judged on the response headers, and the transfer is cut straight away.
- It follows a limited number of redirects, to show our customers where their links really lead.
robots.txt
Our crawls follow your robots.txt by default: Disallow and Allow rules, and the Crawl-delay directive, read from the group that names us or, failing that, from the generic group (User-agent: *).
To close your whole site to our crawls:
User-agent: expansel-monitor
Disallow: /To slow them down without blocking them (a delay in seconds between two pages):
User-agent: expansel-monitor
Crawl-delay: 5As with other tools of this kind, three uses do not read robots.txt: checking a link found elsewhere (a visitor clicking it would send the same request), monitoring an address a customer has registered, and a one-off check run from our free tools. A customer can also turn off robots.txt or Crawl-delay when crawling a site they manage, for example a staging site closed with Disallow: /.
If your firewall refuses it
A refusal (403, 429, 503…) says nothing about whether a page exists. To tell a firewall from a broken link, we may make a second attempt, identifying as a common browser and through another network library. If that fails too, the link is shown to our customer as "not verified", never as broken. During a crawl, a page your robots.txt closes to us is not requested under any identity.
To let it through, allow the User-Agent above in your firewall. Since a User-Agent can be imitated, write to us if you need our current IP addresses.
What we record
For each request: the address requested, the response code, the response time and, for a page, the links it contains. The rest is set out in our privacy policy.
Contact us
A problem, a question, a request to stop? Write to us with the address of your site and, if you can, an extract of your logs (IP address and time). We reply by email.