Technical details

Identity

CrawlerX makes two kinds of visit. An audit is a full crawl of one site. A competitor snapshot reads up to 500 of a site's public HTML pages, with no browser. Both send this User-Agent header.

CrawlerX/1.0 (+https://crawlerx.ai)

Its robots.txt token is CrawlerX. CrawlerX was first called FathomSpider. Until the new name reaches every crawl, requests can carry the header FathomSpider/1.0 followed by a link and match the robots.txt token FathomSpider. Match both names in your logs, server rules and robots.txt.

Requests made outside the browser carry the CrawlerX user agent by default. On an audit, the person running it can set a different one for those requests, such as Googlebot, Googlebot smartphone, Bingbot, an AI crawler or desktop Chrome. Browser visits and the form test always send the CrawlerX user agent. A competitor snapshot always sends it and cannot be set to anything else.

There is no published IP range and no reverse DNS check. The user agent is the only thing to match on.

How to block or slow it

Answer the CrawlerX user agent with a 403 at your server or firewall. That is the one method that turns away audits as well as snapshots. A full audit reads robots.txt, reports what it blocks and still visits those pages. A competitor snapshot always obeys robots.txt.

In nginx, this rule answers both names with a 403.

if ($http_user_agent ~* "(CrawlerX|FathomSpider)") {
    return 403;
}

In Cloudflare, create a custom rule with this expression and set the action to Block.

(http.user_agent contains "CrawlerX") or
(http.user_agent contains "FathomSpider")

CrawlerX records each URL refused with a 401 or 403 and does not retry it. It also skips the 7 requests it would send as other crawlers for any page where its own request got a 401, 403 or 429, so a rule that refuses CrawlerX on every path, robots.txt included, stops those too.

A user agent rule has one gap. The person running an audit can give the requests it makes outside the browser another user agent, such as Googlebot or desktop Chrome. To close that gap, email bot@crawlerx.ai with your domain.

To keep snapshots off your site, add this to robots.txt. It covers both names.

User-agent: CrawlerX
User-agent: FathomSpider
Disallow: /

To slow snapshots down, add a Crawl-delay in seconds. Values above 30 seconds count as 30.

User-agent: CrawlerX
User-agent: FathomSpider
Crawl-delay: 10

An audit ignores Crawl-delay. To slow one down or stop it, email bot@crawlerx.ai with your domain. By default an audit sends 1 request per second to your host, one URL at a time, unless the team has set another rate for your site. Pace and Errors and retries have the full rules.

Pages an audit requests

An audit starts at your home page. It follows <a href> links and sitemap entries on the same host or its www or bare-domain twin. It keeps the same scheme, apart from moving from http to https.

It sends GET requests with no cookies and this Accept header.

text/html,application/xhtml+xml;q=0.9,*/*;q=0.1

When a page links to a file of any type, such as a PDF or an image, CrawlerX requests it. It counts any response that is not text and discards it. CrawlerX reads up to 10 MiB of each response, follows at most 10 redirects and waits at most 30 seconds per request.

An audit stops at 5,000 pages unless the site's settings or the person running it set another limit. One run lasts at most 2 hours, then pauses and is resumed.

CrawlerX records rel="nofollow", meta robots and X-Robots-Tag and does not obey them. It still follows nofollowed links on your site.

Other files an audit requests

Along with your pages, an audit requests these files.

• robots.txt
• /sitemap_index.xml, /sitemap.xml, /wp-sitemap.xml and the sitemaps they list, up to 60 documents
• /llms.txt and /llms-full.txt
• your home page once more, with Accept: text/markdown
• https://mta-sts.yourdomain/.well-known/mta-sts.txt, once per crawl, when your domain publishes an MTA-STS record
• on WordPress, /wp-json/wp/v2/posts?slug= for pages that show no date, stopping at the first 401 or 403

After the page crawl, an audit may fetch the scripts your pages load a second time to read what they do. It may download up to 4 privacy policy documents. It reads PDFs for their text only and never runs them. It may download one logo image from your own site, up to 1 MiB.

Once or twice per audit, CrawlerX opens a TLS connection only to read your certificate and the HTTP version you offer. It sends no HTTP request on those connections.

Browser visits

After the page crawl, an audit opens every stored page that returned 2xx HTML, subdomains included, in headless Chromium. It renders each page at 1366 by 768 pixels and also measures it at 1280 by 720 and 390 by 844. Each page gets up to 45 seconds to load and 3 minutes in total. CrawlerX opens only pages it has already fetched, never links to other sites.

The browser loads your analytics and tracking scripts as any visitor's browser would. Browser visits and form tests can show up in your analytics under the CrawlerX user agent.

Requests sent as other crawlers

On audits only, CrawlerX runs a reachability check as its last step. It takes your home page and 3 other pages. For each page it sends one request as CrawlerX, then one request each as Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot and ClaudeBot, using each crawler's published user agent string. That is at most 32 requests per crawl.

Only the User-Agent header changes. No other header is added. The check records whether your server answers those crawlers differently from CrawlerX. CrawlerX sends these requests itself. They do not come from the companies that run those crawlers.

If the CrawlerX request for a page gets a 401, 403 or 429, the seven other requests for that page are not sent. So a server rule that refuses CrawlerX on every path, robots.txt included, also stops these requests. If CrawlerX can read a robots.txt that disallows it, it skips its own request for that page. It can still send the seven others, each checked against that crawler's own robots.txt rules, and a server rule that matches only CrawlerX does not stop them.

A page your robots.txt disallows for a given crawler is never requested as that crawler. If the audit stored no robots.txt, because the site has none or the audit was set not to download it, every crawler counts as allowed.

Each of these requests reads at most 2 MiB, waits its turn in the rate limiter and stays on your site's origin. CrawlerX does not follow a redirect to another origin.

robots.txt

A competitor snapshot always obeys robots.txt and matches the group for its own token. No setting changes that. It fetches nothing unless robots.txt returns a readable 2xx response. A 401, 403, 404, 5xx or timeout on robots.txt stops the snapshot.

By default an audit downloads robots.txt, reports what it would block and crawls those pages anyway. The person running it can set the audit to obey robots.txt, or not to download it at all. An audit that obeys robots.txt matches the rules for Googlebot, not CrawlerX, unless the person running it chooses the CrawlerX group. So a robots.txt rule naming CrawlerX does not stop a default audit. To turn audits away, refuse the CrawlerX user agent at your server or firewall.

When an audit that obeys robots.txt reads the file, CrawlerX treats each response as follows.

• A 404 or other 4xx, apart from 401, 403 and 429, means no rules. CrawlerX keeps that result for 24 hours.
• A 401 or 403 means no rules. CrawlerX keeps that result for 5 minutes.
• A 429, a 5xx, a timeout or a response that is not text means everything is disallowed. CrawlerX keeps that result for 5 minutes.

CrawlerX keeps parsed rules for at most 24 hours and reads robots.txt up to 512 KiB.

Crawl-delay applies only when robots.txt is obeyed, which every competitor snapshot does. CrawlerX reads the Crawl-delay in the matched group, or in the * group, and treats any value above 30 seconds as 30. A default audit ignores Crawl-delay and uses its configured rate.

Sitemaps, the logo and the WordPress date lookup check robots.txt only when the audit obeys it. Pages on third-party booking sites always obey that site's robots.txt.

Pace

By default CrawlerX sends 1 request per second to your host. The rate is set per site, from one request every 100 seconds up to 50 requests per second. A competitor snapshot never goes above 1 request per second.

Every request CrawlerX sends to your host waits its turn in a per-host rate limiter. That covers pages and each redirect, robots.txt, sitemaps, browser page loads, the form test's page loads, the requests sent as other crawlers, the logo, scripts and policy documents. The limiter cannot be switched off.

In the browser, the limiter meters the page load only. The page's own CSS, JavaScript, images and fonts are not metered, up to 500 of them per page, even when they sit on your host.

A crawl handles one URL at a time. Two crawls of the same site running in separate processes do not coordinate their pace.

Errors and retries

A 401 or 403 is recorded and not retried.

A 429 delays that URL only. CrawlerX waits for the Retry-After time if you send one. Without it, the wait starts at about 2 seconds and doubles each time, with some jitter, up to 5 minutes. CrawlerX gives up after 3 attempts, or at once if Retry-After asks for more than 5 minutes. A 429 does not slow the rest of the crawl.

A 5xx or a network error is retried with the same doubling wait, up to 3 attempts. CrawlerX does not read Retry-After on a 5xx.

CrawlerX never tries to solve a bot challenge page. It retries that URL with a wait that starts at 30 seconds and grows. After 5 challenge pages in a row, the crawl stops.

Competitor snapshots

Apart from robots.txt, a snapshot requests HTML pages only, with Accept: text/html,application/xhtml+xml. It drops any other response after reading its headers. It never queues the address of an image, script, stylesheet or file.

A snapshot reads at most 500 pages, 5 links deep, 2 MiB per response and 100 MiB per run. It runs no browser, no form test and no reachability check. For each page it keeps the address, the title, the kind of page and one sentence.

Form test

Every audit runs a consent and form test after its analysis. It uses your home page and one page with a lead form, trying contact and quote pages first. It opens them in three fresh headless browser sessions. One leaves the cookie banner alone, one clicks Accept and one clicks Reject.

The test types the form in the Accept and Reject sessions. It submits the form once, in the Accept session only. The limit today is one submission per crawl. Competitor snapshots never run the test.

The test types only the details in this table.

NNNN is a 4-digit number that changes with each test. The phone number sits in the 555-0100 to 555-0199 range, which is reserved for fiction. A required field the test has no value for stays empty. To filter these submissions, match an email address that starts with form-probe- and ends in @formibly.com.

Before the one submit click, the test blocks every other way the form could send itself. It holds back any request that carries the test data. It checks that every field holds exactly the dummy value, and any mismatch cancels the submit. The click is never retried. CrawlerX writes a record before clicking, so one crawl cannot submit twice. The test may tick a checkbox captcha. It never attempts an image puzzle.

The test also makes one request to Cloudflare's public trace endpoint to note which country it ran from.

Requests to other services

An audit also sends requests about your site to these outside services.

• Google's Chrome UX Report API
• a DNS-over-HTTPS resolver
• certificate transparency logs
• rdap.org
• Google's tag host, up to 6 times
• up to 25 vendor pages named in the script analysis
• up to 25 pages on third-party booking sites your pages link to, at least 2 seconds apart

Addresses it refuses

CrawlerX will not connect to a private, loopback, link-local or cloud metadata address. It checks the address after each DNS lookup and on each redirect. It uses only the scheme's default port and refuses a move from https down to http. The browser runs with Chromium's sandbox on.