Who runs CrawlerX

What it is for

SearchActions builds and runs CrawlerX as part of its website analytics product. CrawlerX audits websites. An audit fetches a site's pages, then opens each one that returned HTML with a 2xx status in a headless browser. For your home page and 3 other pages, it also checks whether your server answers search engines and AI crawlers differently. SearchActions audits the sites it works on. The team may also audit public sites it does not own.

CrawlerX also takes competitor snapshots. A snapshot reads up to 500 of a site's public HTML pages to see what the site offers. It always obeys robots.txt and never goes above 1 request per second. It keeps each page's address, title, kind of page and one sentence.

How a crawl starts

A member of the team starts each crawl by hand from a command line. No dashboard or API can start one. CrawlerX crawls only the sites the team has added to its list.

Contact

Email bot@crawlerx.ai. Include your domain and, if you can, the time and request line from your logs. To block CrawlerX yourself, see How to recognize and block it.