Wappalyzer crawler

Wappalyzer Pty Ltd operates WappalyzerBot to retrieve public webpages and identify the technologies they use. The crawler looks at public signals such as HTML, scripts, response headers, cookies, and page metadata.

Crawls keep Wappalyzer's technology dataset current and can also run when someone requests a website technology lookup through a Wappalyzer product. Both workloads use the same bounded crawler and identify themselves in the same way.

Identification

WappalyzerBot sends the following user agent:

WappalyzerBot/1.0 (+https://www.wappalyzer.com/)

Signed WappalyzerBot requests use HTTP Message Signatures. The public keys needed to verify those signatures are available in our signature directory.

Crawl controls

Each crawl has limits on the number of URLs, link depth, and total running time. Requests for linked pages are delayed, and a site-specific crawl delay can reduce the rate further.

Signed crawls check robots.txt before fetching a page and follow rules for WappalyzerBot, including Allow, Disallow, and Crawl-delay. A signed crawl does not proceed when those rules cannot be evaluated or the requested delay cannot fit within its time limit.

To block WappalyzerBot, add this to your robots.txt file:

User-agent: WappalyzerBot
Disallow: /

Questions or problems

If you have a question about WappalyzerBot or need to report unexpected crawler traffic, email [email protected]. Please include the affected hostname and an example request timestamp so we can investigate.