Module 6Free
AI crawlers
From your server access logs: which bots crawl which page types, how often, and what errors they hit. Upload a log from nginx or Apache, or a Vercel or Cloudflare log export.
Sample: 30 days of logs (Vercel log export)
Bots are identified by user agent. Google, Bing and Apple are also verified by reverse DNS, so fake ones are left out.
AI bot requests
10,934
last 30 days
Building pages seen by AI
29%
71% never fetched by any AI bot
Errors served to AI bots
1.6%
mostly 429 rate limits to ClaudeBot
| Bot | Type | /buildings/{slug} | /neighborhoods/{slug} | /landlords/{slug} | /blog/{slug} | Errors | Last seen |
|---|---|---|---|---|---|---|---|
| OAI-SearchBot | retrieval | 4,210 | 880 | 190 | 120 | 1.2% | 2 min ago |
| ChatGPT-User | retrieval | 1,120 | 640 | 40 | 210 | 0.4% | 11 min ago |
| PerplexityBot | retrieval | 0 | 0 | 0 | 0 | — | Blocked by robots.txt |
| GPTBot | training | 0 | 0 | 0 | 0 | — | Blocked by robots.txt |
| ClaudeBot | training | 2,890 | 310 | 260 | 64 | 3.1% | 1 hr ago |
| Googlebot | search | 18,400 | 2,210 | 1,430 | 410 | 0.6% | just now |
| Bingbot | search | 3,020 | 640 | 280 | 120 | 0.9% | 4 min ago |
Share of each template's pages fetched by an AI bot (30 days)
- /buildings/{slug}29%
- /neighborhoods/{slug}92%
- /landlords/{slug}41%
- /blog/{slug}100%