Cite as: Real Problem AI problem “Why is an AI crawler multiplying my hosting bill with no human visitors?”. Opportunity score 7.9 out of 10 (severity 9, AI feasibility 8, market signal 8, competition gap 6). Category AI / Agents. Trend AI Infra. Source signal: Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill.. Canonical URL: https://www.realproblem.ai/idea/why-is-an-ai-crawler-multiplying-my-hosting-bill-with-no-human-visitors.
Why is an AI crawler multiplying my hosting bill with no human visitors?
Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.
Who has it: Indie developers and small SaaS teams hosting public content on usage-billed platforms like Vercel.
- Severity9/10
- Feasible today8/10
- Market signal8/10
- Competition gap6/10
Heard in public
“Facebook’s crawler requested the same pages over 3 million times in 2 days”
Quoted word for word from the public post linked below. Nobody submitted it to Real Problem AI.
Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill.What nothing does yet
A spend-aware bot guard that watches request patterns against the hosting bill in real time, separates crawlers from people and user-directed agents by behaviour rather than user agent, and caps or serves a cheap cached view before costs spike.
Tools that solve this
4 named so far, and where each one stops.
- Cloudflare Bot Management
Strong bot scoring, but the poster still could not stop proxied scrapers with it.
- Vercel Firewall and BotID
Platform bot controls that need tuning before the bill arrives, not after.
- robots.txt
Ignored by the crawlers that cause the damage.
- CAPTCHAs such as Turnstile and hCaptcha
Add friction for real users and were bypassed by account-creating bots in the thread.
How you might build it
- 01Edge middleware request logging
- 02Behavioural ML classification of traffic
- 03Hosting cost API integration and alerts
- 04Rate limiting and cheap crawler views
#N29G · Canonical URL: https://www.realproblem.ai/idea/why-is-an-ai-crawler-multiplying-my-hosting-bill-with-no-human-visitors