Why is an AI crawler multiplying my hosting bill with no human visitors?
Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.
Category: AI / Agents · Trend: AI Infra · Opportunity score: 7.9 / 10
What is the “Why is an AI crawler multiplying my hosting bill with no human visitors?” problem in 2026?
Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.
Who has this problem?
Indie developers and small SaaS teams hosting public content on usage-billed platforms like Vercel.
Recorded source context
Dataset source note: Facebook’s crawler requested the same pages over 3 million times in 2 days
This note may summarize the referenced material rather than quote it verbatim. Source label: Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill. (primary source).
Existing players in this space
- Cloudflare Bot Management: Strong bot scoring, but the poster still could not stop proxied scrapers with it.
- Vercel Firewall and BotID: Platform bot controls that need tuning before the bill arrives, not after.
- robots.txt: Ignored by the crawlers that cause the damage.
- CAPTCHAs such as Turnstile and hCaptcha: Add friction for real users and were bypassed by account-creating bots in the thread.
What existing players are missing
A spend-aware bot guard that watches request patterns against the hosting bill in real time, separates crawlers from people and user-directed agents by behaviour rather than user agent, and caps or serves a cheap cached view before costs spike.
How Real Problem AI scores this opportunity
Aggregate score: 7.9 / 10. Four-axis rubric:
- Problem severity: 9 / 10
- AI feasibility today: 8 / 10
- Market signal: 8 / 10
- Competition gap: 6 / 10
How to build a solution: stack hints
- Edge middleware request logging
- Behavioural ML classification of traffic
- Hosting cost API integration and alerts
- Rate limiting and cheap crawler views
Related AI / Agents problems on Real Problem AI
- My AI agent repeated one broken action all night, and I only saw the cost in the morning. (9.0/10)
- Someone opened a function to fix one bug and it was 300 lines nobody wrote by hand. (9.0/10)
- Why do I have to trust a benchmark score published by the company selling the model? (8.5/10)
- Why does reviewing AI-written code take longer than writing it would have? (8.3/10)
- Why did my coding agent sign a contract from my inbox without asking? (8.1/10)