Why is an AI crawler multiplying my hosting bill with no human visitors?

Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.

Category: AI / Agents · Trend: AI Infra · Opportunity score: 7.9 / 10

What is the “Why is an AI crawler multiplying my hosting bill with no human visitors?” problem in 2026?

Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.

Who has this problem?

Indie developers and small SaaS teams hosting public content on usage-billed platforms like Vercel.

Recorded source context

Dataset source note: Facebook’s crawler requested the same pages over 3 million times in 2 days

This note may summarize the referenced material rather than quote it verbatim. Source label: Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill. (primary source).

Existing players in this space

  • Cloudflare Bot Management: Strong bot scoring, but the poster still could not stop proxied scrapers with it.
  • Vercel Firewall and BotID: Platform bot controls that need tuning before the bill arrives, not after.
  • robots.txt: Ignored by the crawlers that cause the damage.
  • CAPTCHAs such as Turnstile and hCaptcha: Add friction for real users and were bypassed by account-creating bots in the thread.

What existing players are missing

A spend-aware bot guard that watches request patterns against the hosting bill in real time, separates crawlers from people and user-directed agents by behaviour rather than user agent, and caps or serves a cheap cached view before costs spike.

How Real Problem AI scores this opportunity

Aggregate score: 7.9 / 10. Four-axis rubric:

  • Problem severity: 9 / 10
  • AI feasibility today: 8 / 10
  • Market signal: 8 / 10
  • Competition gap: 6 / 10

How to build a solution: stack hints

  • Edge middleware request logging
  • Behavioural ML classification of traffic
  • Hosting cost API integration and alerts
  • Rate limiting and cheap crawler views

Related AI / Agents problems on Real Problem AI