AI / Agents AI Infra

Cite as: Real Problem AI problem “Why is an AI crawler multiplying my hosting bill with no human visitors?”. Opportunity score 7.9 out of 10 (severity 9, AI feasibility 8, market signal 8, competition gap 6). Category AI / Agents. Trend AI Infra. Source signal: Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill.. Canonical URL: https://www.realproblem.ai/idea/why-is-an-ai-crawler-multiplying-my-hosting-bill-with-no-human-visitors.

Why is an AI crawler multiplying my hosting bill with no human visitors?

Small sites on usage-billed hosting are seeing crawlers hit the same pages millions of times, routed through residential proxies and even creating accounts, and the first sign is a hosting bill ten times higher than normal that the host will not waive.

Who has it: Indie developers and small SaaS teams hosting public content on usage-billed platforms like Vercel.

Heard in public

“Facebook’s crawler requested the same pages over 3 million times in 2 days”

Quoted word for word from the public post linked below. Nobody submitted it to Real Problem AI.

Ask HN thread on out-of-control AI bot traffic, 14 September 2026, plus a separate HN post two days later on a Meta crawler causing a 10x Vercel bill.

What nothing does yet

A spend-aware bot guard that watches request patterns against the hosting bill in real time, separates crawlers from people and user-directed agents by behaviour rather than user agent, and caps or serves a cheap cached view before costs spike.

Tools that solve this

4 named so far, and where each one stops.

  • Cloudflare Bot Management

    Strong bot scoring, but the poster still could not stop proxied scrapers with it.

  • Vercel Firewall and BotID

    Platform bot controls that need tuning before the bill arrives, not after.

  • robots.txt

    Ignored by the crawlers that cause the damage.

  • CAPTCHAs such as Turnstile and hCaptcha

    Add friction for real users and were bypassed by account-creating bots in the thread.

How you might build it

  1. 01Edge middleware request logging
  2. 02Behavioural ML classification of traffic
  3. 03Hosting cost API integration and alerts
  4. 04Rate limiting and cheap crawler views

Founders working in this area

Not affiliated with Real Problem AI.

#N29G · Canonical URL: https://www.realproblem.ai/idea/why-is-an-ai-crawler-multiplying-my-hosting-bill-with-no-human-visitors