A significant share of traffic to most sites is automated. Search crawlers you want. Uptime monitors you configured. Beyond that sit scrapers, vulnerability scanners, credential-stuffing tools and content harvesters — and they cost real money in compute and bandwidth.
Classify before you block
Blocking indiscriminately removes you from search results or breaks legitimate integrations. Verify well-behaved crawlers by reverse DNS rather than trusting the user agent string, which anyone can set. Identify what your automated traffic actually is before deciding what to do about it.
Protect what is expensive
- Search, export, report generation and anything calling a paid API deserve stricter limits than static pages.
- Authentication endpoints need their own thresholds — this is where credential stuffing lands.
- Forms that send email or create records need low limits, because abuse there has an external cost.
- Cache aggressively; a cached response served to a scraper costs almost nothing.
robots.txt is not security
It is a request that well-behaved crawlers honour. Hostile automation reads it as a map of what you consider sensitive. Never rely on it to protect anything, and never list paths there that you would not want found.
Reading your own logs
Credential stuffing looks like many failed sign-ins across many accounts from a modest number of addresses, often at a steady rate. Scraping looks like sequential access through identifier ranges. Both are visible in ordinary access logs if you look. Alert on rate-limit rejections spiking — that is usually the first sign.
Proportion
Aggressive bot defence adds friction for real users and creates accessibility problems. For most small sites, sensible rate limits, caching, and multi-factor authentication on accounts achieve nearly all of the benefit without a challenge page in front of every visitor.