# Bots and crawlers are hammering my website
Source: https://vpsdime.com/knowledgebase/troubleshooting/bot-traffic
Last reviewed: 2026-07-02
Summary: When scrapers, AI crawlers, and bad bots eat your CPU and traffic allowance, how to identify them in the access log, what robots.txt can and can't do, rate limiting in nginx, fail2ban for the abusive tail, and Cloudflare's bot tools as the heavy artillery.

## What this is

Your [traffic allowance](https://vpsdime.com/knowledgebase/acceptable-use/bandwidth-overage) is draining, or the [CPU graph climbed](https://vpsdime.com/knowledgebase/troubleshooting/slow-vps), and the visitors aren't human. Automated clients, search crawlers, **AI training and scraping bots**, SEO tools, vulnerability scanners, and plain scrapers, now make up a very large share of all web traffic, and a small site can receive more bot requests than human ones. Most are harmless at low volume; the problem is the aggressive tail. Here's how to see who's hitting you and how to push back proportionately.

## Step 1: identify the culprits in the access log

The access log has everything. Two summaries answer most questions:

```
# Top requesting IPs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Top user agents
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
```

(For an ongoing view, `goaccess /var/log/nginx/access.log` gives you a live terminal dashboard from the same data.) What you'll typically find: named crawlers (`Googlebot`, `bingbot`, `GPTBot`, `ClaudeBot`, `Amazonbot`, `Bytespider`, assorted `*-AI` agents), SEO crawlers (`AhrefsBot`, `SemrushBot`, `MJ12bot`), and the shadier end, blank or fake user agents from rotating IPs.

Decide what you actually mind: Googlebot is usually earning its keep; an AI scraper pulling your entire catalog every night on your bandwidth may not be.

## Step 2: robots.txt, the polite request

`robots.txt` is a **request, not a control**: legitimate crawlers honor it, abusive ones ignore it, so it's the right tool exactly for the named, well-behaved bots:

```
User-agent: GPTBot
Disallow: /

User-agent: AhrefsBot
Disallow: /
```

Also set `Crawl-delay` for bots that honor it, and make sure you're not wasting crawl budget on infinite URL spaces (calendars, faceted filters), those make even polite bots look like floods.

## Step 3: rate limiting, the enforced request

For clients that ignore robots.txt, make the server enforce a budget. nginx's built-in limiter, applied per IP:

```
# in http {}
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;

# in the server/location block
limit_req zone=perip burst=20 nodelay;
```

Humans browse well under 5 requests/second; scrapers slam into it and get 503s instead of your rendered pages. This alone converts most floods from a CPU problem into a log line. (Caddy and Apache have equivalents.)

## Step 4: fail2ban for the abusive tail

For IPs that keep hammering after being limited, ban them at the firewall: a fail2ban jail watching the access log for repeated 403/503s (or requests to paths like `/wp-login.php` and `/.env` that only scanners touch) turns the pattern into a temporary IP ban automatically. This pairs with, not replaces, the rate limit.

## Step 5: Cloudflare, the heavy artillery

If the traffic is distributed (thousands of IPs, fake agents), per-IP tools run out of road, and this is exactly what [Cloudflare's](https://www.cloudflare.com/) free tier is built for: **Bot Fight Mode**, managed challenges for suspicious clients, per-path WAF rules ("challenge everything hitting /api at >n/min"), and their **AI crawler blocking** toggle that handles the whole scraper category in one switch. Bonus: challenged bots never reach your VPS at all, so they stop costing you [traffic](https://vpsdime.com/knowledgebase/acceptable-use/bandwidth-overage) entirely, cached-and-challenged at the edge beats rate-limited at the origin. New to Cloudflare? The full setup, done right, is in [Cloudflare best practices](https://vpsdime.com/knowledgebase/getting-started/cloudflare-best-practices).

## Keep perspective

- Check the damage honestly on your [Graphs](https://vpsdime.com/knowledgebase/client-area/services/graphs): a bot wave that costs 2 GB and 3% CPU needs robots.txt, not an arms race.
- Bots hunting `/wp-login.php`, `/xmlrpc.php`, and `/.env` aren't a traffic problem but a [security probe](https://vpsdime.com/knowledgebase/getting-started/securing-your-vps), make sure those paths are locked or absent rather than just rate-limited.
- If what you're seeing isn't crawling but a flood meant to take you down, that's the [DDoS lane](https://vpsdime.com/knowledgebase/acceptable-use/ddos-policy).

## Still need help?

You can [open a support ticket](https://vpsdime.com/open-ticket/). So we can help on the first reply, it's worth mentioning:

- the VPS and the site being hammered,
- the top IPs or user agents from your access log,
- what you've already put in place (robots.txt, rate limiting, Cloudflare).

## Related questions

- "Bots are eating my bandwidth, how do I stop them?"
- "How do I see which bots are crawling my site?"
- "Does robots.txt actually block crawlers?"
- "How do I block AI crawlers like GPTBot?"
- "How do I rate limit requests in nginx?"
- "Should I use Cloudflare to stop bot traffic?"