A meaningful share of the traffic reaching any public website is not people. It is software: search engine crawlers building their index, feed readers checking for updates, uptime monitors, link preview fetchers, scrapers collecting content, security scanners probing for weaknesses, and lately a growing wave of AI crawlers gathering material for training and for answering questions. None of this is new, and most of it is not sinister. Automated traffic has been part of the web since the beginning, and some of it is genuinely useful to you.
So the question is not how to get rid of bots. It is how to make sure the automated visitors your site wants are served well, the ones it does not want are kept within polite limits, and neither group ever slows the site down for an actual person. That sorting work is something we handle at the platform level for every site we host, and this article explains how we think about it.
Not all bots are the same
It helps to sort automated visitors into rough groups. The first group is the software you actively want: search engine crawlers that keep your pages in the index, the fetchers that build link previews when someone shares your page in a chat or on social media, and the monitoring probes that check your site is up. Blocking these would hurt you directly, so they need to be recognized and let through.
The second group ranges from neutral to unwelcome: scrapers that copy content, commercial tools crawling the whole web to sell reports about it, bulk downloaders, and the endless background hum of scripts probing every public address for known weaknesses. Most of this traffic is not dangerous to a well-maintained site, but it consumes resources and gives nothing back. And in between sits a newer category: AI crawlers. Reasonable site owners disagree about how welcome those are, which is exactly why they deserve their own controls rather than a blanket yes or no. We will come back to them.
Why this is a performance topic, not just a security one
Server capacity is finite, and every request costs something. A page served straight from cache is cheap. A search results page, a filtered listing, or a calendar that can be paged into the distant future is not, because each variation has to be built fresh. Crawlers do not browse the way people do. They follow every link, including combinations no human would ever click through, and they can request pages far faster than any person reads.
The practical result is that an enthusiastic crawler can spend your server capacity on pages nobody will ever see, while an actual visitor waits behind it. Nothing is broken, nothing is hacked, and yet the site feels slower. That is the outcome we care about preventing. Automated traffic is fine as long as it never competes with a person for resources.
Caching absorbs a good part of this on a healthy site, and we invest in it heavily, but caching alone is not the answer. The most expensive pages are often precisely the ones that cannot be cached, and a crawler that hammers them walks straight past every layer of protection a fast cache provides. Capacity planning that ignores automated traffic ends up paying for hardware that mostly serves software.
How we tell people and software apart
The sorting happens at the edge of our platform, before a request ever reaches the site itself. Several signals feed into it. Honest bots declare themselves, and the important ones can be verified, so a request claiming to be a search engine crawler is checked against the network it actually came from. Impersonating a well-known crawler is a common trick, and verification quietly takes it off the table.
Another strong signal is where the traffic comes from. Real visitors browse from home connections, office networks, and mobile carriers. Requests arriving from cloud datacenters are almost always software, because people do not browse the web from a rented server. We maintain that classification at the platform level and label datacenter traffic accordingly. On top of that come behavioral patterns: the pace of requests, the spread of pages touched, whether the client behaves like a browser at all. No single signal decides anything on its own. Together they give a reliable picture of who is a person and what is a program.
Rate limits, not walls
What we do with that picture matters just as much as the picture itself. Our default posture is not to block automated traffic but to keep it within limits. Verified search crawlers keep their access, because being indexed is part of why the site exists. Unidentified automation and datacenter traffic get a budget: a pace of requests that is perfectly adequate for legitimate software but stops any single client from monopolizing the server.
The limits are sized so that a real person never encounters them. Someone reading your site, even quickly, even with many tabs open, stays far below the thresholds. Only clients behaving like machines are asked to slow down, and well-built software handles that politely, backing off and retrying later. The site stays fully open to the world. The world just cannot stampede through it.
Where AI crawlers fit
AI crawlers are the group we get asked about most. They are not attacks, and many of them identify themselves honestly and respect the signals sites publish. At the same time, they can be heavy consumers of both content and capacity, and site owners hold genuinely different views on whether appearing in AI training data or AI-generated answers is good for their business. Both views are defensible, and the right answer depends on what the site is for.
Our position is that this is a call for the site owner, not for us. The technical side, recognizing these crawlers and keeping their request rate reasonable, we handle for every site as standard. The policy side, whether a particular crawler should be allowed at all, is a conversation we are happy to have per site, and the controls exist to enforce whatever gets decided.
What this means for a site we host
Nothing to install and nothing to configure. Traffic sorting and rate limiting run at the edge for every site on our platform, tuned on real traffic rather than assumptions, and adjusted as patterns change. Search engines keep crawling, link previews keep working, monitoring keeps monitoring, and your pages keep arriving quickly for the people they were written for.
What you notice is mostly what does not happen. The site does not slow down when a scraper discovers it. A burst of automated requests does not turn into a slow afternoon for your visitors. Bots are a fact of the internet, and they are not going away. Kept in their lane, they are also not a problem worth losing sleep over.