In their never-ending search for content to feed into their Large Language Models, ‘AI’ companies are using distributed networks of bots (automated programs) to download the contents of web sites everywhere on the Internet. Many sites are being overwhelmed by this traffic. Some site owners are being forced to upgrade their hosting service, while others have simply shut down.
If your response to that is to point out that Google does the exact same thing, and has for years, you’d be right, except for a few important differences:
- Google’s searchbot doesn’t try to download every page on every web site. It’s much more selective about what it grabs.
- Googlebot clearly identifies itself, which makes it much easier to classify its requests and limit its impact. The vast majority of ‘AI’ scraper bots falsely identify themselves as ordinary web users.
- Google publishes a list of host IP ranges used by Googlebot, and they consist of specific, limited subnets that are clearly owned by Google. The vast majority of ‘AI’ scraper bots are spread around the world, on largely unrelated groups of hosts, often running on virtual private servers owned by VPS providers.
- When Googlebot crawls your site, one or two hosts read a small batch of related/linked pages, then go away and come back for more later. ‘AI’ crawlers spread their requests across hundreds or even thousands of hosts, each one reading as little as one page, which makes identifying and blocking them difficult.
All of these things make the job of managing traffic for a web site much more difficult than it was before the current ‘AI’ boom.
In my work on managing traffic for clients, I have been lucky in that two of them are not legally allowed to sell to anyone outside of Canada. For those clients, I eventually just blocked traffic from every country except Canada. Traffic from major search providers is always allowed, and special procedures are required for certificate updates, but this made a huge difference. Web sites that need traffic from other countries obviously can’t use this drastic approach.
Thankfully, all of this pointless and damaging traffic will stop when the current ‘AI’ bubble bursts, and that can’t come soon enough for me.
Here’s Drew Curtis, the owner/operator of the long running humour site fark.com, talking about this very problem.
boot13
I’ve enjoyed playing around with generative AI tools like ChatGPT and the new, generative features of image editors. I used ChatGPT to generate most of the
Security professionals have been saying it for years. The specific dangers associated with email have changed during that time, but the advice remains valid, and applies to everyone who uses email.

The folks at Microsoft are once again demonstrating their creativity in thinking up ways to trick people into using its web search service, Bing.


In the small dialog that appears, enter a name for the new search (I call it ‘Google Classic’), the URL 