How to actually block AI bots from scraping your website

OpenAI , Google, Microsoft and Anthropic have agreed to support a new way for website owners to stop their content being used to train AI models.

<![CDATA[ <article> <p><a href="https://www.techradar.com/uk/ai-platforms-assistants/openai">OpenAI</a>, Google, Microsoft and <a href="https://www.techradar.com/pro/anthropic-wants-to-use-claude-ai-to-help-discover-new-lifesaving-drugs-says-the-best-way-to-identify-and-work-through-these-bottlenecks-is-to-get-deeply-involved-in-real-drug-programs">Anthropic</a> have agreed to support a new way for website owners to stop their content being used to train AI models. The companies are backing a shared opt-out signal in a website’s robots.txt file, which tells AI crawlers whether they're allowed to use its content for training. </p><p>Like a modern day Library of Alexandria, the internet is the single largest source of human knowledge that exists. Countless people have spent their lives cataloguing and uploading the entire history of humanity for the betterment of everyone.</p><p>AI companies are a tinderbox in this regard. To train and develop better capabilities for their models, AI companies are repeatedly scraping every corner of the internet to create archives of training data. Compounded by changes in search and how people seek information, AI threatens to become the single point of access to information.</p><p>But just because you have a website, it doesn’t mean you want it constantly scraped by bots stealing your answers to life’s difficult questions so people can access them elsewhere. </p><p>Luckily, you’re not alone. Outrage from users and website hosts across the internet has been fueled into developing methods to keep AI companies from harvesting data from your websites.</p><p>Like layering slices of Swiss cheese, these methods work best to keep out scrapers when used in a defense-in-depth manner. But some malicious bots will still find ways to navigate through the holes. Here’s a few ways to keep the honest bots at bay:</p><figure class="van-image-figure inline-layout" data-bordeaux-image-check="" ><div class='image-full-width-wrapper'><div class='image-widthsetter' style="max-width:1920px;"><p class="vanilla-image-block" style="padding-top:56.25%;"><img id="djX7joLwsdetjvNWwzvmjE" name="Real-time web scraping Listing v2" alt="A man on a laptop, close on the hands hovering over the keys" src="https://cdn.mos.cms.futurecdn.net/djX7joLwsdetjvNWwzvmjE-1920-80.png" mos="" align="middle" fullscreen="" width="1920" height="1080" attribution="" endorsement="" class="inline" /></p></div></div><figcaption itemprop="caption description" class=" inline-layout"><span class="credit" itemprop="copyrightHolder">(Image credit: Shutterstock)</span></figcaption></figure><h2 id="1-robots-txt">1. Robots.txt</h2><p>This is the crux of <a href="https://openai.com/index/helping-websites-express-ai-training-preferences/?trk=public_post_comment-text&amp;utm_source=chatgpt.com" target="_blank">OpenAI’s method to honor anti-web scraping markers</a>. The ‘robots.txt’ text file sits in the root directory of your website and provides guidance on how bots should behave. This can be a useful way to provide a digital map to search engine crawlers and can also prevent your site’s servers from becoming overburdened by bot requests.</p><p>But by adding just a few characters to this file, you can block the majority of bot traffic from accessing your website. Of course, this unfortunately only applies to scrapers that actually abide by the rules of your robots.txt file.</p><p>OpenAI has set something of a precedent for other AI companies here, adding that, "We encourage other AI providers to honor the signal."</p><p>There are a few ways to manage which specific bots you would like to allow on your website, but to block AI traffic you can take the following steps:</p><ul><li>Open your robots.txt file, or create one if you don’t have one using a text editor</li><li>Add a section to the file using the heading ‘<strong># AI crawlers</strong>’</li><li>To block all AI crawlers, add a line below stating ‘<strong>Disallow: /</strong>’</li><li>To block a specific AI crawler, such as ChatGPT, place ‘<strong>User-agent: ChatGPT-User</strong>’ on one line, with ‘<strong>Disallow: /</strong>’ underneath</li><li>You can prevent AI crawlers from accessing specific pages by adding the web address after the ‘/’. For example, I could stop AI crawlers from accessing my author page by using the rule ‘<strong>Disallow: /author/benedict-collins</strong>’</li><li>Update your robots.txt file in the source files of your website</li></ul><h2 id="2-user-only-viewing">2. User-only viewing</h2><p>Alternatively, you could take the <a href="https://www.techradar.com/computing/social-media/hard-disagree-reddits-mandatory-log-in-shift-sparks-backlash-as-users-worry-itll-go-the-way-of-pinterest">planned Reddit approach to blocking AI traffic</a> by requiring all users to have an account in order to access your website. This is an excellent way to filter out human traffic from bot traffic because user accounts usually need to be verified with an email or phone number - something mostly humans have.</p><p>But this comes with its own caveats, and Reddit is finding out the hard way in feedback from users. Not requiring an account gives users from all over the web an easier search experience. They can find the information they need without any obstacles, and without handing over personal information to yet another website.</p><p>Having to log in or create an account to view web content can cause people to ‘bounce’ - they click on your website, see they need to create an account, and immediately back out of your site to seek the information they need elsewhere. So tread carefully if your traffic matters to you.</p><h2 id="3-ip-limiting">3. IP limiting</h2><p>AI bots may descend en-masse using a single IP address to access your website. Third-party services offer this as an option to prevent your website’s servers from being bombarded with quick-moving traffic, but it can come with a monetary cost.</p><p>This method can be used to set up a timer on specific IP addresses that are detected as bot traffic. For example, if a single IP is sending multiple access requests, you can adjust the settings so that only 5 requests from the IP can be made in a 10 second period.</p><p>But beware that some AI bots or web scrapers will use rotating IPs, hopping between them to avoid a block from a single IP.</p><h2 id="4-captchas">4. Captchas</h2><p>A slightly archaic method of human verification that most people have a love/hate relationship with. I don’t mind counting the number of buses or bicycles, or aligning the correct jigsaw piece with its place, but some people see it as a humiliating ritual to prove their humanity.</p><p>What’s worse is that some AI models are getting good enough to solve captchas themselves.</p><p>It can help to filter out relentless attempts from web scrapers and less capable AI bots, but it could also cause users to bounce - especially those with a particular fear of ClickFix attacks who may see a complicated captcha as a way to hijack their device.</p><p>So use these methods cautiously for they is a double-edge sword. You can stop your data from being harvested by AI bots but it may also harm the experience of your users.</p> </article> ]]>