Many websites block AI bots across the board. However, it is often unclear whether this is intended to prevent model training, visibility in an AI search, or even a page request explicitly initiated by the user.

As a result, the robots.txt file evolves from a technical detail into a strategic decision regarding data sovereignty, visibility, and digital distribution.

79 percent are blocking—but what exactly?

A BuzzStream analysis of 100 major news outlets published in January 2026 showed that 79 percent blocked at least one bot used for AI training. At the same time, 71 percent blocked at least one bot used for live search or retrieval.

These figures refer to major U.S. and U.K. news sites. They are not automatically applicable to every small and medium-sized enterprise (SME). However, they illustrate how often different objectives are conflated in a single decision to block access.

Training, indexing, and user retrieval serve different purposes

Providers are increasingly distinguishing between multiple crawler roles. Anthropic, for example, documents separate identifiers for training, search indexing, and user-triggered retrieval. The current classification can be found in the Anthropic Help Center.

• Training: Data can be processed to improve future models.
• Search and Indexing: Content becomes discoverable and can be included in current responses.
• User-initiated request: A bot accesses a page on behalf of a specific user.

Those who block everything across the board may be protecting content from being used for training, but at the same time they are sacrificing some of its visibility and usability in AI-powered search and assistant systems.

Four Questions to Ask Before Making a robots.txt Decision

1. What is our goal? Protection from training, protection from automated retrieval, or deliberate visibility?
2. What types of content are affected? Public articles require different rules than customer portals, paid content, or internal documents.
3. How is value created? Through reach, leads, subscriptions, licensing, or direct sales?
4. How do we monitor implementation? robots.txt is a published directive, not a strict access restriction.

robots.txt alone is not a security architecture

Cooperative bots follow the rules. Others may ignore them or log in using a fake identifier. Sensitive content should therefore be protected by robust access controls. In addition, server logs, rate limits, WAF rules, and bot management can help monitor and control actual access.

A more nuanced standard is usually better

For many corporate websites, a tiered strategy makes sense: technically protecting confidential and non-public content, restricting access based on specific objectives, but deliberately keeping public expertise accessible for search results and user-initiated access.

The question isn't simply "Should we allow or ban AI?" It is: Which bot is allowed to use what content for what purpose?

This post was inspired by my LinkedIn article from March 16, 2026.