S&P 500 Robots.txt and LLMs.txt Analysis & Research

Independent Landscape & Competitive AI Adoption Crawler Policy Research

Overview & Questions

The Robots Exclusion Protocol (REP), standardized under IETF RFC 9309, provides a framework for website operators to communicate crawling preferences to automated agents through robots.txt directives. These publicly accessible instructions specify which resources compliant crawlers may access but do not enforce security, prevent indexing, or govern subsequent use of collected content.

As AI-powered search, retrieval, and model training expand, these directives provide an observable signal of how organizations approach AI discoverability, content governance, and intellectual property protection.

This independent research examines enterprise adoption patterns, crawler-specific policies, and emerging machine-readable AI discovery conventions to identify competitive trends, governance risks, and strategic opportunities.

• Which companies and industries are adopting AI-specific crawler policies?
• Which AI crawlers are most frequently allowed, restricted, or explicitly blocked?
• How does AI crawler governance vary across competitors and industry sectors?
• Are organizations prioritizing AI discoverability or intellectual property protection?
• Which companies are adopting emerging AI discovery files such as llms.txt?
• How do AI-specific restrictions compare with traditional search engine crawler policies?
• What governance gaps, misconfigurations, and security risks are emerging?
• How are enterprise AI policies evolving, and what strategic actions should leadership prioritize?

Research Scope & Methodology

Research Scope: This study analyzes publicly accessible robots.txt directives, AI discovery files, and sitemap configurations across a defined enterprise sample. Findings represent observed website policies at the time of collection, not verified crawler compliance, AI search visibility, or organizational AI adoption.

Research Methodology: A Python collector requested a defined set of public, machine-readable files from each host: robots.txt, llms.txt, llms-full.txt, sitemap.xml, sitemap_index.xml, and sitemaps declared in robots.txt. Each file was requested once, with a delay between requests. If robots.txt was not recognized on the first request, the collector retried once at the www address. Responses were saved and classified, then analyzed offline from the saved files. Collection took place on October 26, 2026.

Urgency & Recent Developments

“An AI agent visits a company website, checks a legitimate llms.txt machine-readable file, and installs malware. It follows the official instructions that point to non-existent packages. Security researchers have found hundreds of inadvertent mistakes – hackers are already leveraging them, and tricking AI agents into self-compromise.” Source: Link
— Cybernews

Today (10/30/2026), we’re announcing our plans to support a shared AI training control in robots.txt. This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). This builds on our long history of providing website owners with clear controls, including through Google-Extended, the AI training control we’ve offered since 2023. Source: Link
— Google

S&P 500 AI Crawler Governance: Key Findings

An analysis of 498 S&P 500 hosts reveals that explicit AI crawler governance remains relatively uncommon. Although most hosts provide a readable robots.txt, comparatively few name AI-specific crawlers, and adoption of the emerging llms.txt convention remains limited.

Research at a Glance

Key Findings

1. Most websites do not explicitly address AI crawlers.

Of the 438 hosts with a recognized robots.txt, 376 (85.8%) do not name any of the nine AI bots examined. Individual bots are explicitly named by only 20–51 hosts.

Generic wildcard permissions are considerably more common: 62–73 hosts permit each bot through a wildcard rule, compared with 3–27 hosts that explicitly name and allow a particular bot.

This distinction matters. Access permitted by a wildcard or the absence of a disallow rule is not equivalent to an explicit, bot-specific Allow directive. Effective access and declared intent should be analyzed separately.

2. Blocking patterns follow crawler purpose.

Training and dataset crawlers face substantially more restrictions than AI search or user-triggered agents.

• Training and dataset agents: 13–16 root-level blocks per bot across five agents.
• Search and user-triggered agents: 2–5 root-level blocks per bot across four agents.

Thirteen hosts block GPTBot while leaving OAI-SearchBot accessible. Similarly, 12 hosts block ClaudeBot while leaving Claude-SearchBot accessible. No hosts demonstrate the reverse pattern.

These findings suggest that some organizations distinguish between AI model training and search-related discovery, although the relatively small sample of explicit policies limits broader conclusions.

3. Crawler treatment differs by vendor.

Among hosts that explicitly name individual crawlers, major AI vendors’ bots are more frequently allowed than blocked:

CCBot and Bytespider are the only two crawlers in this comparison blocked more frequently than allowed.

These counts describe observed crawler policies rather than actual crawl activity or compliance.

4. llms.txt adoption is smaller than raw retrieval counts suggest.

Although 115 hosts returned a file at the expected llms.txt location, not all responses represented usable Markdown documents.

• 74 hosts (14.9%) returned content beginning with the required Markdown H1.
• 7 additional hosts appeared to provide intentional llms.txt content but lacked that heading.
• 31 responses were HTML pages rather than valid llms.txt documents.
• 6 hosts had a recognized llms-full.txt, compared with 41 initially retrieved responses.

This highlights the importance of validating file contents rather than treating HTTP success as evidence of adoption.

5. robots.txt retrieval coverage is higher than initially reported.

A recognized robots.txt was retrieved from 438 of 498 hosts (88.0%).

An earlier figure of 74.11% reflected request-level success rather than unique-host coverage. Ninety-three hosts required a second fetch after an unsuccessful initial attempt.

Host-level deduplication provides a more accurate measure of coverage.

6. Missing or inaccessible files create uncertainty.

Sixty hosts did not return a recognized robots.txt:

The 26 inaccessible or errored hosts represent 5.2% of the 498-host sample. Their crawler policies cannot be reliably classified from the available evidence.

Research Implications

The strongest observable pattern is not widespread AI crawler blocking, but limited explicit governance combined with selective restrictions on training-oriented crawlers.

Organizations increasingly have the technical ability to differentiate AI training, search discovery, and user-triggered retrieval. However, the relatively low frequency of explicit bot-specific rules indicates that these distinctions are not yet consistently reflected in publicly observable crawler policies.

The findings also reinforce an important methodological principle: explicit permission, wildcard-derived access, implicit permission, explicit blocking, and unknown retrieval status must remain distinct categories when interpreting AI crawler adoption and governance.