AI Crawlers and robots.txt: What Law Firms Should Allow, Block, and Test

November 16, 2026
Posted by: Shawn

Managing AI crawlers with robots.txt starts with a business decision: which public pages should be available for search, which training uses should be restricted, and which user-directed retrievals need separate review? Those are different questions, and a blanket “block AI” switch can hide the tradeoffs.

For a law firm that wants AI search discovery while limiting selected training crawlers, a defensible approach is to allow intended search access to public marketing content, express training preferences through supported controls, and test the resulting policy through the whole delivery stack.

This guide provides a current bot reference, an annotated policy example, and a practical test plan. It does not recommend exposing client information or promise that allowing a crawler will produce citations.

Key Takeaways

  • Separate search crawling, model-training collection, and user-triggered fetching before choosing a rule.
  • OpenAI and Anthropic identify different bots for search and training. Google-Extended has a broader scope than training alone.
  • A specific crawler group does not simply inherit the restrictions in User-agent: *. Review the effective group and path rules.
  • robots.txt communicates crawler instructions. Authentication, authorization, and infrastructure controls protect private information.
  • Test the published file, representative URLs, and verified bot traffic. An accessible page is eligible for retrieval, not guaranteed a recommendation.

AI Bot Access Controls: Decide by Purpose

Documentation checked November 16, 2026. Provider names, controls, and rollout dates can change. The table summarizes the official sources linked in each row; it is not an exhaustive directory of every AI client.

Search, training, and user-directed access require separate decisions
Bot or control Documented purpose Decision for a visibility-focused law firm Important limit
OAI-SearchBot OpenAI: ChatGPT search discovery. Allow intended public pages if ChatGPT search inclusion is a goal. Access also depends on infrastructure; permission is not guaranteed selection.
GPTBot OpenAI: content that may support foundation-model training. Express a training opt-out if that is the firm’s policy. This is separate from OAI-SearchBot.
ChatGPT-User OpenAI: certain user-initiated visits. Assess user-directed access separately. OpenAI says robots.txt rules may not apply; this agent does not determine search inclusion.
Claude-SearchBot / ClaudeBot Anthropic: search relevance / potential training collection, respectively. Allow intended search access and decide training independently. A ClaudeBot training restriction is not a blanket instruction for every Claude client.
Claude-User Anthropic: user-requested retrieval. Review whether intended public retrieval should be available. Anthropic documents its own controls; do not assume another provider’s fetcher behaves identically.
PerplexityBot / Perplexity-User Perplexity: search crawling / user-requested fetching. Allow intended public search crawling; evaluate fetches separately. Perplexity says its user fetcher generally ignores robots.txt. PerplexityBot is not a foundation-model training crawler.
Googlebot Google: Google Search crawling. Keep intended public Search pages crawlable when Search visibility is a goal. AI Search inclusion has indexing and snippet eligibility requirements, not merely crawl permission.
Google-Extended Google: a product token covering specified Gemini training and grounding uses. Make a deliberate product-use choice. It is not a separate HTTP crawler identity and does not control Google Search inclusion or ranking.
Bingbot and Microsoft usage preferences Microsoft: crawl access and training/grounding preferences are different layers. Keep intended Bing search crawling separate from decisions about AI use. The announced new usage preferences begin taking effect February 1, 2027 as pages are crawled.

Start with the pages the firm actually wants people to discover: attorney profiles, practice pages, office information, and accurate public guides. Client portals, case materials, and private intake records need properly enforced access boundaries.

Can a Law Firm Block AI Training and Allow Search?

For providers with separate controls, yes. OpenAI explicitly describes allowing OAI-SearchBot while disallowing GPTBot. Anthropic also distinguishes its search and training bots in its crawler guidance.

That does not create one universal “no training, yes search” setting. Map the choice to each provider, product, and supported mechanism. A rule does not establish that previously collected information has been removed, nor does it guarantee that every requester complies.

Google-Extended is not a training-only switch

Google’s current documentation says Google-Extended covers specified training uses and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Blocking it can therefore affect those grounding uses as well.

Google Search, including its AI features, is a separate scope. Review Googlebot access and Search eligibility requirements instead of treating Google-Extended as an AI Overviews control.

Shared training signals are evolving

Anthropic says it began recognizing a shared machine-readable training preference on October 30, 2026, with rollout continuing. It states that an explicit ClaudeBot rule takes priority over the general signal for a path. Google says it intends to support the shared signal in early 2027. Do not mistake those different adoption timelines for universal current support.

Example robots.txt Policy: Public Search Access, Selected Training Restrictions

Illustration only. This example represents a firm choosing public access for three search crawlers while blocking two named training crawlers. The excluded path is fictional. Merge the intended policy with the site’s actual CMS rules, exceptions, and sitemap declarations rather than replacing a working file wholesale.

# Illustration only: merge with the existing site policy.
User-agent: *
Disallow: /crawl-excluded-example/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /crawl-excluded-example/
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

The example does three things:

  1. Wildcard group: communicates the illustrative path exclusion to otherwise unmatched compliant crawlers.
  2. Search group: applies the same public-page policy to the three listed agents. The excluded path is repeated explicitly.
  3. Training group: disallows all paths for the two listed training agents under this policy.

The example does not include an explicit Google-Extended group; review that product token’s grounding scope separately. It also makes no claim to control user-triggered fetchers through the search-bot group.

Why the repeated exclusion matters

Under the group-selection behavior described in the Robots Exclusion Protocol, a matching agent group takes precedence over the wildcard fallback. Google’s implementation documentation explicitly notes that specific-agent and wildcard groups are not combined.

If the search group contained only Allow: /, do not assume a restriction in the wildcard group would also apply. In this example, the longer excluded path governs that path despite the broader allowance.

The policy received 12 local parser checks covering public and excluded paths for the three search agents, Googlebot, and the two training agents. This is a limited syntax-and-decision check, not evidence of live crawler behavior or a provider conformance certification.

Expected decisions for the illustration, not observations of real bot visits
Agent /practice-area/ /crawl-excluded-example/page/
OAI-SearchBot Allowed Disallowed
Claude-SearchBot Allowed Disallowed
PerplexityBot Allowed Disallowed
Googlebot, through wildcard fallback Allowed Disallowed
GPTBot Disallowed Disallowed
ClaudeBot Disallowed Disallowed

Test AI Search Crawlability in Four Layers

1. Check the file actually served to the public

Save a dated copy before editing and identify whether WordPress, a plugin, the server, or the CDN supplies the response. Retrieve /robots.txt from the intended host and confirm that the response is the text file you expect, not an HTML error page or login screen.

The protocol scopes the file to the service authority. Test the relevant hosts and subdomains separately. An office microsite or portal on a different host may not share the main website’s policy.

2. Test effective rules for representative paths

Use a parser or tester with documented behavior and check the final merged file. Include a public service page, an attorney profile, a public PDF if relevant, an excluded path, and any existing CMS exception.

Record the agent token, exact path, selected group, matching rule, expected decision, and result. Inspect duplicate agent groups and conflicting rules. A successful test of the short illustration does not validate a larger production file assembled by several plugins.

3. Check the CDN, firewall, and page response

A robots allowance does not bypass a firewall, rate limit, authentication requirement, or browser challenge. Perplexity’s WAF guidance specifically pairs each bot’s identity with its published IP ranges.

For intended public access, check status codes and returned content. A 403 response suggests blocked access; 429 indicates rate limiting; server errors need investigation. Even a 200 response can contain a challenge or empty shell instead of the useful page.

A request made with a copied user-agent string is only a simulation of your response path. It does not authenticate the requester as that provider. Use current provider verification guidance and published ranges where available before creating targeted exceptions. Preserve protection for private applications.

4. Observe real requests and downstream evidence

Record actual requests with timestamp, path, response status, verified requester information where available, and the policy version. Allow time for documented refresh behavior rather than interpreting an immediate test as final.

Keep four observations separate: the policy permits crawling, infrastructure delivers content, a system retrieves or indexes it, and an answer visibly cites it. None automatically proves the next stage. For citation observations, see our Bing AI Performance walkthrough and repeatable AI share of voice protocol.

Microsoft’s Upcoming Changes: Check the Dates Before Copying Settings

As of November 16, 2026, Microsoft’s October announcement describes new training and grounding preferences that begin taking effect on February 1, 2027 as pages are crawled. Training preferences can be expressed in robots.txt or at page level; the new grounding preference is page-level only.

Its transition guidance includes temporary grounding preferences opening December 15, 2026 for qualifying sites using NoArchive or NoCache as of that date, and a March 1, 2027 action deadline for maintaining those restrictions. Consult the official transition FAQ for the conditions and required steps.

Do not present this schedule as functionality already effective in November, or assume a generic robots.txt instruction implements Microsoft’s future page-level grounding choices. Usage preferences also do not override a crawl block.

Common Mistakes in Law-Firm AI Bot Policies

  • Blocking every bot to avoid training: this can prevent desired search discovery as well. Decide by supported purpose.
  • Adding an unrestricted named group: a wildcard exclusion may no longer govern that crawler. Test the effective policy.
  • Treating robots.txt as security: the RFC explicitly distinguishes it from access authorization. Use enforced controls for client materials.
  • Assuming a crawl block is a deindexing tool: Google’s robots.txt guidance explains that a blocked URL can still appear in Search. Indexing controls are a separate decision.
  • Trusting a bot name without verification: a log label alone does not establish requester identity.
  • Using llms.txt to solve an access problem: a curated text index is not a substitute for crawler controls. See our llms.txt evidence guide.

Review the policy after changes to hosting, security plugins, bot-management services, or provider documentation. Assign an owner, preserve the earlier version, and repeat the relevant path tests after a change.

Make AI Search Access a Measured Part of Your Visibility Plan

Start with the public content your firm wants discovered, document the training and retrieval choices, and verify that the published rules match those decisions. Then measure actual mentions, citations, and client outcomes separately.

Explore AI Search Visibility for Law Firms from Best Law Firm Ads to discuss how access, content quality, and reporting fit your practice and market. A useful plan should explain its evidence and tradeoffs without promising that a bot allowance guarantees a recommendation.

Frequently Asked Questions

Can a law firm block AI training and still allow AI search?

For services offering separate controls, yes. Choose the relevant search and training settings independently. Review the provider’s scope rather than treating one rule as a universal instruction.

Does blocking GPTBot block ChatGPT search?

OpenAI documents GPTBot and OAI-SearchBot as separate controls. Review the search bot’s own access and the delivery infrastructure rather than assuming the training rule covers it.

Should we block Google-Extended?

That depends on the firm’s choices about the specified Gemini training and grounding uses. It is not a Google Search ranking or inclusion switch, and its scope is not limited to training.

Do user-triggered fetchers always obey robots.txt?

No universal rule should be assumed. OpenAI says its user-triggered agent’s rules may not apply, while Perplexity says its user fetcher generally ignores them. Review each provider’s documentation and enforce private access separately.

Can robots.txt protect confidential client documents?

No. It communicates instructions to compliant crawlers. Client documents need appropriate authentication and authorization, with access tested independently of crawler preferences.

Why can a bot still receive a 403 after we allow it?

The allowance and the server’s delivery decision are separate layers. Check firewall events, bot verification, rate limits, authentication, and the exact URL before changing a broader security policy.

Does allowing AI crawlers guarantee citations?

No. Access is one prerequisite for intended retrieval. It does not demonstrate indexing, answer selection, endorsement, or a new client inquiry.

Is the sample a complete robots.txt file for every WordPress site?

No. It illustrates a limited policy and passed local parser checks. A production file must preserve the site’s actual rules and be tested with its infrastructure and relevant provider behavior.

Table Of Contents

Related Post