Arbé Digital

Article

robots.txt Beyond the Basics: What Marketers and Site Owners Should Actually Put in It

Most robots.txt files are still written for 1996. Here is a practical tour of what the file is for, the parts most sites get wrong, and the modern directives—AI bots, Content Signals, sitemap pointers, host hints—that make it genuinely useful in 2026.

Arbé Digital
  • robots.txt
  • SEO
  • AI agents
  • crawl control
  • content strategy
robots.txt Beyond the Basics: What Marketers and Site Owners Should Actually Put in It

If your site has been online for more than five minutes, you almost certainly have a robots.txt. There is a decent chance nobody has looked at it since launch. And there is an even better chance that what is in there was copy-pasted from a tutorial written before TikTok existed.

That used to be fine. It is not fine anymore.

The file went from “a polite note to Googlebot” to the single most-read public document on your site for non-human clients—search crawlers, AI training bots, agent frameworks, monitoring tools, archive services, and increasingly the autonomous agents people use to research and buy. Every one of them looks at robots.txt first. Most of them are not Googlebot.

This is a guide to what actually belongs in that file in 2026, what is safe to skip, and where teams quietly leak SEO equity, training data, or money by leaving it on autopilot.

What robots.txt actually is (and what it is not)

robots.txt is a plain-text file at the root of your domain (https://example.com/robots.txt) that follows the Robots Exclusion Protocol, formalized as RFC 9309 in 2022. It tells well-behaved automated clients which paths they may or may not request.

Two things it is not:

  1. It is not security. Disallowing /admin does not protect /admin. It just publishes the URL to anyone reading the file. Use authentication for protection; use robots.txt for guidance.
  2. It is not authoritative for indexing. A crawler that respects robots.txt will not fetch a disallowed URL, but search engines can still index that URL based on external links, showing it as a bare result without a snippet. If you want to keep something out of search results, use the noindex directive—either via a meta tag on the page or an X-Robots-Tag HTTP header. (The noindex directive in robots.txt itself was never standardized, and Google stopped honoring it in 2019.)

Get those two facts straight and 80% of common mistakes evaporate.

The mistakes we see most often

Before adding anything new, audit what is already there. In dozens of agency takeovers, the same handful of issues come up again and again.

1. Blocking CSS, JS, or image directories

A surprising number of sites still ship something like:

User-agent: *
Disallow: /wp-content/
Disallow: /assets/

That used to be common advice. Today it actively hurts you—Google needs to fetch your CSS and JS to render the page and judge mobile-friendliness, layout shift, and Core Web Vitals. Blocking those directories tells the crawler “render my site without styles or interactivity, then rank it.” Don’t.

2. Disallowing the entire site by accident

User-agent: *
Disallow: /

This one line, often left over from a staging environment, has tanked more launches than we can count. It belongs on staging.example.com, never on production. Any pre-launch checklist worth its salt verifies the production robots.txt after DNS cuts over.

3. Treating it like a security file

If something is sensitive, do not list it here. Listing it is broadcasting it. Use server-level auth, IP allow-listing, or genuine access control. Then, if you still want crawlers to skip it, you can add a Disallow—but the protection comes from the auth, not the file.

4. Wildcards that don’t do what you think

Disallow: /*?* is a common attempt to block all URLs with query strings. Most well-known crawlers honor * and $ extensions to RFC 9309, but support is uneven across the long tail. If you care about a specific parameter (say, faceted-search URLs blowing up your crawl budget), use Disallow: /search? and verify with each engine’s tester rather than assuming * works everywhere.

5. Forgetting it exists when the site changes

robots.txt is treated as a launch artifact, not a living document. New blog section, new app subdirectory, new headless API, new AI policy—none of it gets reflected. The file should be reviewed on the same cadence as your sitemap.

What useful, modern robots.txt files include

Once the basics are clean, here is what is genuinely worth adding.

Sitemap references (the easy win nobody does)

Every search engine and most agent frameworks will look for sitemap hints in robots.txt:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Use absolute URLs, list every sitemap (you can have several), and keep them current. Crawlers that have never seen your domain often discover content this way before they ever crawl your homepage. The line is not tied to any User-agent block—put it at the top or bottom of the file.

Explicit rules for AI crawlers, not just search engines

Search bots are no longer the dominant non-human traffic on most sites. AI training and agent crawlers now make up a meaningful share, and their user agents are well-documented. If your brand has an opinion about AI training, say so:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Allow agent-time fetches that answer user questions
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

A few notes on this pattern:

  • Training vs. retrieval is a real distinction. GPTBot is OpenAI’s training crawler; OAI-SearchBot is the one that fetches pages when ChatGPT is answering a user’s question right now. Most publishers want very different policies for those two cases.
  • Be specific, not symbolic. User-agent: AI does nothing; the protocol matches by exact agent token. Maintain a real list, and treat it like dependency hygiene—review it quarterly.
  • Don’t block everything reflexively. If your business depends on being cited or recommended by AI assistants (most service businesses do, even if they don’t know it yet), blocking retrieval bots is the equivalent of blocking Google in 2008.

Content Signals: declaring what AI may do with your content

A newer convention—pushed hard by Cloudflare and discussed on Cloudflare Radar’s AI Insights—lets you express not just whether bots may crawl, but what they may do with the result. The format is still emerging, but the shape looks like:

User-agent: *
Content-Signal: search=yes, ai-train=no, ai-input=yes

That maps to: “you may include this in a search index, you may not use it to train a model, you may use it as live context for an AI answer.” As of early 2026, only about 4% of sites had adopted Content Signals, which means there is a real differentiation window for brands that care about how their content is used downstream.

A Host directive when you have one canonical hostname

Some crawlers (notably Yandex, and a few smaller engines) honor:

Host: example.com

It tells them which hostname is canonical when you have multiple. Most sites should be solving this with 301 redirects and rel="canonical" instead, but if you serve the same content under several domains for legitimate reasons, the directive is harmless and sometimes helpful.

Comments—yes, really

# 2026-05 — disallow internal search to protect crawl budget
Disallow: /search?

Comments cost nothing and save the next person (often a future version of you) from undoing a deliberate decision. If a Disallow line has a reason, write it down. If it does not have a reason, ask why it is there.

Where robots.txt ends and other files begin

robots.txt is one of several discovery files. The mistake is trying to make it do everything.

  • sitemap.xml — Tells crawlers what exists. Reference it from robots.txt; do not try to inline URLs here.
  • X-Robots-Tag HTTP headers — Apply noindex, nofollow, or nosnippet to non-HTML resources (PDFs, images, RSS feeds) where a meta tag isn’t possible.
  • <meta name="robots"> — Page-level indexing control for HTML. Always wins over robots.txt for indexing decisions, because it is read after the page is fetched.
  • security.txt — A separate /.well-known/security.txt file for vulnerability disclosure contacts. Do not stuff this into robots.txt.
  • humans.txt — A retro convention listing the team behind the site. Charming, optional, and unrelated to crawl behavior.
  • llms.txt — A newer proposal (https://llmstxt.org/) for a curated, AI-friendly summary of your site at /llms.txt. Adoption is still small, but it pairs well with a thoughtful robots.txt policy if you want LLMs to see a clean version of your content.
  • /.well-known/agent-skills.json, MCP server cards, API catalogs — The agent-readiness layer. Out of scope for robots.txt, but increasingly part of the same conversation. We covered the broader picture in our piece on Cloudflare’s Agent Readiness score.

The mental model: robots.txt is the front door policy. The other files are signs hung inside the building.

A practical template to start from

Drop this into https://yoursite.com/robots.txt, edit the comments and bot list to match your situation, and you will already be ahead of most of the web:

# robots.txt for example.com
# Last reviewed: 2026-05-03
# Owner: marketing@example.com

# ── Default policy: be crawlable ────────────────────────────
User-agent: *
Disallow: /search?
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /

# ── AI training crawlers (decide your stance, then state it) ─
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# ── AI retrieval / agent crawlers (usually allow) ───────────
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# ── Sitemaps ────────────────────────────────────────────────
Sitemap: https://example.com/sitemap.xml

Tweak the disallow list to your stack. Tweak the AI policy to your brand’s appetite. Re-read the comments in six months.

How to validate it

Three quick checks before you ship:

  1. Fetch it from production. curl -i https://yoursite.com/robots.txt. Confirm a 200, a Content-Type: text/plain, and that the body matches what you committed.
  2. Run it through Google Search Console’s robots.txt report. Catches syntax errors and shows how Googlebot interprets each rule.
  3. Test specific URLs. Pick five real URLs (a marketing page, a product page, a blog post, a tag archive, a search result), and confirm each one resolves to allow/disallow as expected.

Then add a quarterly calendar reminder. Files that nobody reviews drift into being wrong.

Closing thought

robots.txt is the cheapest, highest-leverage piece of public infrastructure on a website. A few minutes of attention removes embarrassing default rules, expresses a clear stance on AI use, points crawlers at your sitemaps, and signals to anyone reading—human, search bot, or agent—that the site is run by someone who is paying attention.

If yours has not been touched since launch, that is the entire post: open it, read it, fix it, comment it, and put a date at the top. Future you will thank present you.

Further reading