Robots.txt Definition

Robots.txt is a plain text file placed at the root of a website (https://www.example.com/robots.txt) that tells crawlers which parts of the site they may or may not visit.

It controls crawling, not indexing. A URL blocked in robots.txt is not downloaded, but it can still appear in search results if other pages link to it. To keep a page out of Google, use noindex instead.

robots.txt definition

What Is Robots.txt?

Robots.txt implements the Robots Exclusion Protocol, created in 1994 and standardized as RFC 9309 in 2022. Before crawling a site, a well-behaved bot requests this file and applies the rules written for it.

A typical file:

User-agent: *
Disallow: /cart/
Disallow: /search/
Allow: /

Sitemap: https://www.example.com/sitemap_index.xml
  • User-agent: the crawler the group applies to (* means all).
  • Disallow: a path the crawler should not request.
  • Allow: an exception inside a disallowed path.
  • Sitemap: the address of the XML sitemap, read by all major engines.

The rules are public and voluntary. Search engines follow them; malicious bots may not. Robots.txt is not a security tool.

📌 In brief
Robots.txt definition: what the file does, Disallow and Allow syntax, examples for WordPress and AI crawlers, limits, mistakes and how to test it.
On the menu:
  • How Robots.txt Works
  • Why Robots.txt Matters for SEO
  • Robots.txt vs Noindex
  • Robots.txt Examples
  • Best Practices
By SeoYass, SEO consultant since 2010 · Updated on

How Robots.txt Works

When a crawler arrives on a host, it first fetches /robots.txt. Then:

  1. It picks the most specific User-agent group that matches its name (for example Googlebot before *).
  2. For each URL, it compares the path with the rules. At Google, the longest matching rule wins; if an Allow and a Disallow have the same length, the less restrictive one applies.
  3. Paths are case-sensitive. Google supports two wildcards: * (any sequence) and $ (end of URL).

Each host and protocol has its own file: blog.example.com and http:// versions need their own robots.txt. Google caches the file, generally for up to 24 hours, and reads up to 500 KiB of it.

The server response matters too. If robots.txt returns a 404, Google crawls everything. If it returns a server error (5xx) for a long period, Google stops or slows crawling, because it cannot know what is allowed. It is one of the steps in how a search engine crawls a site.

Why Robots.txt Matters for SEO

  • Crawl efficiency: on large sites, blocking endless filter combinations, internal search pages or calendar URLs keeps crawlers on pages that matter.
  • Server load: aggressive bots on heavy sections can slow a site; robots.txt reduces the requests from compliant crawlers.
  • Sitemap discovery: the Sitemap line is an easy way for every engine to find your sitemap.
  • AI crawler control: many AI companies publish crawler names that respect robots.txt, which lets site owners choose who may collect their content.

The risk is equal to the benefit: one wrong line such as Disallow: / can stop Google from crawling the entire site.

Robots.txt vs Noindex

Goal Right tool
Stop crawlers from wasting time on a section Robots.txt Disallow
Keep a page out of search results Noindex, with the page left crawlable
Consolidate duplicate URLs Canonical tag or 301 redirect
Protect private content Password or login, never robots.txt alone

The classic trap: blocking a page in robots.txt and adding noindex. The crawler never downloads the page, never sees the noindex, and the URL can stay indexed without a description. Google also stopped supporting noindex inside robots.txt in 2019.

Robots.txt Examples

1. WordPress

A light setup is usually best on WordPress. Do not block CSS, JavaScript or /wp-content/: Google needs them to render pages.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

2. Blocking parameter URLs with wildcards

User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$

3. AI crawlers

Each AI company names its crawlers. For example GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity). Google-Extended is a separate token that controls whether Google may use your content for its Gemini models; it does not change Google Search crawling or ranking.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Blocking AI crawlers is a business choice: it protects content, but it can also reduce your visibility in AI answers. Check each vendor’s documentation, since names and purposes change.

Best Practices

  • Keep it short: block only what really wastes crawl or creates infinite URLs.
  • Never block resources needed for rendering (CSS, JS, images used in the layout).
  • Never use robots.txt to hide a page from results: use noindex or a password.
  • Add the Sitemap line with an absolute URL.
  • Test before publishing: one character can change the meaning of a rule (Disallow: / vs Disallow:).
  • Keep the file reachable: it must return 200, or 404 if you want no rules, never a server error.
  • Document changes and review the file after each migration or redesign.

Common Robots.txt Mistakes

  • Disallow: / copied from the staging site to production.
  • Blocking /wp-content/ or JavaScript folders, so Google renders broken pages.
  • Trying to deindex pages by blocking them, which leaves them indexed without content.
  • Relying on Crawl-delay for Google: Google ignores it (Bing supports it).
  • Forgetting that rules are case-sensitive: /Admin/ is not /admin/.
  • Listing private folders in robots.txt, which publicly reveals their location.

Robots.txt is always one of the first files reviewed in a technical SEO audit.

How to Test Robots.txt

  • Open the file at /robots.txt and read every rule.
  • robots.txt report in Google Search Console (Settings): it shows the files Google found, when it last read them and any parsing errors.
  • URL Inspection: tells you if a specific URL is blocked by robots.txt.
  • A crawler such as Screaming Frog can crawl the site while respecting or ignoring the file, which shows exactly which URLs are blocked.

Advantages and Limits of Robots.txt

Advantages

  • Simple, standard and read by all major crawlers.
  • Effective to save crawl resources on large sites.
  • The easiest way to declare a sitemap and to address AI crawlers.

Limits

  • Does not remove pages from the index.
  • Not respected by malicious bots, and not a protection for private data.
  • Blocks link discovery: links on blocked pages are not followed.
  • A single error can have site-wide consequences.

Robots.txt: Key Takeaways

Robots.txt tells crawlers where they may go. It is a crawl management tool, not an indexing or security tool. Keep it minimal, never block rendering resources, declare your sitemap, decide deliberately about AI crawlers, and use noindex or passwords when the goal is to keep content out of search results.

From theory to a crawlable site

Knowing the robots.txt syntax is the first step. The next is checking that your file blocks only what it should, and nothing that earns you traffic.

  • An action plan adapted to your situation
  • Clear explanations, no needless jargon
  • Direct application to your sector
Robots.txtDefinitionCustomers ↑
// RESULTS & REFERENCES

From diagnosis to results, step by step

Here is how I handle robots.txt issues on client sites, and what my clients say about my work.

+350COMPLETED PROJECTS
+320SATISFIED CLIENTS
5.0 ★VERIFIED GOOGLE REVIEWS
+17YEARS OF EXPERIENCE
001REFERENCES / VERIFIED REVIEWS

What Is Robots.txt?

Robots.txt implements the Robots Exclusion Protocol, created in 1994 and standardized as RFC 9309 in 2022. Before crawling a site, a well-behaved bot requests this file…

EXPERTISEMETHODFIELD WORK
11ACTIVE PROJECTS
5.0GOOGLE RATING
READ THE REFERENCES →
002PORTFOLIO / PROJECTS

How Robots.txt Works

When a crawler arrives on a host, it first fetches /robots.txt. Then:

EXPERTISEMETHODFIELD WORK
6-12MONTHS FOR SOLID RESULTS
+350COMPLETED PROJECTS
VIEW THE PORTFOLIO →
003METHOD / EXPERTISE

Why Robots.txt Matters for SEO

Crawl efficiency: on large sites, blocking endless filter combinations, internal search pages or calendar URLs keeps crawlers on pages that matter.Server load:…

EXPERTISEMETHODFIELD WORK
100%POSITIVE REVIEWS 2026
11ACTIVE PROJECTS
SEE MY SEO SERVICES →
WANT TO BE THE NEXT CASE?Tell me about your project and I will reply with a first diagnosis.
SEND A BRIEFFREE MINI-AUDIT

FAQ: Robots.txt

What is robots.txt used for?

It tells crawlers which parts of a website they may or may not request. It manages crawling and declares the sitemap location.

Does robots.txt remove a page from Google?

No. A blocked URL can stay indexed if other pages link to it. To remove a page from results, let it be crawled and use noindex.

Where should the robots.txt file be placed?

At the root of each host and protocol, for example https://www.example.com/robots.txt. Subdomains need their own file.

Does Google support Crawl-delay?

No. Google ignores the Crawl-delay rule. Bing and some other crawlers support it.

Can I block AI crawlers with robots.txt?

Yes, for crawlers that respect it, such as GPTBot, ClaudeBot or PerplexityBot, by adding a Disallow rule for their user agent. Google-Extended controls Gemini training use without affecting Google Search.

What happens if my robots.txt returns an error?

A 404 means no rules, so everything can be crawled. A server error (5xx) that lasts can make Google stop or slow crawling of the site.

Do you want to work with me?

SeoYass, SEO consultant17+ years of SEO
Your SEO consultant
SeoYass

I am SeoYass, an SEO expert with over 17 years of experience. I have worked with small and medium-sized businesses, successfully completing more than 350 projects.

Passionate about local SEO, I continue to learn and refine my strategies to deliver the best performance for my clients.

350+Projects
5.0 ★Google reviews
17+Years of experience