Robots.txt Definition
Robots.txt is a plain text file placed at the root of a website (https://www.example.com/robots.txt) that tells crawlers which parts of the site they may or may not visit.
It controls crawling, not indexing. A URL blocked in robots.txt is not downloaded, but it can still appear in search results if other pages link to it. To keep a page out of Google, use noindex instead.
What Is Robots.txt?
Robots.txt implements the Robots Exclusion Protocol, created in 1994 and standardized as RFC 9309 in 2022. Before crawling a site, a well-behaved bot requests this file and applies the rules written for it.
A typical file:
User-agent: * Disallow: /cart/ Disallow: /search/ Allow: / Sitemap: https://www.example.com/sitemap_index.xml
- User-agent: the crawler the group applies to (
*means all). - Disallow: a path the crawler should not request.
- Allow: an exception inside a disallowed path.
- Sitemap: the address of the XML sitemap, read by all major engines.
The rules are public and voluntary. Search engines follow them; malicious bots may not. Robots.txt is not a security tool.
- How Robots.txt Works
- Why Robots.txt Matters for SEO
- Robots.txt vs Noindex
- Robots.txt Examples
- Best Practices
How Robots.txt Works
When a crawler arrives on a host, it first fetches /robots.txt. Then:
- It picks the most specific User-agent group that matches its name (for example
Googlebotbefore*). - For each URL, it compares the path with the rules. At Google, the longest matching rule wins; if an Allow and a Disallow have the same length, the less restrictive one applies.
- Paths are case-sensitive. Google supports two wildcards:
*(any sequence) and$(end of URL).
Each host and protocol has its own file: blog.example.com and http:// versions need their own robots.txt. Google caches the file, generally for up to 24 hours, and reads up to 500 KiB of it.
The server response matters too. If robots.txt returns a 404, Google crawls everything. If it returns a server error (5xx) for a long period, Google stops or slows crawling, because it cannot know what is allowed. It is one of the steps in how a search engine crawls a site.
Why Robots.txt Matters for SEO
- Crawl efficiency: on large sites, blocking endless filter combinations, internal search pages or calendar URLs keeps crawlers on pages that matter.
- Server load: aggressive bots on heavy sections can slow a site; robots.txt reduces the requests from compliant crawlers.
- Sitemap discovery: the Sitemap line is an easy way for every engine to find your sitemap.
- AI crawler control: many AI companies publish crawler names that respect robots.txt, which lets site owners choose who may collect their content.
The risk is equal to the benefit: one wrong line such as Disallow: / can stop Google from crawling the entire site.
Robots.txt vs Noindex
| Goal | Right tool |
|---|---|
| Stop crawlers from wasting time on a section | Robots.txt Disallow |
| Keep a page out of search results | Noindex, with the page left crawlable |
| Consolidate duplicate URLs | Canonical tag or 301 redirect |
| Protect private content | Password or login, never robots.txt alone |
The classic trap: blocking a page in robots.txt and adding noindex. The crawler never downloads the page, never sees the noindex, and the URL can stay indexed without a description. Google also stopped supporting noindex inside robots.txt in 2019.
Robots.txt Examples
1. WordPress
A light setup is usually best on WordPress. Do not block CSS, JavaScript or /wp-content/: Google needs them to render pages.
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.example.com/sitemap_index.xml
2. Blocking parameter URLs with wildcards
User-agent: * Disallow: /*?sort= Disallow: /*.pdf$
3. AI crawlers
Each AI company names its crawlers. For example GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity). Google-Extended is a separate token that controls whether Google may use your content for its Gemini models; it does not change Google Search crawling or ranking.
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /
Blocking AI crawlers is a business choice: it protects content, but it can also reduce your visibility in AI answers. Check each vendor’s documentation, since names and purposes change.
Best Practices
- Keep it short: block only what really wastes crawl or creates infinite URLs.
- Never block resources needed for rendering (CSS, JS, images used in the layout).
- Never use robots.txt to hide a page from results: use noindex or a password.
- Add the Sitemap line with an absolute URL.
- Test before publishing: one character can change the meaning of a rule (
Disallow: /vsDisallow:). - Keep the file reachable: it must return 200, or 404 if you want no rules, never a server error.
- Document changes and review the file after each migration or redesign.
Common Robots.txt Mistakes
Disallow: /copied from the staging site to production.- Blocking
/wp-content/or JavaScript folders, so Google renders broken pages. - Trying to deindex pages by blocking them, which leaves them indexed without content.
- Relying on
Crawl-delayfor Google: Google ignores it (Bing supports it). - Forgetting that rules are case-sensitive:
/Admin/is not/admin/. - Listing private folders in robots.txt, which publicly reveals their location.
Robots.txt is always one of the first files reviewed in a technical SEO audit.
How to Test Robots.txt
- Open the file at
/robots.txtand read every rule. - robots.txt report in Google Search Console (Settings): it shows the files Google found, when it last read them and any parsing errors.
- URL Inspection: tells you if a specific URL is blocked by robots.txt.
- A crawler such as Screaming Frog can crawl the site while respecting or ignoring the file, which shows exactly which URLs are blocked.
Advantages and Limits of Robots.txt
Advantages
- Simple, standard and read by all major crawlers.
- Effective to save crawl resources on large sites.
- The easiest way to declare a sitemap and to address AI crawlers.
Limits
- Does not remove pages from the index.
- Not respected by malicious bots, and not a protection for private data.
- Blocks link discovery: links on blocked pages are not followed.
- A single error can have site-wide consequences.
Robots.txt: Key Takeaways
Robots.txt tells crawlers where they may go. It is a crawl management tool, not an indexing or security tool. Keep it minimal, never block rendering resources, declare your sitemap, decide deliberately about AI crawlers, and use noindex or passwords when the goal is to keep content out of search results.
From theory to a crawlable site
Knowing the robots.txt syntax is the first step. The next is checking that your file blocks only what it should, and nothing that earns you traffic.
- An action plan adapted to your situation
- Clear explanations, no needless jargon
- Direct application to your sector
From diagnosis to results, step by step
Here is how I handle robots.txt issues on client sites, and what my clients say about my work.
What Is Robots.txt?
Robots.txt implements the Robots Exclusion Protocol, created in 1994 and standardized as RFC 9309 in 2022. Before crawling a site, a well-behaved bot requests this file…
How Robots.txt Works
When a crawler arrives on a host, it first fetches /robots.txt. Then:
Why Robots.txt Matters for SEO
Crawl efficiency: on large sites, blocking endless filter combinations, internal search pages or calendar URLs keeps crawlers on pages that matter.Server load:…
FAQ: Robots.txt
What is robots.txt used for?
It tells crawlers which parts of a website they may or may not request. It manages crawling and declares the sitemap location.
Does robots.txt remove a page from Google?
No. A blocked URL can stay indexed if other pages link to it. To remove a page from results, let it be crawled and use noindex.
Where should the robots.txt file be placed?
At the root of each host and protocol, for example https://www.example.com/robots.txt. Subdomains need their own file.
Does Google support Crawl-delay?
No. Google ignores the Crawl-delay rule. Bing and some other crawlers support it.
Can I block AI crawlers with robots.txt?
Yes, for crawlers that respect it, such as GPTBot, ClaudeBot or PerplexityBot, by adding a Disallow rule for their user agent. Google-Extended controls Gemini training use without affecting Google Search.
What happens if my robots.txt returns an error?
A 404 means no rules, so everything can be crawled. A server error (5xx) that lasts can make Google stop or slow crawling of the site.
Do you want to work with me?
I am SeoYass, an SEO expert with over 17 years of experience. I have worked with small and medium-sized businesses, successfully completing more than 350 projects.
Passionate about local SEO, I continue to learn and refine my strategies to deliver the best performance for my clients.

