Robots.txt Generator
Create crawl rules for search engines and AI crawlers.
Input
About This Tool
Generate a robots.txt file with sitemap, crawl-delay, disallow rules, and optional AI crawler controls. Everything runs locally in your browser: nothing you paste is uploaded, stored, or logged, and there is no sign-up or usage limit. Read the full guide below for worked examples, best practices, and common mistakes to avoid.
Frequently Asked Questions
Does robots.txt guarantee pages will not appear in search?
No. Robots.txt controls crawling, not indexing. Use noindex meta tags when you need a page removed from search results.
Should I block AI crawlers?
It depends on your content strategy. Some sites allow discovery but restrict training-oriented crawlers. Review each crawler before publishing rules.
Where should robots.txt be placed?
Upload it to the root of your domain, such as https://example.com/robots.txt.
How to create a robots.txt file that helps crawlers, not hurts you
A robots.txt file is a plain text file at the root of your site that tells crawlers which URLs they may request. Enter your domain, list the paths to block, and the generator writes the file for you. Upload it so it loads at https://yourdomain.com/robots.txt.
The file controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Google's own documentation says robots.txt is not a way to keep a page out of Google; for that, use a noindex rule or password protection.
Step by step: using the generator
- Type your domain, such as
https://example.com. The tool strips trailing slashes and uses it to write aSitemap:line. - List folders or URL paths to block, one per line, such as
/wp-admin/or/cart/. A missing leading slash is added for you. - Leave crawl delay at 0 unless a particular crawler needs it. See the note below.
- Tick Add AI crawler controls if you want separate rules that block six named AI-related bots.
- Copy the output, save it as
robots.txt(plain text, UTF-8), and upload it to your site root.
What the generated file looks like
Inputs: domain https://example.com, disallow /wp-admin/ and /cart/, crawl delay 0, AI controls off.
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlWith no blocked paths, the tool writes Allow: / instead, which means everything may be crawled. Tick the AI box and six more groups are appended, each with Disallow: /.
The tool assumes your sitemap lives at /sitemap.xml. If your site uses another address, such as /sitemap_index.xml in many WordPress SEO plugins, edit that line before uploading. A sitemap line pointing to a 404 does no good.
The syntax, line by line
| Line | Meaning |
|---|---|
User-agent: * | Start of a group that applies to all crawlers without a more specific group |
User-agent: GPTBot | Start of a group for one named crawler |
Disallow: /private/ | Do not crawl URLs beginning with this path |
Allow: /private/public.html | Exception to a broader Disallow |
Disallow: (empty) | Nothing is blocked |
Disallow: / | Block the whole site for that crawler |
Sitemap: https://… | Absolute URL of a sitemap; not tied to a user-agent group |
Paths are case sensitive: /Private/ and /private/ are different. Google also supports * as a wildcard and $ as an end-of-URL marker, so Disallow: /*.pdf$ blocks PDF files. The generator does not build wildcards, but you can add them by hand. The details are in Google's guide to creating a robots.txt file.
When two rules both match a URL, Google follows the more specific one, meaning the longer path. If they tie in length, the less restrictive rule wins.
What to block, and what to leave alone
Good candidates
- Admin and login areas such as
/wp-admin/(keep/wp-admin/admin-ajax.phpreachable on WordPress sites that need it) - Cart, checkout, and account pages
- Internal search results such as
/?s=or/search/ - Faceted filter URLs that create endless combinations
- Staging or test folders (better still, put these behind a password)
Do not block
- CSS and JavaScript files. Google needs them to render the page like a user sees it. Blocking them can hurt how the page is understood.
- Pages you want in search. This sounds obvious, but a stray
Disallow: /left over from a staging site is a classic launch-day disaster. - Pages you have marked
noindex. Crawlers must be able to fetch the page to see the tag. If you block it, the noindex is never read.
The AI crawler option explained
Ticking the AI box adds separate groups that disallow these user-agents: GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, and PerplexityBot. They do different jobs:
GPTBot(OpenAI),ClaudeBot(Anthropic), andCCBot(Common Crawl) collect content that can be used for model training or datasets.ChatGPT-UserandPerplexityBotare tied to assistants that fetch or surface pages when answering questions, so blocking them can reduce how often you are cited.Google-Extendedis a control token for whether Google uses your content for certain generative AI products. Google states it does not affect your ranking or inclusion in Google Search.
It is all-or-nothing in this tool. The checkbox blocks all six. If you want to be cited by AI assistants but not trained on, delete the lines for the assistants you want, and keep the others. Crawler names change, so confirm current names in each company's own documentation before relying on the list.
Robots.txt is a request that well-behaved crawlers honour. It is not access control. A scraper that ignores the rules will not be stopped by it.
Crawl-delay: mostly a myth for Google
The crawl delay field writes Crawl-delay: N. Some search engines honour it. Google does not support the directive, so it will have no effect on Googlebot. If your server is struggling under crawl load, fix the capacity or use the crawl-rate tools in Google Search Console. Leave the field at 0 unless you know a specific crawler needs it.
Upload, test, and monitor
- Save the file as
robots.txt, lowercase, UTF-8 plain text. Google reads only the first 500 KiB, which is far more than any normal file needs. - Place it in the top-level directory of the host. A file at
/blog/robots.txtis ignored, and each subdomain needs its own. - Open
https://yourdomain.com/robots.txtin a browser and confirm it loads as text. - Check the robots.txt report in Google Search Console for fetch errors and warnings.
- After launch, review the file whenever you redesign, migrate, or add a store. If you use WordPress, an SEO plugin may already serve a virtual file; make sure two sources do not conflict.
If you also manage page-level search snippets, see the meta description checker. Anyone building the page itself can preview markup in the HTML live preview.
Examples for common site types
A WordPress blog
User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlThis keeps crawlers out of the dashboard and internal search results, but lets them reach the AJAX endpoint that some themes and plugins use on the front end. The tool's output has a single Disallow list, so add the Allow line and adjust the sitemap address by hand.
An online store
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/Carts, checkouts, and account pages have no search value and generate endless private URLs. Product and category pages stay open.
A site under construction
User-agent: *
Disallow: /This asks all compliant crawlers to stay away. It is fine for a short build period, but remove it at launch, and add a password too, since the file does not stop anyone from visiting.
Common mistakes
- Blocking the whole site by leaving
Disallow: /from staging. - Trying to hide private pages with robots.txt. The file is public, and listing a path advertises it.
- Using a relative sitemap URL. It must be absolute.
- Forgetting that rules apply per host and protocol.
wwwand non-wwware separate. - Expecting a rule to remove an already indexed page. Use a removal tool or noindex.
More questions
How long until Google sees my changes?
Google generally caches robots.txt for up to a day, so updates apply soon but not instantly. You can request a recrawl of the file in Search Console.
Can I have more than one Sitemap line?
Yes. Add one line per sitemap, each with a full URL. The generator writes one, so add others by hand.
Does a missing robots.txt hurt?
No. If the file returns a 404, crawlers assume everything is allowed. Persistent server errors (5xx) are worse, because Google may pause crawling.
Does robots.txt affect AI search tools that cite sources?
Some do read it, which is why blocking assistant-related bots can cost you citations. Decide per bot, not as a blanket rule.
Is my domain sent anywhere?
No. The file is built in your browser.