Skip to content
SEO Tool

Robots.txt Generator

Create crawl rules for search engines and AI crawlers.

Input

About This Tool

Generate a robots.txt file with sitemap, crawl-delay, disallow rules, and optional AI crawler controls. Everything runs locally in your browser: nothing you paste is uploaded, stored, or logged, and there is no sign-up or usage limit. Read the full guide below for worked examples, best practices, and common mistakes to avoid.

Frequently Asked Questions

Does robots.txt guarantee pages will not appear in search?

No. Robots.txt controls crawling, not indexing. Use noindex meta tags when you need a page removed from search results.

Should I block AI crawlers?

It depends on your content strategy. Some sites allow discovery but restrict training-oriented crawlers. Review each crawler before publishing rules.

Where should robots.txt be placed?

Upload it to the root of your domain, such as https://example.com/robots.txt.

How to create a robots.txt file that helps crawlers, not hurts you

A robots.txt file is a plain text file at the root of your site that tells crawlers which URLs they may request. Enter your domain, list the paths to block, and the generator writes the file for you. Upload it so it loads at https://yourdomain.com/robots.txt.

The file controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Google's own documentation says robots.txt is not a way to keep a page out of Google; for that, use a noindex rule or password protection.

Step by step: using the generator

  1. Type your domain, such as https://example.com. The tool strips trailing slashes and uses it to write a Sitemap: line.
  2. List folders or URL paths to block, one per line, such as /wp-admin/ or /cart/. A missing leading slash is added for you.
  3. Leave crawl delay at 0 unless a particular crawler needs it. See the note below.
  4. Tick Add AI crawler controls if you want separate rules that block six named AI-related bots.
  5. Copy the output, save it as robots.txt (plain text, UTF-8), and upload it to your site root.

What the generated file looks like

Inputs: domain https://example.com, disallow /wp-admin/ and /cart/, crawl delay 0, AI controls off.

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

With no blocked paths, the tool writes Allow: / instead, which means everything may be crawled. Tick the AI box and six more groups are appended, each with Disallow: /.

The tool assumes your sitemap lives at /sitemap.xml. If your site uses another address, such as /sitemap_index.xml in many WordPress SEO plugins, edit that line before uploading. A sitemap line pointing to a 404 does no good.

The syntax, line by line

LineMeaning
User-agent: *Start of a group that applies to all crawlers without a more specific group
User-agent: GPTBotStart of a group for one named crawler
Disallow: /private/Do not crawl URLs beginning with this path
Allow: /private/public.htmlException to a broader Disallow
Disallow: (empty)Nothing is blocked
Disallow: /Block the whole site for that crawler
Sitemap: https://…Absolute URL of a sitemap; not tied to a user-agent group

Paths are case sensitive: /Private/ and /private/ are different. Google also supports * as a wildcard and $ as an end-of-URL marker, so Disallow: /*.pdf$ blocks PDF files. The generator does not build wildcards, but you can add them by hand. The details are in Google's guide to creating a robots.txt file.

When two rules both match a URL, Google follows the more specific one, meaning the longer path. If they tie in length, the less restrictive rule wins.

What to block, and what to leave alone

Good candidates

  • Admin and login areas such as /wp-admin/ (keep /wp-admin/admin-ajax.php reachable on WordPress sites that need it)
  • Cart, checkout, and account pages
  • Internal search results such as /?s= or /search/
  • Faceted filter URLs that create endless combinations
  • Staging or test folders (better still, put these behind a password)

Do not block

  • CSS and JavaScript files. Google needs them to render the page like a user sees it. Blocking them can hurt how the page is understood.
  • Pages you want in search. This sounds obvious, but a stray Disallow: / left over from a staging site is a classic launch-day disaster.
  • Pages you have marked noindex. Crawlers must be able to fetch the page to see the tag. If you block it, the noindex is never read.

The AI crawler option explained

Ticking the AI box adds separate groups that disallow these user-agents: GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, and PerplexityBot. They do different jobs:

  • GPTBot (OpenAI), ClaudeBot (Anthropic), and CCBot (Common Crawl) collect content that can be used for model training or datasets.
  • ChatGPT-User and PerplexityBot are tied to assistants that fetch or surface pages when answering questions, so blocking them can reduce how often you are cited.
  • Google-Extended is a control token for whether Google uses your content for certain generative AI products. Google states it does not affect your ranking or inclusion in Google Search.

It is all-or-nothing in this tool. The checkbox blocks all six. If you want to be cited by AI assistants but not trained on, delete the lines for the assistants you want, and keep the others. Crawler names change, so confirm current names in each company's own documentation before relying on the list.

Robots.txt is a request that well-behaved crawlers honour. It is not access control. A scraper that ignores the rules will not be stopped by it.

Crawl-delay: mostly a myth for Google

The crawl delay field writes Crawl-delay: N. Some search engines honour it. Google does not support the directive, so it will have no effect on Googlebot. If your server is struggling under crawl load, fix the capacity or use the crawl-rate tools in Google Search Console. Leave the field at 0 unless you know a specific crawler needs it.

Upload, test, and monitor

  1. Save the file as robots.txt, lowercase, UTF-8 plain text. Google reads only the first 500 KiB, which is far more than any normal file needs.
  2. Place it in the top-level directory of the host. A file at /blog/robots.txt is ignored, and each subdomain needs its own.
  3. Open https://yourdomain.com/robots.txt in a browser and confirm it loads as text.
  4. Check the robots.txt report in Google Search Console for fetch errors and warnings.
  5. After launch, review the file whenever you redesign, migrate, or add a store. If you use WordPress, an SEO plugin may already serve a virtual file; make sure two sources do not conflict.

If you also manage page-level search snippets, see the meta description checker. Anyone building the page itself can preview markup in the HTML live preview.

Examples for common site types

A WordPress blog

User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

This keeps crawlers out of the dashboard and internal search results, but lets them reach the AJAX endpoint that some themes and plugins use on the front end. The tool's output has a single Disallow list, so add the Allow line and adjust the sitemap address by hand.

An online store

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Carts, checkouts, and account pages have no search value and generate endless private URLs. Product and category pages stay open.

A site under construction

User-agent: *
Disallow: /

This asks all compliant crawlers to stay away. It is fine for a short build period, but remove it at launch, and add a password too, since the file does not stop anyone from visiting.

Common mistakes

  • Blocking the whole site by leaving Disallow: / from staging.
  • Trying to hide private pages with robots.txt. The file is public, and listing a path advertises it.
  • Using a relative sitemap URL. It must be absolute.
  • Forgetting that rules apply per host and protocol. www and non-www are separate.
  • Expecting a rule to remove an already indexed page. Use a removal tool or noindex.

More questions

How long until Google sees my changes?

Google generally caches robots.txt for up to a day, so updates apply soon but not instantly. You can request a recrawl of the file in Search Console.

Can I have more than one Sitemap line?

Yes. Add one line per sitemap, each with a full URL. The generator writes one, so add others by hand.

Does a missing robots.txt hurt?

No. If the file returns a 404, crawlers assume everything is allowed. Persistent server errors (5xx) are worse, because Google may pause crawling.

Does robots.txt affect AI search tools that cite sources?

Some do read it, which is why blocking assistant-related bots can cost you citations. Decide per bot, not as a blanket rule.

Is my domain sent anywhere?

No. The file is built in your browser.