Robots.txt Generator

Runs in your browserSEO & Web

Build a robots.txt with crawler rules, AI-bot blocking and sitemaps, then test any URL against it.

RulesWhich paths each crawler may visit

Group 1

* means every crawler. Separate several names with commas.

Ignored by Google; Bing uses it.

Paths to keep out, one per line.

Paths inside a blocked folder that may be crawled.

AI crawlersAsk AI companies' crawlers to stay out of the whole site

Model training

AI search

Fetches when a user asks (may ignore robots.txt)

Well-known AI companies say their crawlers follow these rules, but obeying robots.txt is voluntary.

Sitemaps

Full addresses, one per line. Crawlers read every Sitemap line.

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/

No problems found

Upload it as robots.txt at the root of your site, for example https://www.example.com/robots.txt.

Test a URLChecks the rules above the way crawlers do

BlockedLine 2: Disallow: /admin/ is the longest matching rule. Googlebot has no group of its own, so it follows the * group.
How to use, limits & privacy

About Robots.txt Generator

Build a correct robots.txt file for your website: which paths each crawler may visit, which AI crawlers to keep out, and where your sitemaps are. The tool points out common mistakes, never writes a group that would accidentally block your whole site, and lets you test any URL against the rules the way crawlers read them. It runs in your browser.

How to Use

1

Set the rules

In Rules, enter a user-agent (* for every crawler) and the paths to keep out under Disallow, one per line. Use Allow for exceptions, or start from Allow all, Block all or Example.

2

Choose AI crawlers

Under AI crawlers, pick Block AI training, Block all AI, or tick individual bots such as GPTBot, ClaudeBot or Google-Extended. Add your sitemap addresses under Sitemaps.

3

Test and check

Read the notes under the file, then type a path in Test a URL and pick a crawler to see whether it is allowed and which line decides it.

4

Upload the file

Click Download and upload robots.txt to the root of your site, so it opens at https://yourdomain/robots.txt.

Privacy & Processing

  • Mode: local
  • Files Leave Browser: Local tool processing; review details below
  • Max Input Size: Device memory limits
  • Account Required: No
  • Data Stored Locally: Nothing is saved; reloading the page clears your rules.
  • Network Processing: Assets or models may require an initial download

The file is built and tested in your browser. Nothing is uploaded.

Rules & Limitations

  • robots.txt must sit at the root of each host (https://www.example.com/robots.txt) as plain UTF-8 text. Google reads only the first 500 KiB.
  • It stops crawling, not indexing: a blocked page can still appear in Google if other sites link to it. Use a noindex meta tag to keep a page out of results.
  • Google ignores Crawl-delay; Bing uses it.
  • Obeying robots.txt is voluntary. Major AI companies say their crawlers follow it, but fetchers acting on a user's request may not.
  • The tester follows RFC 9309: the crawler uses the group that names it (or the * group), the longest matching rule wins and Allow wins a tie.

Top Suggestions

  • Keeping admin, cart, checkout and search result pages out of crawlers
  • Opting a blog or news site out of AI model training
  • Pointing search engines to one or more sitemaps
  • Checking why Google says a page is blocked by robots.txt

Robots.txt Generator FAQ

How do I block AI bots like ChatGPT in robots.txt?

Choose Block AI training to add GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider and Amazonbot, each with Disallow: /. Block all AI also adds AI search crawlers such as OAI-SearchBot and PerplexityBot, which removes your pages from those AI search answers.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended only controls whether Google may use your pages for Gemini. Google Search uses Googlebot, which follows your other rules.

Where do I put the robots.txt file?

In the top folder of your website, so it opens at https://yourdomain/robots.txt. Each subdomain, such as shop.example.com, needs its own file.

What is the difference between Disallow and noindex?

Disallow in robots.txt asks crawlers not to fetch a page. A noindex meta tag tells search engines not to show a page, and they must be able to crawl the page to see it, so do not block a page you want removed with noindex.

Why does an empty group get "Disallow:"?

A group needs at least one rule. Without it, crawlers join the user-agent line to the next group and follow that group's rules, which could block your whole site. An empty Disallow means everything is allowed.

Is my robots.txt sent anywhere?

No. The file is built and tested in your browser, and nothing is uploaded until you put it on your own site.

Related Tools

SEO

Sitemap Generator

Build an XML sitemap from a list of URLs, or validate an existing sitemap and see every error.

Runs locallyOpen tool
SEO

Meta Tag Generator

Write title, description and Open Graph tags; preview the link on Google, WhatsApp, Facebook, LinkedIn and X.

Runs locallyOpen tool
Apache

.htaccess Generator

Generate Apache .htaccess rules for HTTPS and www redirects, IP blocking, compression, caching and 301s.

Runs locallyOpen tool