Robots.txt Generator

Configure your crawl rules and copy a ready-to-use robots.txt file.

Default — All Robots are

Crawl-Delay

Sitemap

Robots

Restricted Directories

Your Generated Robots.txt File

Copy this into a file named robots.txt at your site root.

Step 1 - Set the default rule for all crawlers — choose Allowed to permit crawling by default or Refused to block the entire site from all bots.

Step 2 - Override the default for individual crawlers — allow or block any of 30+ search engines, AI crawlers, and SEO tools by name.

Step 3 - Add a sitemap URL, set an optional crawl delay, and enter any directory paths you want blocked from all crawlers.

Step 4 - Your robots.txt file generates live in the output panel. Click Copy Code to copy it and paste it into your site's root directory.

Robots.txt Generator FAQs

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website that tells web crawlers which pages or sections they are allowed or not allowed to access. It is one of the first files a crawler checks before indexing your site and is a standard part of technical SEO.

Where does my robots.txt file need to be placed?

Your robots.txt file must be placed in the root directory of your website — for example, https://www.yourdomain.com/robots.txt. If it is placed anywhere else, crawlers will not find it and the rules will have no effect.

Does robots.txt prevent pages from appearing in search results?

Not reliably. Blocking a page in robots.txt prevents crawlers from accessing it, but if other sites link to that page, search engines may still index it and show it in results without any content preview. To prevent a page from appearing in search results entirely, use a noindex meta tag instead.

What is the difference between Disallow and noindex?

Disallow in robots.txt tells crawlers not to crawl a URL. Noindex is a directive in the page's HTML that tells crawlers not to include the page in search results. If you block a page with Disallow, crawlers cannot see the noindex tag — so use noindex on pages you want deindexed but accessible, and Disallow on pages that should not be crawled at all.

What is crawl delay and should I use it?

Crawl delay tells crawlers how many seconds to wait between requests to your server. It can reduce server load from aggressive bots. However, Google ignores the crawl-delay directive entirely — if you need to manage Googlebot's crawl rate, do so through Google Search Console instead. Bing and Yandex do honor the directive.

Should I block AI crawlers in my robots.txt?

That depends on your goals. Blocking AI training crawlers like GPTBot or ClaudeBot prevents your content from being used to train AI models, but does not remove your site from AI-generated answers or search results. Some crawlers — such as certain Meta AI and social media bots — are documented to ignore robots.txt, so blocking them in this file alone may not be effective.

What are restricted directories and when should I block them?

Restricted directories are folder paths on your site that you want to prevent all crawlers from accessing — for example /wp-admin/, /checkout/, /account/, or /private/. Blocking these keeps internal, duplicate, or sensitive pages out of search indexes and helps crawlers focus their budget on your most important public content.

Does blocking a crawler in robots.txt stop it completely?

For reputable crawlers that follow the standard, yes. However, some bots — particularly certain AI scrapers and social media crawlers — are known to ignore robots.txt. For those, a firewall or WAF (Web Application Firewall) rule at the server level is a more reliable enforcement method.

How often should I update my robots.txt file?

Update your robots.txt file whenever your site structure changes significantly — such as when you add new sections you want excluded from crawling, launch a new subdirectory, or want to opt out of a newly identified AI crawler. It is also good practice to review it during any major site migration or replatform.

Is the Yotpo Robots.txt Generator free to use?

Yes, this Robots.txt Generator is completely free to use. For tools that help your product pages rank and convert — through customer reviews, ratings, and UGC — check out Yotpo Reviews here.