Enter your website details and choose the pages you want to block from search engines.
Select the common pages you don't want Google and other search engines to crawl.
Stop AI companies from using your content to train their models. This won't affect whether your pages appear in ChatGPT, Gemini or Perplexity's live answers — only their training crawlers.
Place the file in your website root so it becomes available at example.com/robots.txt.
Block feeds, search, checkout and tag pages while keeping important pages crawlable.
Robots.txt optimises crawl budget. It should not be used to protect sensitive content.