Your first 5 pages, free. Approve every change or run on autopilot.
Control 5 min

Manage your robots.txt in MetaPilot

Your robots.txt tells crawlers which parts of your site they may access. Not sure what that means? Read What is robots.txt? This guide shows how to generate, edit, and control yours in MetaPilot.

Before you start

robots.txt reaches crawlers through whichever channel you have connected: a Cloudflare Worker serves it from your zone, the WordPress plugin writes it to your web root, and on Wix and Shopify MetaPilot merges its rules into the file the platform already serves. Changes go live once a channel is connected — see Connect Cloudflare to MetaPilot. You can still generate and edit the file beforehand.

The advanced end of MetaPilot. What each file does, and why to change one thing at a time. 53 seconds.
  1. 1

    Open Agent Web → Robots & LLMs

    This is where MetaPilot stores your robots.txt, llms.txt, and llms-full.txt.
  2. 2

    Generate your robots.txt

    Click Generate robots.txt. MetaPilot pre-fills a sensible default — allowing normal crawling and pointing crawlers at your sitemap so new pages are discovered faster.
  3. 3

    Review and edit

    The content opens in an editor. Adjust the rules if you need to — for example, disallowing an admin path — then click Save.
  4. 4

    Control AI crawlers the easy way

    Instead of hand-editing rules for each bot, use the one-click Allow / Block toggle in the AI-crawler roster on the same page (GPTBot, ClaudeBot, PerplexityBot, and more). MetaPilot rewrites its own managed block in your robots.txt — it never touches the rules you wrote by hand.
  5. 5

    It's served at the edge

    Your delivery channel serves the current robots.txt on the next crawl — no upload, no deploy.

Take control of your robots.txt

Sign in and open Robots & LLMs.

Get started

Stuck? Ask an AI to walk you through it

ChatGPT Claude Perplexity Meta AI Grok