# Manage your robots.txt in MetaPilot

**Category:** Control · **Reading time:** 5 min

Your `robots.txt` tells crawlers which parts of your site they may access. Not sure what that means? Read [What is robots.txt?] This guide shows how to generate, edit, and control yours in MetaPilot.

---

## Before you start

`robots.txt` reaches crawlers through whichever channel you have connected:

- A **Cloudflare Worker** serves it from your zone.
- The **WordPress plugin** writes it to your web root.
- On **Wix and Shopify**, MetaPilot merges its rules into the file the platform already serves.

Changes go live once a channel is connected — see [Connect Cloudflare to MetaPilot]. You can still generate and edit the file beforehand.

---

## Steps

### 1. Open Agent Web → Robots & LLMs

This is where MetaPilot stores your `robots.txt`, `llms.txt`, and `llms-full.txt`.

### 2. Generate your robots.txt

Click **Generate robots.txt**. MetaPilot pre-fills a sensible default — allowing normal crawling and pointing crawlers at your sitemap so new pages are discovered faster.

### 3. Review and edit

The content opens in an editor. Adjust the rules if you need to — for example, disallowing an admin path — then click **Save**.

### 4. Control AI crawlers the easy way

Instead of hand-editing rules for each bot, use the one-click **Allow / Block** toggle in the AI-crawler roster on the same page (GPTBot, ClaudeBot, PerplexityBot, and more). MetaPilot rewrites *its own managed block* in your `robots.txt` — it never touches the rules you wrote by hand.

### 5. It's served at the edge

Your delivery channel serves the current `robots.txt` on the next crawl — no upload, no deploy.

---

## Take control of your robots.txt

Sign in and open **Robots & LLMs** to get started.