# robots.txt and llms.txt — generated, served, and always fresh.

**Category:** SEO and AI files

Generate a `robots.txt` and `llms.txt` tuned for search engines and AI crawlers straight from your dashboard. Auto-refresh rebuilds your `llms.txt` as pages change — served at the edge alongside your structured data.

---

## The Problem

`llms.txt` is how you hand AI assistants a clean map of your site. But a stale `llms.txt` is worse than none: it points models at pages that moved, changed, or no longer exist, and they believe it.

Maintaining that file — and a sane `robots.txt` for AI crawlers — by hand is exactly the busywork nobody does. So both rot the moment the site evolves, and the tool you generated them with has long since forgotten you exist.

---

## How It Works

1. **Generate** — MetaPilot pre-fills your `robots.txt`, `llms.txt`, and `llms-full.txt` from your sitemap data. Edit and save from the dashboard — no file to upload.
2. **Keep them fresh** — Auto-refresh rebuilds the managed `llms.txt` and `llms-full.txt` from current sitemap data whenever pages change or schema regenerates. It diffs before it writes, and never touches your `robots.txt`.
3. **Served at the edge** — The same Cloudflare Worker serves all three files, so an edit is live on the next request. No deploy, no cache purge.

---

## What You Get

- **An AI site map, not a guess** — `llms.txt` and `llms-full.txt` hand answer engines a curated view of what your content is and where it lives — instead of leaving them to infer it from your nav.
- **An llms.txt generator that keeps going** — Most generators produce a file once. Auto-refresh rides the same change signal as auto-pilot, so your llms files rebuild as the site evolves.
- **robots.txt tuned for AI crawlers** — Allow the engines you want citations from, block the ones you do not, and see it enforced at the edge.
- **Safe by design** — Auto-refresh only rewrites the managed llms files, diffs before writing, and never edits your `robots.txt` or anything you wrote by hand.
- **No upload step** — You do not commit these files, you do not deploy them, and you do not have to remember where they live. The Worker serves the current version on the next request.
- **Editable when you want control** — Everything is generated as a starting point, not a cage. Edit any of the three files in the dashboard and save — or turn auto-refresh off and own them yourself.

---

## Plan Availability

Available on **every plan, including Free**. Auto-refresh is a single per-organization toggle and is **on by default**, so your llms files stay current unless you deliberately turn it off.

---

## Frequently Asked Questions

**What is llms.txt?**
A plain-text map of your site written for AI assistants: it tells them what your content is and where to find it. `llms-full.txt` carries fuller detail.

**Will auto-refresh overwrite my custom robots.txt?**
No. Auto-refresh only rebuilds the managed llms files. Your `robots.txt` and anything hand-written is never touched.

**How do the files get served?**
By your channel. The Cloudflare Worker serves them at the edge, so a change is live on the next request — no upload, no deploy, no CDN purge. On Wix the app writes `robots.txt` and `llms.txt`; Shopify keeps control of its root files.

**Does llms.txt update when my content changes?**
Yes, with auto-refresh on. It rebuilds from current sitemap data whenever pages change or schema regenerates, and diffs first so an unchanged file is not rewritten.

**Can I turn auto-refresh off?**
Yes — it is one toggle on your dashboard, applied per organization. Turn it off and the files stay exactly as you last saved them.

---

## Keep Reading

- What is llms.txt?
- Guide: set up llms.txt
- Guide: manage your robots.txt
- See which AI crawlers read your site