robots.txt and llms.txt — generated, served, and always fresh.
Generate a robots.txt and llms.txt tuned for search engines and AI crawlers straight from your dashboard. Auto-refresh rebuilds your llms.txt as pages change — served at the edge alongside your structured data.
# Northwind Tools > Field-service scheduling for HVAC teams. ## Docs - [Quickstart](/docs/quickstart): install in 5 min - [API reference](/docs/api): REST + webhooks ## Product - [Pricing](/pricing): plans and limits - [Changelog](/changelog): shipped weekly # rebuilt 4 min ago · 118 pages · diffed before write
The problem
llms.txt is how you hand AI assistants a clean map of your site. But a stale llms.txt is worse than none: it points models at pages that moved, changed, or no longer exist, and they believe it.
Maintaining that file — and a sane robots.txt for AI crawlers — by hand is exactly the busywork nobody does. So both rot the moment the site evolves, and the tool you generated them with has long since forgotten you exist.
How it works
-
Generate
MetaPilot pre-fills your
robots.txt,llms.txtandllms-full.txtfrom your sitemap data. Edit and save from the dashboard — no file to upload. -
Keep them fresh
Auto-refresh rebuilds the managed
llms.txtandllms-full.txtfrom current sitemap data whenever pages change or schema regenerates. It diffs before it writes, and never touches yourrobots.txt. -
Served at the edge
The same Cloudflare Worker serves all three files, so an edit is live on the next request. No deploy, no cache purge.
What you get
An AI site map, not a guess
llms.txt and llms-full.txt hand answer engines a curated view of what your content is and where it lives — instead of leaving them to infer it from your nav.
An llms.txt generator that keeps going
Most generators produce a file once. Auto-refresh rides the same change signal as auto-pilot, so your llms files rebuild as the site evolves. A build-time tool cannot rebuild a live file on content change.
robots.txt tuned for AI crawlers
Allow the engines you want citations from, block the ones you do not, and see it enforced at the edge — the same managed block AI-crawler controls write to.
Safe by design
Auto-refresh only rewrites the managed llms files, diffs before writing, and never edits your robots.txt or anything you wrote by hand.
No upload step
You do not commit these files, you do not deploy them, and you do not have to remember where they live. The Worker serves the current version on the next request.
Editable when you want control
Everything is generated as a starting point, not a cage. Edit any of the three files in the dashboard and save — or turn auto-refresh off and own them yourself.
Plan availability
Available on every plan, including Free. Auto-refresh is a single per-organization toggle and is on by default, so your llms files stay current unless you deliberately turn it off.
Frequently asked
A plain-text map of your site written for AI assistants: it tells them what your content is and where to find it. llms-full.txt carries fuller detail. Longer explainer →
No. Auto-refresh only rebuilds the managed llms files. Your robots.txt and anything hand-written is never touched.
By your channel. The Cloudflare Worker serves them at the edge, so a change is live on the next request — no upload, no deploy, no CDN purge. On Wix the app writes robots.txt and llms.txt; Shopify keeps control of its root files.
Yes, with auto-refresh on. It rebuilds from current sitemap data whenever pages change or schema regenerates, and diffs first so an unchanged file is not rewritten.
Yes — it is one toggle on your dashboard, applied per organization. Turn it off and the files stay exactly as you last saved them.
Give AI assistants a fresh map of your site
Generate all three files from your sitemap in a minute, then let auto-refresh keep the llms files honest.
Ask an AI whether you need an llms.txt