Why can't AI crawlers see my single-page app?
TL;DR
A single-page app builds its pages in the visitor's browser. The HTML it serves first is a near-empty shell, and most AI crawlers never run the JavaScript that fills it, so the shell is all they read. MetaPilot cannot run your bundle for them. It can put your metadata, your identity and your SEO files into the HTML they do fetch.
What it is
A single-page application is a website built as one JavaScript program. React, Vue, Svelte and most things bundled by Vite work this way. Your server sends a small HTML file that is little more than an empty container and a script tag, and the script then draws the header, the copy, the product grid and everything else inside the visitor's browser.
A person never notices. A crawler is the whole story. Anything that does not run JavaScript gets the file your server sent, not the page the browser built. It is the difference between being handed a recipe and being handed dinner.
Here is one measured example: a distributor's newly rebuilt homepage served roughly 3.4 KB of HTML carrying about five words of readable text, one script bundle, and no structured data at all. The same site's older server-rendered product pages carried thousands of words each. Nothing was broken. The new page simply did its work somewhere the crawler was not looking.
Why it matters
- Search engines got there first. Google can run your JavaScript before it indexes a page, on its own schedule. That is why a client-rendered site can rank perfectly well and still feel a step behind.
- Answer engines mostly do not. The crawlers feeding AI assistants fetch your HTML and read what is in it. Content assembled in the browser is not there when they look, so it cannot be quoted, summarized or cited, however good it is.
- The parts that decide how you appear live in the shell anyway. Your title, your meta description, your Open Graph preview and your JSON-LD all belong in the
<head>, which is served rather than rendered. An app that hands the sameindex.htmlto every route ships the same ones on every URL until something changes them. - You cannot see any of this in a browser. Your browser runs the bundle, so the page looks finished to you. The gap only shows in the HTML your server actually sent.
How MetaPilot handles it
Start with the honest half. MetaPilot does not run your JavaScript. It reads the HTML your server sends, the same way a crawler does, and it is not a prerendering service. It will not make your body content appear for something that will never run your bundle. Only a rendering change on your side does that, which means server-rendering or pre-rendering your pages.
What it can do is put real answers into the HTML that is fetched. On a Cloudflare zone, MetaPilot's Worker sits in the live response path and rewrites each HTML response on the way out. A near-empty shell is still HTML with a <head>, so injection at the edge works on it exactly as it does on a server-rendered page.
- Your identity, on a page with nothing on it. The Organization entity is written once inside MetaPilot and published to the URLs you choose. It is not derived from page content, so a shell can still carry a correct, complete description of who publishes the site.
- Per-page metadata that never touches your bundle. Structured data, Open Graph, title and description overrides and your Markdown copy for agents are stored in MetaPilot and injected at the edge. None of it depends on a browser running anything.
- Your files, served by us.
llms.txt,llms-full.txt, your sitemap and yourrobots.txtcome from MetaPilot, so they are right whatever your app does. - Proof it stayed there. Schema liveness monitoring re-checks the live page and tells you when what was injected stops being served.
Two limits are worth knowing before you start. Where MetaPilot writes page content for you, it is grounded in the HTML your server sends, and on a shell there is almost nothing to read. Per-page schema and Markdown on an SPA are yours to author and review rather than something to generate. And autopilot spots a change by watching the served response, so a shell that never changes will not tell it the content behind the shell did.
Not sure what your own pages serve? The free site audit fetches them and grades exactly this, no sign-up. When you are ready to fix the served half: Connect Cloudflare to MetaPilot →
Related
FAQ
Does MetaPilot render my JavaScript?
No. MetaPilot fetches the HTML your server sends, the same way a crawler does, and adds to it. It does not run your bundle and it is not a prerendering service.
Does ChatGPT read JavaScript?
The crawlers that fetch pages for AI assistants read the HTML they are served, and most of them do not execute JavaScript. If your content is assembled in the browser, it is not in what they read.
Will structured data fix an empty page?
It fixes a different half. Structured data tells an engine what a page is and who publishes it, in a form that needs no rendering. It is not a replacement for body text a crawler can read, so it works best alongside server-rendering rather than instead of it.
Do I have to rewrite my site?
No. Getting your metadata, your identity and your SEO files right takes no change to your app at all. Getting your words in front of crawlers that will not run your bundle is a rendering change on your side, and most frameworks support it directly through server-rendering or static generation.
Ask an AI to explain this