Can AI Crawlers Execute JavaScript? The 2026 Answer
Vorec Team · 2026-09-27 · About 12 min read
Some do and most don't say: Google documents that Googlebot renders JavaScript in headless Chromium, Bing has said bingbot generally can, Apple's Applebot docs imply rendering, and Common Crawl states that CCBot does not execute JavaScript. OpenAI, Anthropic and Perplexity document their crawlers' names, purposes and robots.txt rules but do not state whether GPTBot, ClaudeBot, PerplexityBot or their user-triggered fetchers execute JavaScript, so the safe assumption is to put your important content in the initial HTML.
"AI crawlers don't execute JavaScript" is one of the most repeated claims in AI-search advice. It is used to justify expensive work: rebuild the docs site, add prerendering, move to server-side rendering, or the models will never see your content.
We went looking for the vendor documents behind it. This post separates what each operator actually publishes from what third parties have measured, and then covers what to do either way.
Checked on 24 October 2026. Every vendor page cited below was fetched on that date.
Which AI crawlers render JavaScript: the documented answer
| Crawler | Operator | Purpose | Renders JavaScript? | Date checked |
|---|---|---|---|---|
| GPTBot | OpenAI | Collects content for training foundation models | Not documented (OpenAI crawler docs) | 24 Oct 2026 |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results | Not documented (OpenAI crawler docs) | 24 Oct 2026 |
| ChatGPT-User | OpenAI | Visits pages during user-initiated actions in ChatGPT and Custom GPTs | Not documented (OpenAI crawler docs) | 24 Oct 2026 |
| ClaudeBot | Anthropic | Collects web content for model training | Not documented (Anthropic crawler guidance) | 24 Oct 2026 |
| Claude-SearchBot | Anthropic | Improves search result quality | Not documented (Anthropic crawler guidance) | 24 Oct 2026 |
| Claude-User | Anthropic | Fetches pages in response to a user's question | Not documented (Anthropic crawler guidance) | 24 Oct 2026 |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results | Not documented (Perplexity crawler docs) | 24 Oct 2026 |
| Perplexity-User | Perplexity | Fetches pages for user-initiated actions | Not documented (Perplexity crawler docs) | 24 Oct 2026 |
| Googlebot | Crawls for Google Search | Yes: headless, evergreen Chromium in a separate rendering stage (JavaScript SEO basics) | 24 Oct 2026 | |
| Google-Extended | robots.txt token controlling use of crawled content for Gemini training and grounding | Not a crawler: it has no user agent of its own; crawling uses existing Google user agents (Google's crawler list) | 24 Oct 2026 | |
| Bingbot | Microsoft | Crawls for Bing | Yes, "generally", with limits on frameworks and scale (Bing Webmaster Blog, 31 Oct 2018) | 24 Oct 2026 |
| Applebot | Apple | Powers Spotlight, Siri and Safari search; data may train Apple's models | Implied yes: Apple warns that blocked JavaScript and CSS may stop it rendering content properly (About Applebot) | 24 Oct 2026 |
| CCBot | Common Crawl | Builds the open Common Crawl dataset that many models train on | No: "JavaScript is not executed" (Common Crawl FAQ) | 24 Oct 2026 |
Two things the table does not say. "Not documented" is not "No": the OpenAI, Anthropic and Perplexity pages are silent on rendering, which tells you nothing about behaviour either way. And the Bing statement is a 2018 blog post, the most recent vendor statement on rendering we found for bingbot, not a current documentation page.
What third-party measurement says
The strongest public evidence on the undocumented crawlers is a measurement study, not vendor documentation.
Third-party measurement: "The Rise of the AI Crawler" by Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl and Ryan Siddle (Vercel and MERJ, published 17 December 2024) analysed crawler traffic on nextjs.org, Vercel's network and two job board sites. It reported that none of the major AI crawlers it observed rendered JavaScript, naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot. It also found that OpenAI's and Anthropic's crawlers did fetch JavaScript files without executing them, and that Applebot rendered JavaScript through a browser-based crawler. Read the study on Vercel's blog.
How to weigh it: it is a careful observation of real traffic, but it is from late 2024, from one network, and the vendors have not confirmed it. Crawlers change without announcement. Treat it as the best available evidence, not as a permanent property of any bot.
First, "AI crawler" is three different things
Most of the confusion comes from one phrase covering three categories with different economics:
| Category | What it does | Example user agents |
|---|---|---|
| Bulk/training crawlers | Fetch pages at scale to build datasets | GPTBot, ClaudeBot, CCBot |
| Search crawlers | Build an index for an AI search product | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-triggered fetchers | Fetch one page because someone asked right now | ChatGPT-User, Claude-User, Perplexity-User |
A browser-based agent driving a real browser session is a fourth case again, and it is the one most likely to render, because it is a browser.
There is no reason to assume these behave identically. A claim that covers all of them at once should be treated with suspicion.
Robots.txt handling differs too. OpenAI says robots.txt rules may not apply to ChatGPT-User because its visits are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says Claude-User honors robots.txt signals.
A claim repeated in a hundred posts is not better evidenced than a claim repeated in one. Check whether any of them links to a vendor document or a named measurement, or whether they all link to each other.
What Google does document
Google documents its behaviour explicitly, which makes it worth separating from the rest.
Google's JavaScript SEO basics describes crawling, rendering and indexing as distinct stages. Pages that return 200 are queued for rendering, and once resources allow, a headless, evergreen Chromium executes the JavaScript. The page may wait in that queue for seconds or longer. That tells you rendering is a separate, deferred step for Google. It tells you nothing about any other company's crawler.
On Gemini specifically: Google-Extended is only a robots.txt control. Google states it has no separate user agent, that crawling is done with existing Google user agents, and that blocking it does not affect inclusion or ranking in Google Search.
Google also documents dynamic rendering (serving crawlers a prerendered version) as "a workaround and not a long-term solution". It recommends server-side rendering, static rendering or hydration instead.
The decision, with the conditions attached
It is tempting to present this as a decision that makes itself. That is too neat. Here is the version with the conditions left in:
| If crawlers render JS | If they don't | |
|---|---|---|
| Server-rendered HTML | Content available immediately; still no guarantee of indexing | Content available; still no guarantee of indexing |
| Client-side only | May work, but rendering can fail on timeouts, errors or blocked resources | Content likely unavailable to that crawler |
Two conditions the table needs:
HTML availability is not indexing. Serving content to a crawler is necessary, not sufficient. Pages can be fetched and still not indexed, for reasons that have nothing to do with rendering.
Server rendering has real costs. Infrastructure, caching complexity, cache-invalidation bugs, and a build or runtime path to maintain. "Just render on the server" is a genuine trade-off, not a free win.
What survives: client-side-only rendering has a failure mode that server-rendered HTML does not, it is invisible unless you test for it, and for the crawlers whose behaviour is undocumented you cannot rule it out. That is a strong reason to lean toward HTML-first. It is not a proof.
How to make your pages readable to AI crawlers
1. Render on the server or prerender
Use server-side rendering, static generation, or prerendering at build time so the response already contains the article text. If you cannot change the architecture yet, serving crawlers a prerendered copy of the same content is a stopgap; Google calls it a workaround, and Bing's 2018 guidance accepted it as long as the content matches what users get.
2. Put critical content in the initial HTML
Headings, body copy, prices, tables, product names and internal links should be in the first response, not injected after load. Structured data (JSON-LD) belongs there too. Content behind tabs, accordions or "load more" buttons that fetch on click is the first thing an HTML-only fetcher misses.
3. Test with curl and a crawler user agent
Fetch the page without a browser and look for your text:
A count of zero, or a body that is mostly an empty `<div id="root">`, means an HTML-only crawler sees nothing. A spoofed user agent only shows what your server sends to that string, not what the real bot receives; see the verification notes below.
4. Allow the bots you want in robots.txt
Training and search access are separate decisions with separate user agents. A typical setup that welcomes AI search and training crawlers:
Swap `Allow` for `Disallow` on any crawler you want to opt out of. Also make sure you are not disallowing the JavaScript and CSS files themselves; Apple specifically warns that blocked resources can stop Applebot rendering the page.
5. Consider an llms.txt file
`llms.txt` is a community proposal for a plain-text map of your most useful pages. It costs little to publish, but none of the crawler pages cited above mention reading it. Treat it as optional and never as a substitute for content in your HTML. More on that in What Google Actually Says About llms.txt.
What we do, and its limits
Our blog pages are normally assembled in the visitor's browser, with content stored separately. We run middleware that detects crawler user agents and returns rendered HTML (article body, headings, tables and structured data) without requiring JavaScript.
Exactly what we tested, and when. On 24 October 2026, sending Googlebot and GPTBot user-agent strings with `curl` to `https://vorec.ai/blog/process-documentation-guide`, we received HTTP 200 with a rendered `<h1>` and one `Article` JSON-LD block in the raw response body, with no JavaScript executed. A must-differ control, `/blog/definitely-not-real-xyz`, returned HTTP 404 with no `Article` JSON-LD for both user agents, confirming the check distinguishes a real article from a missing one rather than passing everything. A default user agent on the real URL also returned 200, but without the `Article` JSON-LD block, which is the difference the middleware exists to make.
The limits of that, stated plainly: it covers blog article routes, on that date, and it is a request we made ourselves. A spoofed user agent is not an authenticated crawler, and what we receive is not proof of what any real bot receives. Server logs show a user-agent string, which anyone can send, so attributing an entry to a genuine crawler means using whatever verification the vendor supports, such as published IP ranges or reverse DNS where offered, and those resources differ between vendors. Even then, logs typically record status and bytes rather than the response body, so establishing what a crawler actually received means pairing verified requests with logged response evidence and, where available, the vendor's own rendered inspection tooling.
We are also not claiming every route on our site behaves this way. We tested blog article paths.
Two things we got wrong along the way, which are more useful than the parts that worked:
Returning 200 for pages that do not exist. The app served its shell with a 200 status for unknown URLs, which looks like a real page. For crawler user agents, blog article paths now return a genuine 404 with `noindex` when we can confirm the content is absent, and deliberately do not 404 when the lookup itself fails, because a transient database error must never tell a crawler that a live page was deleted. The lookup returns one of three states (found, absent, or unknown) and only `absent` produces the 404; a failed fetch, an unparseable response or a thrown error all resolve to `unknown` and fall through to the previous behaviour.
Checking status codes instead of content types. A missing image also returned 200 with the HTML shell. `curl -I` looked healthy; the image was broken. Assert the `Content-Type`, not just the status.
On cloaking, since prerendering raises it: serving crawlers a rendered version of the same content users get is not generally treated as cloaking, and Bing's 2018 guidance says so explicitly. The problem is serving materially different content. Keeping the two in parity is the thing to verify.
How to check your own site
- [ ] Fetch a page without JavaScript, `curl -s https://yoursite.com/page`, and confirm your article text is in the response, not an empty `<div id="root">`
- [ ] Repeat with a crawler user agent and compare; the content should match what users see
- [ ] Check that a URL which does not exist returns 404, not 200 with your app shell
- [ ] Check images return `Content-Type: image/*`, not `text/html`
- [ ] Use Search Console's URL Inspection for Google's rendered view specifically
- [ ] For what real bots receive, pair server logs with whatever verification each vendor supports (published IP ranges or reverse DNS), rather than trusting user-agent strings, and remember logs usually record status and size, not the body
- [ ] Run a must-fail control: if everything passes, test something you know is broken and confirm it fails
That last step matters because a negative control can reveal a check that passes for the wrong reason.
FAQ
Can AI crawlers execute JavaScript?
It depends on the crawler. Googlebot does, and Google documents it. Bing has said bingbot generally can, and Apple's guidance implies Applebot renders pages. Common Crawl states CCBot does not. OpenAI, Anthropic and Perplexity do not document it either way; a December 2024 third-party study by Vercel and MERJ observed that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot fetched pages without executing JavaScript.
Do AI crawlers render JavaScript-heavy single-page apps?
Assume not, unless the vendor documents it. A client-side-only page may reach Google intact while appearing nearly empty to crawlers that read only the initial HTML. Server-side rendering, static generation or prerendering removes the question.
Does ChatGPT see JavaScript content on my site?
OpenAI's crawler documentation does not say whether GPTBot, OAI-SearchBot or ChatGPT-User execute JavaScript. The only public evidence is third-party measurement, which observed no execution in late 2024. If content matters for ChatGPT search, put it in the initial HTML and allow OAI-SearchBot in robots.txt.
Does Googlebot rendering mean Gemini sees my JavaScript content?
Google-Extended is a robots.txt token, not a separate crawler; Google says crawling is done with its existing user agents. Google documents rendering for Search, but does not spell out how rendered output feeds Gemini training or grounding, so do not rely on rendering for AI features beyond Search.
Should I block AI crawlers?
Training access and search access are separate decisions with separate user agents, and vendors document them separately. You can allow a search crawler while blocking a training crawler. Decide each deliberately rather than treating it as one switch.
Is dynamic rendering a good long-term answer?
Google describes it as a workaround and not a long-term solution. If you are building now, server-side rendering, static generation or hydration are the better-supported paths. We run a crawler path because of how our application is built, which is a constraint, not a recommendation.
Related reading: What Google Actually Says About llms.txt.
Want the recording to narrate itself? Record with Vorec, using its own macOS recorder (an AI agent can drive it for you), or upload a recording you already have. Vorec drafts narration matched to the workflow it captured and generates the voiceover, so nothing is spoken into a microphone. The same capture can also produce a written step-by-step guide. Start your free trial, no card needed.