We Tested What AI Crawlers Can Read on Four Major Retail Product Pages. Target’s Price Wasn’t There.

We expected to find gaps. We did not expect to find a top-five US retailer publishing product pages where the price does not appear in the HTML at all, with no structured data to fall back on.

What follows is the full method, the numbers for all four retailers, and a five-minute test you can run on your own site. But the finding rests on one mechanic worth getting straight first, because it’s the piece that most often gets assumed rather than checked: how an AI crawler obtains a web page in the first place.

Four different questions that keep getting merged

Most confusion in this area comes from treating one question as four. They are worth separating, because the answers differ and only two of them are what this article measures.

Scroll table sideways →

A search partner can hand an AI system your URL. That is discovery, and it is genuinely a route to being surfaced. But discovery does not put your price into the response. This article is about the last two rows, extraction and rendering, which is where the measurable damage happens and where our test has evidence.

Screenshots are the exception, not the rule

It’s a common assumption that AI models “look at” web pages the way a person does, taking something like a screenshot and interpreting it visually. The assumption is reasonable, because there genuinely is a mode where AI uses screenshots. When you watch a demo of an AI agent booking a flight or filling in a form (Claude in Chrome, OpenAI’s Operator, “computer use”), you are watching a model work with visual input. That’s the version of AI-meets-web that gets filmed, so it’s the version people remember.

But that’s an agent doing a task on one page at a time, in a real browser, on demand. It has nothing to do with how models are trained on your site or how AI search engines assemble an answer. Those run on a completely different pipeline, and even the agents prefer text when they can get it, because reading the DOM is faster, cheaper and more reliable than squinting at pixels.

There are three distinct ways an LLM touches a web page, and only one of them involves images at all.

Scroll table sideways →

Crawling and answer-time retrieval are how AI systems obtain the content of your pages at scale. Neither executes your JavaScript.

Table 2: Which crawlers render JavaScript, and what each is for

The distinction in the third column matters more than most people realise. Blocking a training crawler is not the same decision as blocking a search crawler, and the operators document them separately.

Scroll table sideways →

Rendering behaviour from Vercel’s December 2024 analysis of its own network traffic across three production sites, which found that no major AI crawler rendered JavaScript. GPTBot fetched JS files on about 11.5% of requests and ClaudeBot on about 23.8%, executing them on effectively none. Those are Vercel’s measurements on its own network, not vendor documentation. A January 2026 server-log study found GPTBot’s script execution “extremely limited” rather than literally zero. Anthropic also documents Claude-User and Claude-SearchBot; neither was in Vercel’s dataset, so we make no rendering claim about them.

The practical consequence is the one that catches teams out. Your site ranks in Google, so it must be readable. Except Googlebot is a renderer, and it is not a proxy for crawlers that aren’t. One important nuance from Vercel’s work: content that sits in the initial HTML response can still be ingested even when it’s inside a JSON payload, which is what the next section is about.

The three tiers of AI visibility

“Is it in the HTML or not” turns out to be too blunt. When we pulled apart real retail pages, content fell into three clearly different buckets, and the middle one is where most teams get a nasty surprise.

What we found: four retailers, measured

On 25 August 2026 we took one product page from each of four major US retailers, loaded it in a real browser, and compared every block of text a shopper can see against the HTML the server actually returns. The test is deliberately generous to the retailers: we requested the page as a signed-in-capable browser with full cookies, so if anything, these sites gave us more than they’d give GPTBot.

A note on units. The percentages below count text blocks, not words or characters. A one-word heading counts the same as a 400-word product description. That makes the measure a good indicator of how much of a page’s structure survives, and a poor proxy for how much prose survives. We report it as blocks throughout for that reason.

Scroll table sideways →

Read the Tier 1 row as a structural visibility score. Home Depot, the best of the four, still loses half its blocks. Target loses nearly nine-tenths of them.

That right-hand column is not an excerpt. It is the entire readable content of the page. Look at what happens near the end:

About this item, Details, Specifications, Size Guide, Shipping & Returns, Q&A (206) … Frequently bought together … Guest ratings & reviews.

Every section heading is present. Not one of them has any content underneath it. The crawler gets a table of contents for a book with no pages.

And there is no price. Not a formatted price, and not even a $ character anywhere in the readable markup of a retail product page. Target also ships zero JSON-LD blocks, so there’s no structured-data fallback either. We checked the fully-rendered DOM too: still zero.

The grey box in that screenshot isn’t a loading bug we caught at a bad moment. It’s the page as delivered. JavaScript fills it a moment later for human visitors. Non-rendering clients never see it.

It’s worth stating the conclusion precisely, because this is where articles on this subject usually overreach. A search provider may well tell an AI system that this Target page exists, and that link may be surfaced. That is discovery, and it is real. What it does not do is put the price into the page’s response. In our test the price was absent from both the readable HTML and any Product JSON-LD. Any retrieval path that relies on that non-rendered response cannot obtain the price from the product page itself. It would have to come from somewhere else entirely.

Best Buy does much better. Its price, title, rating and specifications are all properly server-rendered. But it still loses the parts of the page that do the persuading.

The headline “4.7” survives. The rating distribution, the 96% would recommend to a friend line, the Verified Purchaser badges, the accessory recommendations and the in-store availability all arrive by separate API calls. When an AI is asked “is this TV any good?”, the specific evidence that would answer the question isn’t in what it fetched.

Three failures in one robots.txt file

While we were testing, we read Best Buy’s robots.txt, and it earns a detour because it shows how quietly this category of work fails.

The file is unusually thoughtful. It’s organised into labelled buckets. “BUCKET 1: AI SEARCH & DISCOVERY BOT ACCESS” deliberately admits answer engines into /site/ and /discover-learn/. “BUCKET 2: AI TRAINING / DATA INGESTION FIREWALL” is meant to block model-training scrapers outright. Someone thought hard about the difference between being cited and being harvested, which is exactly the right instinct and more than most retailers attempt.

It does not work, for three separate reasons.

1. The firewall names crawlers that don’t exist

Bucket 2 targets GPTBot-Web, CCBot, Anthropic-AI, Claude-Web and Bytespider. OpenAI documents four crawlers: GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User. There is no GPTBot-Web. Anthropic’s current documentation, dated 7 April 2026, lists three: ClaudeBot, Claude-User and Claude-SearchBot. Anthropic-AI and Claude-Web are retired identifiers. Three of the five names in the firewall correspond to no crawler that will ever visit the site. Bucket 1 has the same problem in miniature: it lists Gemeni-AI, which is a misspelling.

2. The blocking rule blocks almost nothing

User-agent: GPTBot-Web
User-agent: CCBot
User-agent: Anthropic-AI
User-agent: Claude-Web
User-agent: Bytespider
Disallow: /[1.1]

Disallow: /[1.1] is syntactically valid. Under RFC 9309 a Disallow value is matched as a literal path prefix, and the only special characters are * and $. Square brackets are not a regular-expression character class. So the rule means “block URLs whose path begins with the literal characters /[1.1]“, which on a normal site is no URLs at all. It is valid and useless, which is worse than broken, because nothing will ever flag it.

The second bucket’s Disallow: / [2.1] is a different case: the RFC’s path grammar doesn’t permit an unencoded space, so that one is genuinely malformed. Both bracketed numbers look exactly like footnote markers pasted in from a reference document.

3. The real training crawlers are in the other bucket

GPTBot is OpenAI’s model-training crawler and ClaudeBot is Anthropic’s. Both are named in Bucket 1, the allow-oriented group, which grants Allow: /site/ and Allow: /discover-learn/ and then lists specific exclusions rather than a blanket Disallow: /. Under RFC 9309, a URL matching no rule in its applicable group is allowed. So the two crawlers the firewall exists to stop are governed by the permissive bucket instead.

The lesson isn’t that Best Buy is careless. The file is more sophisticated than most, and Bucket 1 shows real understanding that being cited and being harvested are separable decisions worth controlling separately. The lesson is that nobody validated it, and robots.txt fails silently. So does server-side rendering. So does structured data. This whole category of problem produces no error, no alert and no ticket. It just quietly does the opposite of what you intended.

Why any of this pays

AI-referred traffic is still a small slice of the total, but it is growing fast and converting unusually well. Adobe Digital Insights, drawing on more than a trillion visits to US retail sites, found AI-driven retail traffic up 138% year on year in May 2026, with AI-referred visitors generating 53% more revenue per visit and converting 54% better than non-AI traffic. Shopify’s own Q1 2026 data shows more than half of AI-referred sessions starting on a product detail page, against about 20% for organic search.

Separately, in April 2026, Adobe scored retail page types for machine readability: homepages 75%, category pages 74%, product pages 66%, with the best retailers reaching 82.5% and the weakest 54.2%. The pages that convert are the pages machines read worst, which matches what we measured. The commercial logic is uncomfortable: high-intent visitors are arriving from AI, landing directly on product pages, and product pages are the least legible thing you publish.

Scroll table sideways →

The pattern in Table 3 is almost always the last two rows: a modern framework rendering the shell on the server, then filling the commercially important modules (price, stock, reviews, variants) on the client.

Test your own site in five minutes

None of these is a crawler simulation. Each one answers a narrower question, which is what makes them useful.

1. Is the phrase in the response at all?

curl -s https://yoursite.com/your-page | grep -i "your key phrase"

No output means the phrase is not present in the HTML response returned by that request. If it only appears after JavaScript runs, don’t assume a non-rendering crawler can reach it. Bear in mind that some sites return different responses by user agent, geography or cookies, so this is one data point, not a verdict.

2. A rough no-JavaScript text approximation.

curl -s https://yoursite.com/your-page | sed 's/<[^>]*>//g' | tr -s ' \n' ' \n'

sed is not an HTML parser, and this will mangle inline scripts, entities and multiline tags. It is a smell test, not a measurement. If the output is a few hundred words of navigation and empty headings, you’ve found your problem.

3. The JavaScript-off test. In Chrome DevTools, press Cmd/Ctrl+Shift+P, type “Disable JavaScript”, reload. This is a useful visual approximation of which content depends on client-side execution. It is not a crawler simulation: headers, cookies, cache, user agent and CDN behaviour all still differ.

4. Check your structured data. Search the source for application/ld+json. On a product page you want a Product block with offers, price and aggregateRating.

5. Validate your robots.txt in Google Search Console rather than trusting that it says what you meant, and check every user-agent token against the operator’s current documentation.

The checklist to hand your developers

• Put price, availability, title, description and specifications in the initial HTML response, not just the page shell
• Emit complete Product JSON-LD with offers, price, priceCurrency, availability and aggregateRating
• Server-render the first 5 to 10 reviews and the rating distribution; lazy-load the rest
• Put spec tables and product copy in real HTML, not behind a click or an accordion that fetches on open
• Confirm robots.txt doesn’t block the AI answer engines you want citing you, and that the tokens it names are real
• Add the check to CI: fetch the page without JS, assert the price appears

None of this requires abandoning React. SSR, SSG and ISR all exist inside the frameworks you already use. It’s a configuration and architecture decision, not a rewrite.

Frequently asked questions

Do LLMs read websites by taking screenshots?

No. AI crawlers make a standard HTTP request and read the raw HTML text that comes back, stripping the tags to get plain text. They do not open a browser or capture images. The only time an AI works from screenshots is agentic browsing, meaning an AI operating a real browser to complete a task, and even then text extraction from the DOM is preferred because it’s faster and more accurate.

Do AI crawlers execute JavaScript?

Generally no. In the largest published measurement, none of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot or Bytespider executed JavaScript; they parse the HTML response as returned. Google’s Gemini and Apple’s AppleBot are the exceptions, because both use existing search-crawler rendering infrastructure. This means a JavaScript-dependent page can rank well in Google Search while its content is unavailable to the crawlers that feed ChatGPT, Claude and Perplexity.

My site ranks well in Google. Doesn’t that mean AI can read it?

Not necessarily. Googlebot renders JavaScript and executes your scripts before indexing; AI crawlers generally do not. Strong Google rankings on a client-rendered site are evidence that Googlebot’s renderer is doing work no AI crawler will repeat. Discovery and retrieval pipelines differ between Google Search and AI search systems, so the two must be tested separately.

What is the single fastest way to check whether an AI crawler can read my page?

Run curl -s https://yoursite.com/your-page | grep "your key phrase" in a terminal. No output means the phrase isn’t present in the HTML response returned by that request, so a crawler that doesn’t execute your JavaScript cannot extract it from the page. In our August 2026 test of four major retail product pages, between 49% and 80% of visible text blocks were absent from the response entirely, and between 49% and 89% were absent from the readable markup.

Does structured data fix the problem?

It helps significantly but it isn’t a complete fix. Valid Product JSON-LD gives machines an explicit, structured representation of price, availability and ratings even when the visible page is client-rendered, which is why Target’s lack of any JSON-LD compounds its rendering problem. It makes the information accessible; it does not guarantee that a given system ingests or uses it. And structured data only covers the fields it defines, so descriptive copy, specifications and review text still need to exist in the HTML.

Which parts of an ecommerce page are most often invisible?

In our testing, the consistent Tier 3 casualties were customer review text and rating distributions, real-time inventory and store availability, variant and configuration options, financing and promotional messaging, and “frequently bought together” recommendations, precisely the modules that answer a shopper’s buying questions.

Research conducted 25 August 2026. Each product page was loaded in Chrome, then the same URL was re-requested and its raw HTML compared block-by-block against the rendered text. Percentages count text blocks, not words. Crawler identities were checked against operator documentation on 25 August 2026. Method and figures are reproducible with the commands above.

Sources

Crawler rendering behaviour and identity

• Vercel: The rise of the AI crawler (Dec 2024). The primary measurement behind the 11.5% / 23.8% JS-fetch rates and the finding that no major AI crawler renders
• EdgeComet: How GPTBot and ChatGPT-User handle JavaScript (Jan 2026). Independent server-log replication
• OpenAI: crawler documentation. GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and why blocking them are separate decisions
• Anthropic: crawler documentation (7 April 2026). ClaudeBot, Claude-User and Claude-SearchBot
• RFC 9309: Robots Exclusion Protocol · Google robots.txt specification · Google’s open-source robots.txt parser

Commercial impact

• Adobe Digital Insights: Q3 2026 AI Traffic Trends Report (June 2026). 138% YoY, 53% RPV, 54% conversion, May 2026 data
• Adobe: AI traffic grows but retail sites lag in AI search visibility (April 2026). Machine-readability scores by page type, Q1 2026 data
• Shopify: AI-referred shoppers convert better and spend more (May 2026)
• Digital Commerce 360: Adobe AI-referred traffic to retail sites doubles in a year (June 2026)

Pages tested

• Best Buy robots.txt · Target robots.txt · Wayfair robots.txt · Home Depot robots.txt