A thread about Akamai has been doing the rounds this week. The claim is that the cloud and security company built a second version of its website that only AI agents ever see, and that its visibility in AI answers went up 364%. I read the case study behind it. The result holds up as far as it goes, but the lesson most people are taking from it is the wrong one for a small business website.
What Akamai did
The work was done by Scrunch, an AI search company, with a product it calls the Agent Experience Platform. According to Scrunch's case study, the network that sits in front of Akamai's website picks out requests from AI agents such as ChatGPT-User and PerplexityBot and sends them to a separate copy of each page. That copy is plain HTML that needs no JavaScript to read, rewritten from Akamai's existing documentation and blog posts. People visiting the site still get the normal pages, and Googlebot and Bingbot are deliberately kept away from the AI copy.
The test ran from 1 December 2025 to 19 January 2026 on pages about one product line. Compared with pages that weren't changed, Scrunch reports 85% more citations in AI answers, citations across 38% more prompts and a 364% rise in how often Akamai appeared for questions that didn't mention it by name. Akamai has since turned the idea into a product of its own, called AI Brand Presence.
Keep two things in mind when you read those numbers. Both companies now sell this, so the figures are their own, and the case study doesn't say how many pages were tested or what the starting figures were. A percentage with no starting point can't tell you how big the change was.
Why AI couldn't read Akamai's pages properly
Most AI crawlers don't run JavaScript. When Vercel studied AI crawler traffic across its network in December 2024, it found none of the major ones rendered it, including OpenAI's crawlers, Anthropic's ClaudeBot and PerplexityBot. Google's Gemini was the exception, because it uses Googlebot, which does. If a page builds its words in the browser with JavaScript, those crawlers fetch it and find very little to read.
That's the problem Akamai paid to get around. Its site is large and leans on a JavaScript framework, and putting a plain copy in front of the bots was a smaller job than rebuilding the whole thing. For a company that size, I can see why they made that call.
Why I wouldn't build one for a small business
In search, showing robots one page and people another is called cloaking. Google's spam policies describe it as presenting different content to users and search engines to manipulate rankings, and treat it as spam. Akamai stays on the right side of that by keeping Google's and Bing's crawlers off the AI copy and building it from the same information as the real pages. Doing that properly on a small site means keeping two versions of every page and making sure they never drift apart, which is ongoing work for no clear gain.
The search engines don't want the second version either. In February this year, Google's John Mueller answered a question about separate pages for AI crawlers with "Why would they want to see a page that no user sees?" Bing's Fabrice Canel said Bing would crawl the normal page anyway to check the two matched, and that versions built only for bots tend to get neglected and break. Both were reported by PPC Land.
There's a simpler reason as well. If your pages are already plain HTML that any crawler can read, a second version has nothing left to fix.
What we found when we tested our own site
So I checked ours. On 20 November I requested our web design page while identifying as six different crawlers: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot. Every one got the same 24,624-byte page a normal browser gets, and our robots.txt allows all of them. I also stripped the scripts out of four of our pages, the homepage among them, and each one still carried between 1,056 and 1,103 words of content. A crawler that never runs JavaScript reads the same page you do.
Some of that comes from how the site is built, and some from a mistake we made in August. A scroll animation on our support and pricing pages left most of their content invisible on phones, because the stylesheet hid sections that the script never brought back. We changed it so nothing gets hidden unless that script is running, which means anything that can't run scripts, crawlers included, gets the whole page.
My test does have a gap, and most small business sites share it. I sent those crawler names from my own connection, and a security service can tell that isn't where the real bots come from. Services like Cloudflare recognise genuine crawlers by the networks they run from, and since July 2025 Cloudflare has blocked AI crawlers by default on new domains unless the owner says otherwise.
Where a request comes from changes what it gets back, and we learned that on our own systems. A security rule in front of our domains once turned away requests from cloud servers while browsers loaded the same page without a problem, so one of our services couldn't reach another. AI crawlers run from cloud servers too, which is how a site can look perfect in your browser and still be closed to ChatGPT.
How to check your own site
- Look at the page the way a crawler does. In Chrome, right-click the page, choose View page source and search for a sentence you can see on screen. If it isn't in the source, a crawler that doesn't run JavaScript can't see it either.
- Read your robots.txt at yourdomain.com.au/robots.txt. Blocking GPTBot keeps your pages out of OpenAI's training data and nothing more. Blocking OAI-SearchBot is different, because OpenAI says sites that opt out of it won't be shown in ChatGPT search answers.
- Check your security settings. If your site sits behind Cloudflare or a similar service, find its AI crawler or bot settings and see what's blocked. Search crawlers and training crawlers are usually listed separately, so you can allow one and block the other.
- Ask an AI tool about your business. Ask ChatGPT or Perplexity what you do and where you are, then look at what it links to. If the answer comes from a directory listing rather than your own site, the AI tools are relying on somebody else's description of you.
If your site does depend on JavaScript
The fix is to make the one version everyone gets readable, by having the server send the finished page rather than assembling it in the browser. Most WordPress sites already work that way. Some app-style frameworks and a few page builders don't, and it's worth finding out which kind you have before you pay anyone for an "AI version" of your site. It's one of the first things we look at when we build or rebuild a website in Ballarat, because a page that Google, ChatGPT and your customers all read the same way only has to be kept up to date once.
Want to know what AI tools can read on your site?
Ask us to check it →