How to see your site the way an AI crawler does
The quickest test takes ten seconds. Open your page, right-click and choose View page source, or press Ctrl+U (Cmd+Option+U on a Mac). That window shows the HTML your server sends, before any JavaScript runs. Search it for a sentence from your page, such as your headline or a price.
If you find it, crawlers that skip JavaScript can read it. If you only find scripts and an empty body, they cannot. Most AI crawlers skip JavaScript: Vercel found that GPTBot, ClaudeBot and PerplexityBot fetch pages without running their scripts (Vercel, December 2024).
Ask the way a crawler asks
Some sites and firewalls treat crawlers differently from browsers, so it is worth asking the way a crawler asks. In a terminal on Windows, Mac or Linux:
curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com/ -o page.htmlOpen page.html in a text editor and search it. Repeat with ClaudeBot or PerplexityBot in place of GPTBot. A 403 error or a challenge page means something is stopping that crawler before it reaches your content.
That happens more often than people expect. On 1 July 2025 Cloudflare changed its default to block AI crawlers unless they pay (Cloudflare). If your site is on Cloudflare, check its AI crawler settings as well as your robots.txt.
What we saw on our own pages
On 13 November 2026 we fetched two home pages both ways, as a Chrome browser and as GPTBot. This site returned 18.1 KB with 3,486 readable characters to both. TradeCatcher, which builds its pages in the browser, returned 24.6 KB with zero readable characters to both. Each site gave both visitors the same answer, but one answer had text in it and the other did not. The details are in why AI crawlers see an empty page on a no-code app.
Let a tool do it
The free GEO audit fetches your page from a server without running JavaScript. It checks the raw HTML for a title, a description, a single H1 heading, a canonical tag and structured data. It also checks whether robots.txt lets the main AI crawlers in, and whether you have a sitemap and an llms.txt file. It gives a score out of 100 and writes copy-paste fixes for anything that needs work.
It measures readiness, not rankings. Nobody outside the AI companies can see how their assistants choose sources, and any tool that claims to is guessing.
Sources
- Our measurement: TradeCatcher and redcenterstudio.com home pages, fetched as Chrome and as GPTBot, 13 Nov 2026
- The rise of the AI crawler, Vercel, 17 Dec 2024
- Content Independence Day: no AI crawl without compensation, Cloudflare, 1 July 2025
- Overview of OpenAI crawlers, OpenAI