Can ChatGPT Read Your Website? 5 Checks in 10 Minutes
AI readability is whether ChatGPT may fetch a website and can read the text without running JavaScript. In “Can ChatGPT Read Your Website? 5 Checks in 10 Minutes”, Pavlo Zhydkykh tests robots.txt, the firewall, the raw HTML, the first paragraph and ChatGPT’s own answer before any AI visibility work.
Pavlo ZhydkykhUpdated · 7 min read
Short answer
ChatGPT can only recommend your company if OpenAI's crawlers are allowed to fetch your pages and can read the text in them. On most sites I check, one of those two fails without anyone noticing: a robots.txt rule, a firewall setting or a page whose words only appear after JavaScript runs. Here is the 10-minute check I run before any AI visibility work, with a free way to test each step yourself.
Key takeaways
- OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers.[2]
- Since 1 July 2025 Cloudflare asks every new domain at sign-up whether to allow AI crawlers.[3]
- Vercel’s study found that none of the major AI crawlers render JavaScript.[4]
- Google says no new AI text files are needed to appear in AI Overviews or AI Mode.[5]
- From 1 July to 28 September 2026 Pavlo Zhydkykh’s own site got 91 visitors from ChatGPT and about 18 from Google.
What is AI readability?
AI readability is whether the crawlers behind ChatGPT and other assistants may fetch a website and can read the words on its pages. AI readability depends on robots.txt, on firewall and CDN bot settings, and on whether the words are in the HTML the server sends or appear only after JavaScript runs.
“I never start AI visibility work before this check. Clever copy is worthless if ChatGPT gets a 403 or an empty page.”
Last checked · 7 min read
Why I run this check before any AI visibility work
From 1 July to 28 September 2026 my own site got 91 visitors from ChatGPT and about 18 from Google, according to my site analytics. For a small B2B site that made AI answers a real channel, and it changed the order of my work: before I touch copy, schema or llms.txt for a client, I check that ChatGPT can reach and read the site at all. Everything else is wasted if it cannot.
The check takes about 10 minutes, needs no paid tool and finds the problems I see most often on startup sites. When I ran the same kind of check on all 50 companies of the 2026 Forbes Fintech 50, most sites passed, but the failures were exactly the ones a team never notices from the inside.
The 10-minute check at a glance
| Check | How to run it | Pass |
|---|---|---|
| 1. robots.txt | Open yoursite.com/robots.txt | No Disallow for OAI-SearchBot or ChatGPT-User |
| 2. Firewall | Your CDN's bot or AI crawler settings | AI search crawlers allowed |
| 3. Raw HTML | View page source, search for your headline | Headline, pricing and key facts are in the source |
| 4. First paragraph | Read only the first paragraph of key pages | It says what you sell, for whom and roughly what it costs |
| 5. ChatGPT itself | Ask what your company does, then ask buyer questions | The description is right and you appear for some questions |
Run them in this order. A site that fails check 1 or 2 is invisible no matter how good its copy is, and checks 4 and 5 only make sense once the crawler can read the page.
Check 1: does robots.txt let ChatGPT in?
OpenAI runs several crawlers with different jobs, and robots.txt treats them separately. The one that matters for ChatGPT search is OAI-SearchBot.
| Crawler | What OpenAI uses it for | If you block it |
|---|---|---|
| OAI-SearchBot | Finding pages for ChatGPT search | Your site is not shown in ChatGPT search answers |
| GPTBot | Collecting content that may train its models | Your content stays out of training; search is not affected |
| ChatGPT-User | Fetching a page when a user asks ChatGPT to open it | OpenAI says robots.txt rules may not apply to these visits |
OpenAI's crawler documentation is explicit: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and a robots.txt change can take about 24 hours to reach its search systems. So open yoursite.com/robots.txt and look for any group that names OAI-SearchBot, or a User-agent: * group with Disallow: /. Blocking GPTBot is a legitimate choice if you do not want your content used for training. It does not remove you from search, and in my Fintech 50 check 3 of 46 sites made exactly that choice while keeping every AI search crawler allowed.
Check 2: is your CDN or firewall blocking AI bots?
robots.txt is a request. A firewall is a wall, and it can block ChatGPT while robots.txt says yes. Since 1 July 2025 Cloudflare asks every new domain at sign-up whether to allow AI crawlers, as its press release describes. A founder who clicked through that screen in a hurry may have blocked them without knowing.
- In Cloudflare, open the domain's AI crawler and bot settings and see which crawlers are blocked.
- On other CDNs and hosts, look for bot protection or "block AI scrapers" switches.
- Check the firewall's event log for requests from OAI-SearchBot that got a 403 or a challenge page.
A quick test from your own computer:
curl -A "OAI-SearchBot" https://yoursite.com/If it returns a challenge page instead of your HTML, something is blocking by user agent. A clean result is not the full answer, because firewalls that verify bots by IP address treat the real crawler differently from your laptop, so the event log stays the final word.
Check 3: is your text in the HTML before JavaScript runs?
This is the failure I find most often on startup sites, because the page looks perfect in a browser. Vercel's study of AI crawlers found that none of the major ones render JavaScript: ChatGPT's and Claude's crawlers download script files but do not run them. They read the HTML your server sends, and nothing a script adds later.
To test it, open your homepage, choose View Page Source and search for your headline, your price and one sentence from the middle of the page. If they are missing, an AI crawler sees a near empty page. In the Fintech 50 check, 44 of 47 homepages had their main text in the HTML. Two did not: Aven sent 16 words before JavaScript ran against 2,015 on the rendered page, and Securitize 14 against 1,338.
The fix is server rendering or static HTML, not a new tool. Framer and Astro both publish real HTML. My own site runs on Astro, which ships no JavaScript by default, so every word is in the source before any script runs.
Check 4: can a stranger tell what you sell from the first paragraph?
A crawler that can read your page still has to understand it. Read only the first paragraph of your homepage and your main product or service page. Would a stranger know what you sell, who it is for and roughly what it costs? Many startup homepages open with a slogan that only makes sense once you already know the product.
I open every key page with two or three plain sentences that answer exactly that, with the company name in them. It is the part of a page an assistant is most likely to quote, and it costs nothing to write.
Check 5: ask ChatGPT what your company does
Now ask ChatGPT, with search on, what your company does. Compare its answer with your own one line description. A wrong or outdated answer usually means the web describes you in different words in different places, or that an old page still ranks for your name.
Then write 10 questions a buyer would type before they know you exist, such as "best payroll API for startups" rather than your brand name, and run them in ChatGPT and Perplexity. Note who gets named and which pages the answers cite. That list of cited pages is where you need to appear next.
Does passing the check make you rank in ChatGPT?
No. People search "how to rank in ChatGPT" as if it worked like Google rankings, but this check only removes the reasons ChatGPT cannot use your site. Whether it names you depends on how clearly your pages answer buyer questions and on what other sites say about you: comparison articles, review sites, directories and forum threads.
Two things I would not spend the first week on. An llms.txt file is cheap and I publish one, but Google says you do not need new AI text files to appear in its AI features. And schema markup helps machines read facts, but it cannot rescue a page whose text the crawler never received. For the bigger picture, my guide to generative engine optimization covers what comes after access.
What I fix first when a site fails
The order is always the same: unblock first, then make the words readable, then make them clear.
- Blocked crawlers. A robots.txt line or a firewall switch, usually fixed in minutes.
- Text that needs JavaScript. Server rendering or a rebuild on a stack that publishes HTML, such as Framer or Astro + Sanity.
- Vague first paragraphs. A rewrite of the opening lines of 5 to 10 key pages.
- Missing mentions. The slow part: getting onto the pages ChatGPT already cites.
If you run these checks with an AI coding agent, my count of the Claude Code skills I actually use for web design shows which skills earn their place and which never get called.
If you would rather have someone run all of this for you, my free AI visibility check asks 20 buyer questions across 5 AI assistants and reports what each one says about your company, and AI visibility work fixes what the check finds.
If you need this built
A landing page from Pavlo starts at $1,200 (1 to 2 weeks), a multi-page site at $3,000 (2 to 4 weeks). The price is fixed before work starts.
See all pricesFrequently asked questions
Can ChatGPT read my website?
It can if robots.txt and your firewall let OAI-SearchBot in and your text is in the HTML the server sends. Open yoursite.com/robots.txt, check your CDN's AI crawler settings and view the page source of your homepage to confirm all three.
Which ChatGPT crawler should I allow?
Allow OAI-SearchBot, the crawler OpenAI uses for ChatGPT search. Blocking GPTBot only keeps your content out of model training. ChatGPT-User fetches pages when a user asks, and OpenAI says robots.txt rules may not apply to it.
Does ChatGPT run JavaScript on my pages?
Not when its crawler reads your site. Vercel found that the major AI crawlers, including ChatGPT's, download JavaScript files but do not execute them, so text that only appears after scripts run is invisible to them.
How long until ChatGPT sees a robots.txt change?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Firewall changes take effect as soon as you save them.
How do I rank in ChatGPT?
There is no ranking you can buy or switch on. First make sure ChatGPT can read your site, then open key pages with plain answers to buyer questions and get mentioned on the comparison articles, review sites and threads that ChatGPT cites for your category.
Do I need llms.txt for ChatGPT to read my site?
No. ChatGPT reads your normal pages. Google says no special AI text files are needed for its AI features either. llms.txt is cheap to add, but fix blocked crawlers and JavaScript-only text first.
Free tool Does ChatGPT recommend you? Free report in 48 hours · 5 AI engines
Checklist
5 checks that show whether ChatGPT can read a website
The order in which Pavlo Zhydkykh runs the checks, from the failures that make a site invisible to the ones that make a page unclear.
1. robots.txt
No Disallow rule for OAI-SearchBot, the crawler behind ChatGPT search.
2. Firewall and CDN
AI search crawlers allowed in the bot settings and absent from the block log.
3. Raw HTML
The headline, pricing and key facts appear in View Page Source.
4. First paragraph
A stranger can tell what the company sells, for whom and roughly at what price.
5. ChatGPT’s answer
ChatGPT describes the company correctly and names the company for some buyer questions.
Fit check
Is An AI readability check on a startup website
right for you?
A good fit if
- ChatGPT describes the company wrongly or not at all, and nobody knows why.
- The site is a JavaScript app, so the text may not be in the HTML.
- The domain sits behind Cloudflare or another CDN with bot protection switched on.
- A redesign or platform move is planned, and Pavlo Zhydkykh makes the new site readable by AI crawlers from day one.
Not the right fit if
- The goal is a guaranteed mention in ChatGPT: access makes a mention possible, not certain.
- The site already passes all five checks and the gap is mentions on other sites.
- The content sits behind a login, which no crawler reads either way.
- The question is about AI training rights, which is a legal decision, not a technical one.
Compared
OAI-SearchBot vs GPTBot
Two OpenAI crawlers, two robots.txt decisions, as OpenAI describes the two and as Pavlo Zhydkykh advises.
| Criterion | OAI-SearchBot | GPTBot |
|---|---|---|
| Used for | ChatGPT search | Training OpenAI’s models |
| If blocked | The site is not shown in ChatGPT search answers | Content stays out of training |
| Effect on ChatGPT search | The site drops out | None |
| Pavlo Zhydkykh advises | Keep allowed | The owner decides |
In their words
ChatGPT and AI crawlers, in the words
of the AI companies and clients
Word for word, with the source linked: research, official documentation and the people Pavlo Zhydkykh has worked for.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
“The results consistently show that none of the major AI crawlers currently render JavaScript.”
“You don't need to create new machine readable files, AI text files, or markup to appear in these features.”
“One of the best contractors I have worked with. Really thoughtful about communication. Went above and beyond in delivering the results that our team needs. Would 100% hire again for our projects.”
“Pavlo was a pleasure to work with on this project. The web and mobile designs he created were amazing. His communication was prompt which made for a smooth project. I highly recommend his services.”
“The collaboration went smoothly and the work was delivered as agreed. Communication was timely, and the final result aligned with the initial expectations. Overall, a very positive and rewarding experience!”
About the author
Pavlo Zhydkykh
Pavlo Zhydkykh is a web designer and developer in Eastern Europe who has built websites for AI, fintech and B2B startups since 2016, designing and building each project himself on Framer or Astro + Sanity. Clients include MStudio Asia, Silicon Data, Inturai, Accolade Pro, NimbleStax and Sancts.
- Top Rated on Upwork
- 100% Job Success
- 10 years since 2016
- 50+ sites shipped
- 90+ PageSpeed at launch
About Pavlo LinkedInUpworkContra Privacy policy
Page reviewed by Pavlo Zhydkykh · Updated
In this topic
AI search
6 articles
- 01 Can ChatGPT Read Your Website? 5 Checks in 10 Minutes You are here
- 02 Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50 Next
- 03 What Is Schema Markup? The Types a SaaS Site Needs in 2026 6 min
- 04 GEO vs SEO vs AEO: What Actually Changes for a SaaS Website 6 min
- 05 What Is Generative Engine Optimization (GEO)? A Guide for B2B SaaS Teams 6 min
- 06 llms.txt in 2026: What It Is, an Example and What It Does 6 min