Skip to content

Can ChatGPT Read Your Website? 5 Checks in 10 Minutes

AI readability is whether ChatGPT may fetch a website and can read the text without running JavaScript. In “Can ChatGPT Read Your Website? 5 Checks in 10 Minutes”, Pavlo Zhydkykh tests robots.txt, the firewall, the raw HTML, the first paragraph and ChatGPT’s own answer before any AI visibility work.

Pavlo Zhydkykh

Updated · 7 min read

A blue circle on a line that passes through five open gates to a page with four lines of text

Short answer

ChatGPT can only recommend your company if OpenAI's crawlers are allowed to fetch your pages and can read the text in them. On most sites I check, one of those two fails without anyone noticing: a robots.txt rule, a firewall setting or a page whose words only appear after JavaScript runs. Here is the 10-minute check I run before any AI visibility work, with a free way to test each step yourself.

Key takeaways

  • OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers.[2]
  • Since 1 July 2025 Cloudflare asks every new domain at sign-up whether to allow AI crawlers.[3]
  • Vercel’s study found that none of the major AI crawlers render JavaScript.[4]
  • Google says no new AI text files are needed to appear in AI Overviews or AI Mode.[5]
  • From 1 July to 28 September 2026 Pavlo Zhydkykh’s own site got 91 visitors from ChatGPT and about 18 from Google.

What is AI readability?

AI readability is whether the crawlers behind ChatGPT and other assistants may fetch a website and can read the words on its pages. AI readability depends on robots.txt, on firewall and CDN bot settings, and on whether the words are in the HTML the server sends or appear only after JavaScript runs.

Wikipedia: Web crawler

“I never start AI visibility work before this check. Clever copy is worthless if ChatGPT gets a 403 or an empty page.”

Pavlo Zhydkykh · Web designer and developer since 2016

Last checked · 7 min read

Why I run this check before any AI visibility work

From 1 July to 28 September 2026 my own site got 91 visitors from ChatGPT and about 18 from Google, according to my site analytics. For a small B2B site that made AI answers a real channel, and it changed the order of my work: before I touch copy, schema or llms.txt for a client, I check that ChatGPT can reach and read the site at all. Everything else is wasted if it cannot.

The check takes about 10 minutes, needs no paid tool and finds the problems I see most often on startup sites. When I ran the same kind of check on all 50 companies of the 2026 Forbes Fintech 50, most sites passed, but the failures were exactly the ones a team never notices from the inside.

The 10-minute check at a glance

CheckHow to run itPass
1. robots.txtOpen yoursite.com/robots.txtNo Disallow for OAI-SearchBot or ChatGPT-User
2. FirewallYour CDN's bot or AI crawler settingsAI search crawlers allowed
3. Raw HTMLView page source, search for your headlineHeadline, pricing and key facts are in the source
4. First paragraphRead only the first paragraph of key pagesIt says what you sell, for whom and roughly what it costs
5. ChatGPT itselfAsk what your company does, then ask buyer questionsThe description is right and you appear for some questions

Run them in this order. A site that fails check 1 or 2 is invisible no matter how good its copy is, and checks 4 and 5 only make sense once the crawler can read the page.

Check 1: does robots.txt let ChatGPT in?

OpenAI runs several crawlers with different jobs, and robots.txt treats them separately. The one that matters for ChatGPT search is OAI-SearchBot.

CrawlerWhat OpenAI uses it forIf you block it
OAI-SearchBotFinding pages for ChatGPT searchYour site is not shown in ChatGPT search answers
GPTBotCollecting content that may train its modelsYour content stays out of training; search is not affected
ChatGPT-UserFetching a page when a user asks ChatGPT to open itOpenAI says robots.txt rules may not apply to these visits

OpenAI's crawler documentation is explicit: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and a robots.txt change can take about 24 hours to reach its search systems. So open yoursite.com/robots.txt and look for any group that names OAI-SearchBot, or a User-agent: * group with Disallow: /. Blocking GPTBot is a legitimate choice if you do not want your content used for training. It does not remove you from search, and in my Fintech 50 check 3 of 46 sites made exactly that choice while keeping every AI search crawler allowed.

Check 2: is your CDN or firewall blocking AI bots?

robots.txt is a request. A firewall is a wall, and it can block ChatGPT while robots.txt says yes. Since 1 July 2025 Cloudflare asks every new domain at sign-up whether to allow AI crawlers, as its press release describes. A founder who clicked through that screen in a hurry may have blocked them without knowing.

  • In Cloudflare, open the domain's AI crawler and bot settings and see which crawlers are blocked.
  • On other CDNs and hosts, look for bot protection or "block AI scrapers" switches.
  • Check the firewall's event log for requests from OAI-SearchBot that got a 403 or a challenge page.

A quick test from your own computer:

curl -A "OAI-SearchBot" https://yoursite.com/

If it returns a challenge page instead of your HTML, something is blocking by user agent. A clean result is not the full answer, because firewalls that verify bots by IP address treat the real crawler differently from your laptop, so the event log stays the final word.

Check 3: is your text in the HTML before JavaScript runs?

This is the failure I find most often on startup sites, because the page looks perfect in a browser. Vercel's study of AI crawlers found that none of the major ones render JavaScript: ChatGPT's and Claude's crawlers download script files but do not run them. They read the HTML your server sends, and nothing a script adds later.

To test it, open your homepage, choose View Page Source and search for your headline, your price and one sentence from the middle of the page. If they are missing, an AI crawler sees a near empty page. In the Fintech 50 check, 44 of 47 homepages had their main text in the HTML. Two did not: Aven sent 16 words before JavaScript ran against 2,015 on the rendered page, and Securitize 14 against 1,338.

The fix is server rendering or static HTML, not a new tool. Framer and Astro both publish real HTML. My own site runs on Astro, which ships no JavaScript by default, so every word is in the source before any script runs.

Check 4: can a stranger tell what you sell from the first paragraph?

A crawler that can read your page still has to understand it. Read only the first paragraph of your homepage and your main product or service page. Would a stranger know what you sell, who it is for and roughly what it costs? Many startup homepages open with a slogan that only makes sense once you already know the product.

I open every key page with two or three plain sentences that answer exactly that, with the company name in them. It is the part of a page an assistant is most likely to quote, and it costs nothing to write.

Check 5: ask ChatGPT what your company does

Now ask ChatGPT, with search on, what your company does. Compare its answer with your own one line description. A wrong or outdated answer usually means the web describes you in different words in different places, or that an old page still ranks for your name.

Then write 10 questions a buyer would type before they know you exist, such as "best payroll API for startups" rather than your brand name, and run them in ChatGPT and Perplexity. Note who gets named and which pages the answers cite. That list of cited pages is where you need to appear next.

Does passing the check make you rank in ChatGPT?

No. People search "how to rank in ChatGPT" as if it worked like Google rankings, but this check only removes the reasons ChatGPT cannot use your site. Whether it names you depends on how clearly your pages answer buyer questions and on what other sites say about you: comparison articles, review sites, directories and forum threads.

Two things I would not spend the first week on. An llms.txt file is cheap and I publish one, but Google says you do not need new AI text files to appear in its AI features. And schema markup helps machines read facts, but it cannot rescue a page whose text the crawler never received. For the bigger picture, my guide to generative engine optimization covers what comes after access.

What I fix first when a site fails

The order is always the same: unblock first, then make the words readable, then make them clear.

  1. Blocked crawlers. A robots.txt line or a firewall switch, usually fixed in minutes.
  2. Text that needs JavaScript. Server rendering or a rebuild on a stack that publishes HTML, such as Framer or Astro + Sanity.
  3. Vague first paragraphs. A rewrite of the opening lines of 5 to 10 key pages.
  4. Missing mentions. The slow part: getting onto the pages ChatGPT already cites.

If you run these checks with an AI coding agent, my count of the Claude Code skills I actually use for web design shows which skills earn their place and which never get called.

If you would rather have someone run all of this for you, my free AI visibility check asks 20 buyer questions across 5 AI assistants and reports what each one says about your company, and AI visibility work fixes what the check finds.

If you need this built

A landing page from Pavlo starts at $1,200 (1 to 2 weeks), a multi-page site at $3,000 (2 to 4 weeks). The price is fixed before work starts.

See all prices

Frequently asked questions

Can ChatGPT read my website?

It can if robots.txt and your firewall let OAI-SearchBot in and your text is in the HTML the server sends. Open yoursite.com/robots.txt, check your CDN's AI crawler settings and view the page source of your homepage to confirm all three.

Which ChatGPT crawler should I allow?

Allow OAI-SearchBot, the crawler OpenAI uses for ChatGPT search. Blocking GPTBot only keeps your content out of model training. ChatGPT-User fetches pages when a user asks, and OpenAI says robots.txt rules may not apply to it.

Does ChatGPT run JavaScript on my pages?

Not when its crawler reads your site. Vercel found that the major AI crawlers, including ChatGPT's, download JavaScript files but do not execute them, so text that only appears after scripts run is invisible to them.

How long until ChatGPT sees a robots.txt change?

OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Firewall changes take effect as soon as you save them.

How do I rank in ChatGPT?

There is no ranking you can buy or switch on. First make sure ChatGPT can read your site, then open key pages with plain answers to buyer questions and get mentioned on the comparison articles, review sites and threads that ChatGPT cites for your category.

Do I need llms.txt for ChatGPT to read my site?

No. ChatGPT reads your normal pages. Google says no special AI text files are needed for its AI features either. llms.txt is cheap to add, but fix blocked crawlers and JavaScript-only text first.

Pavlo Zhydkykh

Written by

Independent designer and developer in Eastern Europe. He builds websites on Framer and on Astro with Sanity for AI, fintech and B2B startups, and writes these guides from client work.

First published · Updated

Start a project
Halftone mouse cursor Free tool Does ChatGPT recommend you?

Free report in 48 hours · 5 AI engines

Checklist

5 checks that show whether ChatGPT can read a website

The order in which Pavlo Zhydkykh runs the checks, from the failures that make a site invisible to the ones that make a page unclear.

  1. 1. robots.txt

    No Disallow rule for OAI-SearchBot, the crawler behind ChatGPT search.

  2. 2. Firewall and CDN

    AI search crawlers allowed in the bot settings and absent from the block log.

  3. 3. Raw HTML

    The headline, pricing and key facts appear in View Page Source.

  4. 4. First paragraph

    A stranger can tell what the company sells, for whom and roughly at what price.

  5. 5. ChatGPT’s answer

    ChatGPT describes the company correctly and names the company for some buyer questions.

Fit check

Is An AI readability check on a startup website
right for you?

A good fit if

  • ChatGPT describes the company wrongly or not at all, and nobody knows why.
  • The site is a JavaScript app, so the text may not be in the HTML.
  • The domain sits behind Cloudflare or another CDN with bot protection switched on.
  • A redesign or platform move is planned, and Pavlo Zhydkykh makes the new site readable by AI crawlers from day one.

Not the right fit if

  • The goal is a guaranteed mention in ChatGPT: access makes a mention possible, not certain.
  • The site already passes all five checks and the gap is mentions on other sites.
  • The content sits behind a login, which no crawler reads either way.
  • The question is about AI training rights, which is a legal decision, not a technical one.

Compared

OAI-SearchBot vs GPTBot

Two OpenAI crawlers, two robots.txt decisions, as OpenAI describes the two and as Pavlo Zhydkykh advises.

OAI-SearchBot vs GPTBot
CriterionOAI-SearchBotGPTBot
Used forChatGPT searchTraining OpenAI’s models
If blockedThe site is not shown in ChatGPT search answersContent stays out of training
Effect on ChatGPT searchThe site drops outNone
Pavlo Zhydkykh advisesKeep allowedThe owner decides

In their words

ChatGPT and AI crawlers, in the words
of the AI companies and clients

Word for word, with the source linked: research, official documentation and the people Pavlo Zhydkykh has worked for.

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

OpenAIOverview of OpenAI crawlersSource ↗

“The results consistently show that none of the major AI crawlers currently render JavaScript.”

VercelThe rise of the AI crawlerSource ↗

“You don't need to create new machine readable files, AI text files, or markup to appear in these features.”

Google Search CentralAI features and your websiteSource ↗

“One of the best contractors I have worked with. Really thoughtful about communication. Went above and beyond in delivering the results that our team needs. Would 100% hire again for our projects.”

Client on UpworkFramer development · Framer Specialist contract on UpworkSource ↗

“Pavlo was a pleasure to work with on this project. The web and mobile designs he created were amazing. His communication was prompt which made for a smooth project. I highly recommend his services.”

Shawntae SpencerClient, QuickQuick Pay · Web and mobile design on UpworkSource ↗

“The collaboration went smoothly and the work was delivered as agreed. Communication was timely, and the final result aligned with the initial expectations. Overall, a very positive and rewarding experience!”

Client on UpworkE-commerce website · Home page redesign on UpworkSource ↗
Pavlo Zhydkykh

About the author

Pavlo Zhydkykh

Pavlo Zhydkykh is a web designer and developer in Eastern Europe who has built websites for AI, fintech and B2B startups since 2016, designing and building each project himself on Framer or Astro + Sanity. Clients include MStudio Asia, Silicon Data, Inturai, Accolade Pro, NimbleStax and Sancts.

  • Top Rated on Upwork
  • 100% Job Success
  • 10 years since 2016
  • 50+ sites shipped
  • 90+ PageSpeed at launch
Kulparkivska St. 200, Lviv 79021, Ukraine +380 93 610 62 68 pavlozhydkykh@gmail.com

Page reviewed by Pavlo Zhydkykh · Updated

AI visibility check

  1. 1Website
  2. 2Business
  3. 3Contact
  4. 4Check

Does ChatGPT recommend your website?

Pavlo Zhydkykh asks ChatGPT, Perplexity, Gemini, Claude and Grok 20 questions your buyers ask, and shows where you show up.

Free · report in 48 hours