Skip to content
Contact me

Agent-ready websites

Agent-ready websites are sites that AI agents and crawlers can read and use: full text in server-rendered HTML, JSON-LD schema, llms.txt and AI crawler access in robots.txt. Pavlo Zhydkykh builds this into every startup site he ships; an existing site gets a fixed quote after a free check, with the report in 48 hours.

Does ChatGPT recommend your website?

Free report in 48 hours · 5 AI engines

HTML
Server-rendered
Schema
JSON-LD
Machine files
llms.txt, sitemap
AI crawlers
Allowed in robots.txt
In every build
Included
Existing sites
Fixed quote

Key takeawaysUpdated By

Key takeaways

  • An agent-ready website serves the full text in the HTML, describes the business in schema and llms.txt, and lets AI crawlers in through robots.txt.
  • OpenAI states that sites opted out of OAI-SearchBot in robots.txt will not be shown in ChatGPT search answers.[3]
  • The llms.txt proposal adds a Markdown file at /llms.txt that gives language models a short, curated overview of a site.[2]
  • Most AI crawlers read the raw HTML without running JavaScript, so text that loads later never reaches those crawlers.
  • Agent-ready setup is included in every site Pavlo Zhydkykh builds, including his own pzhydkykh.com on Astro + Sanity with llms.txt and JSON-LD schema.

What is an agent-ready website?

An agent-ready website is a site that AI agents and crawlers can read and use without a human: the text is in the HTML, the facts are in structured data and llms.txt, and the crawlers are allowed in. Agent-readiness matters because assistants such as ChatGPT and Perplexity fetch pages to answer users, and anything an assistant cannot parse is left out of the answer.

Wikipedia: Web crawler

Pavlo Zhydkykh

“In my checks, sites fail agents for boring reasons: a blocked crawler or text behind JavaScript. I fix those before anything clever.”

Pavlo Zhydkykh · Web designer and developer since 2016

Fit check

Is Agent-ready websites
right for you?

A good fit if

  • AI, fintech and B2B startups whose buyers research vendors through ChatGPT, Perplexity or an AI agent.
  • Sites built as a single-page app, where the HTML is empty until JavaScript runs.
  • Sites whose robots.txt or firewall blocks GPTBot, ClaudeBot or PerplexityBot, often by accident.
  • Teams launching on Framer or Astro + Sanity who want the machine files set up from the start.

Not the right fit if

  • You want to keep AI crawlers out on purpose, for example to protect paid content.
  • You need agents to log in, pay or place orders through an API: that is a product integration, outside a marketing site build.
  • You expect llms.txt alone to change what assistants say: the file helps assistants read the site but does not decide the answer.

Compared

What an AI crawler receives

The same page, as received by a crawler that does not run JavaScript.

What an AI crawler receives
CriterionJavaScript-first siteAgent-ready site
Page textAn empty container until scripts runFull text in the first HTML response
Prices and factsIn images, tabs or slidersIn plain text and in schema
Site summaryNonellms.txt at the root
Crawler accessrobots.txt never reviewedAI crawlers listed and allowed
FormsPlaceholder text onlyLabeled fields and named buttons

Checklist

6 checks for an agent-ready site

Each check takes a minute with a browser or a free tool. Pavlo runs all six before every launch.

  1. 1. robots.txt lets the AI crawlers in

    Open /robots.txt. GPTBot, ClaudeBot and PerplexityBot should not fall under a "Disallow: /" rule.

  2. 2. The text is in the page source

    Press Ctrl+U and search for a sentence from the hero. If the sentence is missing, crawlers that skip JavaScript see an empty page.

  3. 3. llms.txt answers the basics

    Open /llms.txt. The file should say in a few lines who you are, what you sell, for whom and for how much.

  4. 4. Schema validates

    Paste the page into the Google Rich Results Test. Organization, Service and FAQPage should parse without errors.

  5. 5. Forms and buttons have real labels

    An agent filling in a contact form reads field labels and button names. A placeholder and a "Submit" button tell the agent very little.

  6. 6. The firewall lets the bots through

    Some CDN and hosting settings block AI crawlers by default. Check the bot settings, not only robots.txt.

How it works

From check to agent-ready, step by step

In a new build this runs inside the build timeline. For an existing site the work starts with the free check, then a fixed quote.

From check to agent-ready, step by step, step by step
#StepWhenWhat happensYou get
01 Free AI visibility check Within 48 hours 20 buyer questions asked in 5 AI engines, plus a look at robots.txt, the raw HTML and the schema. A report on what AI engines see today
02 Fix list and fixed quote After the report A short call on the gaps and the pages that matter most for sales. A fixed price and timeline in writing
03 Crawler access Set in the quote robots.txt and CDN bot settings opened for the AI crawlers you choose. GPTBot, ClaudeBot and PerplexityBot let in
04 Server-rendered text Set in the quote Key text, prices and FAQ answers moved into the first HTML response. Pages readable without JavaScript
05 Schema and llms.txt Set in the quote JSON-LD for the company, services, offers and FAQ, plus an llms.txt summary of the site. Facts machines do not have to guess
06 Labels and structure Set in the quote One H1, ordered headings, labeled form fields and named buttons. A site an agent can move through
07 Launch and validation Launch day Changes go live and are checked in the raw HTML, the Rich Results Test and robots.txt. A validated site, then $30 an hour when needed

In their words

What the docs say about agents,
what clients say about Pavlo

Word for word, with the source linked: research, official documentation and the people Pavlo Zhydkykh has worked for.

“Agents now use websites constantly: a coding agent fetches a library's documentation to get an API call right, and a chat assistant with search reads pages to answer questions about a product.”

Jeremy Howardllmstxt.org: The /llms.txt fileSource ↗

“AI agents are autonomous systems that can perform tasks on behalf of people, such as booking a reservation or comparing product specifications.”

Google Search CentralOptimizing for generative AI searchSource ↗

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

OpenAIOverview of OpenAI crawlersSource ↗

“It was an absolute pleasure working with Pavlo. The quality of work he delivered was AMAZING.”

Ed ClarkeFounder, Inturai · Website design and SEOSource ↗

“He is amazing in what he contributes, both design-wise and also from the ideas.”

Oliver BockOwner, The B2B Answer · Website design, 2025Source ↗

“One of the best contractors I have worked with. Really thoughtful about communication. Would 100% hire again.”

Remi LabelleCOO, Accolade Pro · Web app designSource ↗

FAQ

Questions about
Agent-ready websites

Still unsure? Ask your AI about Pavlo Zhydkykh:

ChatGPTPerplexityClaudeGeminiGrok

Q.01

How much does an agent-ready setup cost?

Nothing extra on a new site: the setup is part of every build Pavlo ships. An existing site gets a fixed quote after the free check, and small fixes cost $30 an hour.

Q.02

How long does an agent-ready setup take?

The timeline is part of the fixed quote and depends on how the site renders the page text. Crawler access, schema and llms.txt are the quick parts; moving text out of JavaScript takes the longest.

Q.03

What do I need to provide?

Editor access to the site and to the domain or CDN settings, where bot rules often live. Pavlo writes the llms.txt summary and the schema from your existing pages and checks the facts with you.

Q.04

Who owns the setup?

You do. The robots.txt rules, llms.txt and the schema are plain files and code in your own site, with no plugin and no subscription.

Q.05

Is letting AI crawlers read the site risky?

AI crawlers can read and quote anything public, which is the point of a marketing site. Pages you want to keep out, such as gated content, stay blocked, and you choose which crawlers get in.

Q.06

What happens after launch?

Pavlo checks the raw HTML, the schema and robots.txt on the live domain. When prices or services change, llms.txt and the schema need the same update, which costs $30 an hour with no retainer.

Sources and further reading

  1. [1] WikipediaWeb crawler
  2. [2] llmstxt.orgThe /llms.txt file
  3. [3] OpenAIOverview of OpenAI crawlers
  4. [4] PerplexityPerplexity crawlers
  5. [5] Google Search CentralJavaScript SEO basics
  6. [6] Google Search CentralOptimizing for generative AI search
Pavlo Zhydkykh

About the author

Pavlo Zhydkykh

Pavlo Zhydkykh is a web designer and developer in Eastern Europe who has built websites for AI, fintech and B2B startups since 2016, designing and building each project himself on Framer or Astro + Sanity. Clients include MStudio Asia, Silicon Data, Inturai, Accolade Pro, NimbleStax and Sancts.

  • Top Rated on Upwork
  • 100% Job Success
  • 10 years since 2016
  • 50+ sites shipped
  • 90+ PageSpeed at launch
Kulparkivska St. 200, Lviv 79021, Ukraine +380 93 610 62 68 [email protected]

LinkedInUpworkContra Privacy policy

Page reviewed by Pavlo Zhydkykh · Updated

AI visibility check

  1. 1Website
  2. 2Business
  3. 3Contact
  4. 4Check

Does ChatGPT recommend your website?

Pavlo Zhydkykh asks ChatGPT, Perplexity, Gemini, Claude and Grok 20 questions your buyers ask, and shows where you show up.

Free · report in 48 hours