Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50
AI crawler access is whether the bots behind ChatGPT, Claude and Perplexity may fetch a website and can read its text. In “Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50”, Pavlo Zhydkykh tests all 50 homepages for blocked crawlers, llms.txt, text without JavaScript, schema and sitemaps.
Pavlo ZhydkykhUpdated · 7 min read
Short answer
An AI assistant can only recommend a company whose website it is allowed to fetch and able to read. On 11 November 2026 I checked the homepages of all 50 companies on the 2026 Forbes Fintech 50 for exactly that. None of the 46 robots.txt files I could read blocks the crawlers behind ChatGPT search, Claude or Perplexity, and almost half of the sites already publish an llms.txt file. Two homepages send almost no text to a crawler that does not run JavaScript. Here are the numbers, the method and the data as a CSV.
Key takeaways
- In 2023 Originality.AI found that 306 of the 1,000 most visited websites blocked GPTBot in robots.txt.[1]
- OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers.[3]
- Google says AI Overviews and AI Mode have no additional requirements and need no special AI text files.[5]
- Vercel found that none of the major AI crawlers it studied render JavaScript.[6]
- On 11 November 2026 Pavlo Zhydkykh found that none of 46 Fintech 50 robots.txt files blocks an AI search crawler, and 23 of 47 sites publish llms.txt.
What is AI crawler access?
AI crawler access is whether the bots that AI companies use to train models and to answer questions may fetch a website and can read its text. AI crawler access depends on robots.txt, on firewall and CDN settings, and on whether the text is in the HTML or appears only after JavaScript runs.
“Block AI training if you must, never AI search. And put your words in the HTML, because major AI crawlers never run your scripts.”
Last checked · 7 min read
What I checked on the Fintech 50 websites
Forbes published its eleventh Fintech 50 in February 2026: 50 private fintech companies, from business banking and payments to crypto, insurance and real estate. I took the list from Forbes and checked every company's homepage the same way, from one computer, on 11 November 2026. Fintech is a good test group because these companies sell to buyers who research carefully, and more of that research now starts in ChatGPT, Claude, Perplexity or Google's AI Mode.
I ran five checks on every site:
- Does robots.txt allow or block 12 named AI crawlers, six that collect training data and six that fetch pages for AI search and answers?
- Is there an llms.txt file at the root of the site?
- Is the main text of the homepage in the HTML the server sends, or does it appear only after JavaScript runs?
- Does the homepage carry JSON-LD structured data?
- Is there an XML sitemap?
Three sites answered my request with an error, Highnote and Persona with 403 and Kalshi with 429, so the results below cover 47 homepages. One more, Cardless, has no robots.txt at all, which means every crawler is allowed.
The results at a glance
| Sites | Check |
|---|---|
| 47 of 50 (94%) | Homepage answered a plain request |
| 9 of 46 (20%) | robots.txt names at least one AI crawler |
| 3 of 46 (7%) | Blocks at least one AI training crawler |
| 0 of 46 (0%) | Blocks at least one AI search crawler |
| 23 of 47 (49%) | Publishes llms.txt |
| 44 of 47 (94%) | Main text in the HTML without JavaScript |
| 31 of 47 (66%) | JSON-LD on the homepage |
| 45 of 47 (96%) | XML sitemap |
| 30 of 47 (64%) | Passes all four basics |
The four basics are AI search crawlers allowed, main text without JavaScript, JSON-LD and an XML sitemap. The full table, one row per company and one column per crawler, is free to use under CC BY 4.0, with a link to this page: download fintech-50-ai-crawlers-2026.csv.
Which AI crawlers fintech websites block
Almost none. 37 of the 46 robots.txt files I could read do not mention a single AI crawler by name, so their general rules apply and AI crawlers are treated like any other bot. Three companies block AI training crawlers:
- Valon blocks all six training crawlers I checked: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.
- Rillet blocks five of them, all except Meta-ExternalAgent.
- Nala blocks only CCBot, the crawler of Common Crawl.
The wider web is much stricter. According to the robots.txt article on Wikipedia, Originality.AI found in 2023 that 306 of the thousand most visited websites blocked GPTBot. Among the Fintech 50 it is 2 of 46.
Blocking AI training is not blocking AI search
AI companies run separate crawlers for training and for search, and robots.txt can treat them differently. OpenAI's crawler documentation says GPTBot collects content that may be used to train its models, while OAI-SearchBot surfaces websites in ChatGPT search, and a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic draws the same line between ClaudeBot and Claude-SearchBot.
Rillet uses that split on purpose. Its robots.txt disallows GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, and names OAI-SearchBot, ChatGPT-User and PerplexityBot to allow them. The content stays out of training sets and stays open to AI search. Six more companies name AI crawlers only to welcome them: Bilt, Found, Human Interest, Justt, Maybern and Relay. No company on the list blocks a search crawler.
My own site takes the open route. The robots.txt of pzhydkykh.com lists 14 crawlers by name, AI assistants included, and allows every one of them, because AI answers already bring me visitors.
Can a crawler read the homepage without JavaScript?
This is the check that matters most for AI answers, and the one site owners look at least. Vercel found that none of the major AI crawlers it studied render JavaScript: they read the HTML the server sends and nothing more. Googlebot does render JavaScript, which is why a site can look fine in Google and still be close to empty for an AI crawler.
For each homepage I compared two versions: the raw HTML from a plain request, and the page after Chrome ran every script. If the visible headline and most of the first paragraphs were already in the raw HTML, the page passed. 44 of 47 passed. The three that did not:
- Aven: 16 words in the raw HTML and 2,015 on the rendered page. The headline and the offer appear only after scripts run.
- Securitize: 14 words in the raw HTML and 1,338 rendered.
- Human Interest: the headline is in the HTML, but 4 of the 5 paragraphs I sampled from the rendered page are not in the raw HTML in the same words, so part of the page is filled in by script.
Plain HTML first was the main reason I moved my own site from Framer to Astro in October 2026: Astro sends the full text with zero JavaScript by default. From July to September 2026 ChatGPT sent my site about five times more visitors than Google, 91 against about 18, so I treat the raw HTML as the version that AI assistants actually read.
How many fintech websites have llms.txt?
23 of 47, or 49%. That surprised me, because llms.txt is still only a proposal: a Markdown file at the site root with a short summary and links to the pages that matter most. Stripe, Plaid, Column, Increase, Monarch, Polymarket and Socure all publish one. Valon's file says it was generated by the All in One SEO plugin for WordPress, so CMS plugins now add it for site owners who never asked.
Keep the expectations straight. Google says there are no additional requirements to appear in AI Overviews or AI Mode, and no special AI text files are needed. llms.txt costs little and may help assistants that read it, but it does not replace readable HTML. My llms.txt guide covers what the file does and does not do.
Schema and sitemaps: the basics most sites have
45 of 47 sites have an XML sitemap; Cardless and Hyperliquid do not. JSON-LD is less common: 31 of 47 homepages carry it, and 29 describe the company as an Organization or a more specific type. Mercury marks itself up as an Organization and a FinancialService with an OfferCatalog, while Plaid, Ramp and Stripe use Organization. Sixteen homepages have no JSON-LD at all.
Markup does not make a page appear in AI answers, but it states the basic facts, such as the name, logo and official profiles, in a form every system reads the same way. My schema markup guide lists the types a SaaS or fintech site needs.
What I would copy for a fintech startup site
The Fintech 50 set a clear baseline. If your site is below it, a company on this list is easier for an AI assistant to read than you are. This is the order I would fix things in: To run the same checks on your own site in about 10 minutes, follow my guide Can ChatGPT read your website?
- Put the main text in the HTML. Open your homepage, view the page source and search for your headline. If it is not there, AI crawlers do not see it.
- Allow AI search crawlers. To keep content out of training, block GPTBot, ClaudeBot and Google-Extended by name and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed, the way Rillet does.
- Add Organization markup to the homepage and keep an XML sitemap.
- Add llms.txt last, once the HTML is right.
This setup, with structured data, llms.txt and AI crawler access, is included in every site I build on Framer or Astro + Sanity; the agent-ready websites page lists all of it. To see whether ChatGPT and other assistants name your company today, start with the free AI visibility check.
How I ran the check, and its limits
- Sample: the 50 companies on the Forbes Fintech 50 2026, from Forbes' own list data. I matched each company to its website and confirmed the match by the homepage title.
- Date and setup: 11 November 2026, one request per file from one IP address with a desktop Chrome user agent, plus one Chrome render of each homepage.
- robots.txt: read the way Google documents it. The most specific user agent group applies, the longest matching rule wins and Allow wins a tie. A crawler counts as blocked when the homepage is disallowed for it.
- llms.txt: counted when /llms.txt returned a text file rather than an HTML page.
- Limits: robots.txt is a request, not a wall, and a CDN or firewall can treat real AI crawlers differently from my request. Three sites refused my request outright. I checked homepages only, and inner pages can differ.
Websites change, so treat these numbers as a snapshot of 11 November 2026.
If you need this built
A landing page from Pavlo starts at $1,200 (1 to 2 weeks), a multi-page site at $3,000 (2 to 4 weeks). The price is fixed before work starts.
See all pricesFrequently asked questions
Do fintech companies block ChatGPT from their websites?
Not from search. On 11 November 2026 none of the 46 Forbes Fintech 50 sites with a readable robots.txt blocked OAI-SearchBot or ChatGPT-User. Two companies, Rillet and Valon, block GPTBot, the crawler OpenAI uses to collect training data.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot finds pages for ChatGPT search. Blocking GPTBot keeps a site out of training; blocking OAI-SearchBot keeps it out of ChatGPT search answers, according to OpenAI's crawler documentation.
Does a website need llms.txt to appear in AI answers?
No. Google says AI Overviews and AI Mode have no additional requirements beyond normal SEO best practices. Still, 23 of the 47 Fintech 50 sites I could check publish an llms.txt file, because it is cheap to add.
Why does JavaScript matter for AI crawlers?
Vercel found that the major AI crawlers do not render JavaScript. They read the HTML the server sends, so text that appears only after scripts run is invisible to them, even when Google renders the same page without trouble.
How can I check my own website the same way?
Open your homepage, view the page source and search for your headline. Then open yourdomain.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. For a full report, I run a free AI visibility check that also asks the assistants about your company.
Can I use this data?
Yes. The CSV has one row per company with every check. Please link to this page as the source and give the date, 11 November 2026, because websites change.
Free tool Does ChatGPT recommend you? Free report in 48 hours · 5 AI engines
Checklist
5 checks I ran on every Fintech 50 website
The five checks Pavlo Zhydkykh ran on all 50 homepages on 11 November 2026, with the result for the 47 sites that answered.
1. robots.txt for 12 AI crawlers
Six training and six search crawlers: 3 of 46 sites block training crawlers, none blocks a search crawler.
2. llms.txt
23 of 47 sites publish one, including Stripe, Plaid, Column and Monarch.
3. Text without JavaScript
44 of 47 homepages have their main text in the HTML; Aven and Securitize send almost none.
4. JSON-LD markup
31 of 47 homepages carry JSON-LD, and 29 describe the company as an Organization or a more specific type.
5. XML sitemap
45 of 47 sites have one; Cardless and Hyperliquid do not.
Fit check
Is An AI crawler access check on a fintech website
right for you?
A good fit if
- The homepage text appears only after JavaScript runs, so AI crawlers get an almost empty page.
- Content should stay out of AI training but stay visible in AI search answers.
- Nobody has read robots.txt since launch; Pavlo Zhydkykh lists which bots the file blocks.
- A redesign or a platform move is planned, and Pavlo Zhydkykh makes the new site readable by AI assistants from day one.
Not the right fit if
- The goal is a guaranteed mention in ChatGPT: access makes a mention possible, not certain.
- The site already passes all four basics and the gap is content or mentions on other sites.
- The question is a legal one about AI training rights: that needs a lawyer, not a crawler check.
- The pages sit behind a login, which no crawler reads either way.
Compared
AI training crawlers vs AI search crawlers
Two kinds of AI bots, two robots.txt decisions, and what the Fintech 50 chose when Pavlo Zhydkykh checked on 11 November 2026.
| Criterion | Training crawlers | AI search crawlers |
|---|---|---|
| Examples | GPTBot, ClaudeBot, CCBot, Google-Extended | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| What they do | Collect text that may train models | Fetch pages for search answers |
| If blocked | Content stays out of future training | The site drops out of those search answers |
| Fintech 50 sites blocking | 3 of 46 | 0 of 46 |
| Pavlo Zhydkykh advises | The owner decides | Keep allowed |
In their words
AI crawlers, in the words
of the AI companies and clients
Word for word, with the source linked: research, official documentation and the people Pavlo Zhydkykh has worked for.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
“There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.”
“The results consistently show that none of the major AI crawlers currently render JavaScript.”
“One of the best contractors I have worked with. Really thoughtful about communication. Went above and beyond in delivering the results that our team needs. Would 100% hire again for our projects.”
“Pavlo was a pleasure to work with on this project. The web and mobile designs he created were amazing. His communication was prompt which made for a smooth project. I highly recommend his services.”
“The collaboration went smoothly and the work was delivered as agreed. Communication was timely, and the final result aligned with the initial expectations. Overall, a very positive and rewarding experience!”
Sources and further reading
- [1] Wikipediarobots.txt
- [2] ForbesThe 50 hottest fintech startups in 2026 (Fintech 50)
- [3] OpenAIOverview of OpenAI crawlers
- [4] AnthropicDoes Anthropic crawl data from the web, and how can site owners block the crawler?
- [5] Google Search CentralAI features and your website
- [6] VercelThe rise of the AI crawler
About the author
Pavlo Zhydkykh
Pavlo Zhydkykh is a web designer and developer in Eastern Europe who has built websites for AI, fintech and B2B startups since 2016, designing and building each project himself on Framer or Astro + Sanity. Clients include MStudio Asia, Silicon Data, Inturai, Accolade Pro, NimbleStax and Sancts.
- Top Rated on Upwork
- 100% Job Success
- 10 years since 2016
- 50+ sites shipped
- 90+ PageSpeed at launch
About Pavlo LinkedInUpworkContra Privacy policy
Page reviewed by Pavlo Zhydkykh · Updated
In this topic
AI search
6 articles
- 01 Can ChatGPT Read Your Website? 5 Checks in 10 Minutes 7 min
- 02 Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50 You are here
- 03 What Is Schema Markup? The Types a SaaS Site Needs in 2026 Next
- 04 GEO vs SEO vs AEO: What Actually Changes for a SaaS Website 6 min
- 05 What Is Generative Engine Optimization (GEO)? A Guide for B2B SaaS Teams 6 min
- 06 llms.txt in 2026: What It Is, an Example and What It Does 6 min