Skip to content

Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50

AI crawler access is whether the bots behind ChatGPT, Claude and Perplexity may fetch a website and can read its text. In “Can AI Assistants Read the Forbes Fintech 50 Websites? I Checked All 50”, Pavlo Zhydkykh tests all 50 homepages for blocked crawlers, llms.txt, text without JavaScript, schema and sitemaps.

Pavlo Zhydkykh

Updated · 7 min read

Fifty small squares in five rows on pale lilac under a thin blue bar, most solid black, some only outlined and three faint, one square for each company checked

Short answer

An AI assistant can only recommend a company whose website it is allowed to fetch and able to read. On 11 November 2026 I checked the homepages of all 50 companies on the 2026 Forbes Fintech 50 for exactly that. None of the 46 robots.txt files I could read blocks the crawlers behind ChatGPT search, Claude or Perplexity, and almost half of the sites already publish an llms.txt file. Two homepages send almost no text to a crawler that does not run JavaScript. Here are the numbers, the method and the data as a CSV.

Key takeaways

  • In 2023 Originality.AI found that 306 of the 1,000 most visited websites blocked GPTBot in robots.txt.[1]
  • OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers.[3]
  • Google says AI Overviews and AI Mode have no additional requirements and need no special AI text files.[5]
  • Vercel found that none of the major AI crawlers it studied render JavaScript.[6]
  • On 11 November 2026 Pavlo Zhydkykh found that none of 46 Fintech 50 robots.txt files blocks an AI search crawler, and 23 of 47 sites publish llms.txt.

What is AI crawler access?

AI crawler access is whether the bots that AI companies use to train models and to answer questions may fetch a website and can read its text. AI crawler access depends on robots.txt, on firewall and CDN settings, and on whether the text is in the HTML or appears only after JavaScript runs.

Wikipedia: robots.txt

“Block AI training if you must, never AI search. And put your words in the HTML, because major AI crawlers never run your scripts.”

Pavlo Zhydkykh · Web designer and developer since 2016

Last checked · 7 min read

What I checked on the Fintech 50 websites

Forbes published its eleventh Fintech 50 in February 2026: 50 private fintech companies, from business banking and payments to crypto, insurance and real estate. I took the list from Forbes and checked every company's homepage the same way, from one computer, on 11 November 2026. Fintech is a good test group because these companies sell to buyers who research carefully, and more of that research now starts in ChatGPT, Claude, Perplexity or Google's AI Mode.

I ran five checks on every site:

  1. Does robots.txt allow or block 12 named AI crawlers, six that collect training data and six that fetch pages for AI search and answers?
  2. Is there an llms.txt file at the root of the site?
  3. Is the main text of the homepage in the HTML the server sends, or does it appear only after JavaScript runs?
  4. Does the homepage carry JSON-LD structured data?
  5. Is there an XML sitemap?

Three sites answered my request with an error, Highnote and Persona with 403 and Kalshi with 429, so the results below cover 47 homepages. One more, Cardless, has no robots.txt at all, which means every crawler is allowed.

The results at a glance

SitesCheck
47 of 50 (94%)Homepage answered a plain request
9 of 46 (20%)robots.txt names at least one AI crawler
3 of 46 (7%)Blocks at least one AI training crawler
0 of 46 (0%)Blocks at least one AI search crawler
23 of 47 (49%)Publishes llms.txt
44 of 47 (94%)Main text in the HTML without JavaScript
31 of 47 (66%)JSON-LD on the homepage
45 of 47 (96%)XML sitemap
30 of 47 (64%)Passes all four basics

The four basics are AI search crawlers allowed, main text without JavaScript, JSON-LD and an XML sitemap. The full table, one row per company and one column per crawler, is free to use under CC BY 4.0, with a link to this page: download fintech-50-ai-crawlers-2026.csv.

Which AI crawlers fintech websites block

Almost none. 37 of the 46 robots.txt files I could read do not mention a single AI crawler by name, so their general rules apply and AI crawlers are treated like any other bot. Three companies block AI training crawlers:

  • Valon blocks all six training crawlers I checked: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.
  • Rillet blocks five of them, all except Meta-ExternalAgent.
  • Nala blocks only CCBot, the crawler of Common Crawl.

The wider web is much stricter. According to the robots.txt article on Wikipedia, Originality.AI found in 2023 that 306 of the thousand most visited websites blocked GPTBot. Among the Fintech 50 it is 2 of 46.

AI companies run separate crawlers for training and for search, and robots.txt can treat them differently. OpenAI's crawler documentation says GPTBot collects content that may be used to train its models, while OAI-SearchBot surfaces websites in ChatGPT search, and a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic draws the same line between ClaudeBot and Claude-SearchBot.

Rillet uses that split on purpose. Its robots.txt disallows GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, and names OAI-SearchBot, ChatGPT-User and PerplexityBot to allow them. The content stays out of training sets and stays open to AI search. Six more companies name AI crawlers only to welcome them: Bilt, Found, Human Interest, Justt, Maybern and Relay. No company on the list blocks a search crawler.

My own site takes the open route. The robots.txt of pzhydkykh.com lists 14 crawlers by name, AI assistants included, and allows every one of them, because AI answers already bring me visitors.

Can a crawler read the homepage without JavaScript?

This is the check that matters most for AI answers, and the one site owners look at least. Vercel found that none of the major AI crawlers it studied render JavaScript: they read the HTML the server sends and nothing more. Googlebot does render JavaScript, which is why a site can look fine in Google and still be close to empty for an AI crawler.

For each homepage I compared two versions: the raw HTML from a plain request, and the page after Chrome ran every script. If the visible headline and most of the first paragraphs were already in the raw HTML, the page passed. 44 of 47 passed. The three that did not:

  • Aven: 16 words in the raw HTML and 2,015 on the rendered page. The headline and the offer appear only after scripts run.
  • Securitize: 14 words in the raw HTML and 1,338 rendered.
  • Human Interest: the headline is in the HTML, but 4 of the 5 paragraphs I sampled from the rendered page are not in the raw HTML in the same words, so part of the page is filled in by script.

Plain HTML first was the main reason I moved my own site from Framer to Astro in October 2026: Astro sends the full text with zero JavaScript by default. From July to September 2026 ChatGPT sent my site about five times more visitors than Google, 91 against about 18, so I treat the raw HTML as the version that AI assistants actually read.

How many fintech websites have llms.txt?

23 of 47, or 49%. That surprised me, because llms.txt is still only a proposal: a Markdown file at the site root with a short summary and links to the pages that matter most. Stripe, Plaid, Column, Increase, Monarch, Polymarket and Socure all publish one. Valon's file says it was generated by the All in One SEO plugin for WordPress, so CMS plugins now add it for site owners who never asked.

Keep the expectations straight. Google says there are no additional requirements to appear in AI Overviews or AI Mode, and no special AI text files are needed. llms.txt costs little and may help assistants that read it, but it does not replace readable HTML. My llms.txt guide covers what the file does and does not do.

Schema and sitemaps: the basics most sites have

45 of 47 sites have an XML sitemap; Cardless and Hyperliquid do not. JSON-LD is less common: 31 of 47 homepages carry it, and 29 describe the company as an Organization or a more specific type. Mercury marks itself up as an Organization and a FinancialService with an OfferCatalog, while Plaid, Ramp and Stripe use Organization. Sixteen homepages have no JSON-LD at all.

Markup does not make a page appear in AI answers, but it states the basic facts, such as the name, logo and official profiles, in a form every system reads the same way. My schema markup guide lists the types a SaaS or fintech site needs.

What I would copy for a fintech startup site

The Fintech 50 set a clear baseline. If your site is below it, a company on this list is easier for an AI assistant to read than you are. This is the order I would fix things in: To run the same checks on your own site in about 10 minutes, follow my guide Can ChatGPT read your website?

  1. Put the main text in the HTML. Open your homepage, view the page source and search for your headline. If it is not there, AI crawlers do not see it.
  2. Allow AI search crawlers. To keep content out of training, block GPTBot, ClaudeBot and Google-Extended by name and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed, the way Rillet does.
  3. Add Organization markup to the homepage and keep an XML sitemap.
  4. Add llms.txt last, once the HTML is right.

This setup, with structured data, llms.txt and AI crawler access, is included in every site I build on Framer or Astro + Sanity; the agent-ready websites page lists all of it. To see whether ChatGPT and other assistants name your company today, start with the free AI visibility check.

How I ran the check, and its limits

  • Sample: the 50 companies on the Forbes Fintech 50 2026, from Forbes' own list data. I matched each company to its website and confirmed the match by the homepage title.
  • Date and setup: 11 November 2026, one request per file from one IP address with a desktop Chrome user agent, plus one Chrome render of each homepage.
  • robots.txt: read the way Google documents it. The most specific user agent group applies, the longest matching rule wins and Allow wins a tie. A crawler counts as blocked when the homepage is disallowed for it.
  • llms.txt: counted when /llms.txt returned a text file rather than an HTML page.
  • Limits: robots.txt is a request, not a wall, and a CDN or firewall can treat real AI crawlers differently from my request. Three sites refused my request outright. I checked homepages only, and inner pages can differ.

Websites change, so treat these numbers as a snapshot of 11 November 2026.

If you need this built

A landing page from Pavlo starts at $1,200 (1 to 2 weeks), a multi-page site at $3,000 (2 to 4 weeks). The price is fixed before work starts.

See all prices

Frequently asked questions

Do fintech companies block ChatGPT from their websites?

Not from search. On 11 November 2026 none of the 46 Forbes Fintech 50 sites with a readable robots.txt blocked OAI-SearchBot or ChatGPT-User. Two companies, Rillet and Valon, block GPTBot, the crawler OpenAI uses to collect training data.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot finds pages for ChatGPT search. Blocking GPTBot keeps a site out of training; blocking OAI-SearchBot keeps it out of ChatGPT search answers, according to OpenAI's crawler documentation.

Does a website need llms.txt to appear in AI answers?

No. Google says AI Overviews and AI Mode have no additional requirements beyond normal SEO best practices. Still, 23 of the 47 Fintech 50 sites I could check publish an llms.txt file, because it is cheap to add.

Why does JavaScript matter for AI crawlers?

Vercel found that the major AI crawlers do not render JavaScript. They read the HTML the server sends, so text that appears only after scripts run is invisible to them, even when Google renders the same page without trouble.

How can I check my own website the same way?

Open your homepage, view the page source and search for your headline. Then open yourdomain.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. For a full report, I run a free AI visibility check that also asks the assistants about your company.

Can I use this data?

Yes. The CSV has one row per company with every check. Please link to this page as the source and give the date, 11 November 2026, because websites change.

Pavlo Zhydkykh

Written by

Independent designer and developer in Eastern Europe. He builds websites on Framer and on Astro with Sanity for AI, fintech and B2B startups, and writes these guides from client work.

First published · Updated

Start a project
Halftone mouse cursor Free tool Does ChatGPT recommend you?

Free report in 48 hours · 5 AI engines

Checklist

5 checks I ran on every Fintech 50 website

The five checks Pavlo Zhydkykh ran on all 50 homepages on 11 November 2026, with the result for the 47 sites that answered.

  1. 1. robots.txt for 12 AI crawlers

    Six training and six search crawlers: 3 of 46 sites block training crawlers, none blocks a search crawler.

  2. 2. llms.txt

    23 of 47 sites publish one, including Stripe, Plaid, Column and Monarch.

  3. 3. Text without JavaScript

    44 of 47 homepages have their main text in the HTML; Aven and Securitize send almost none.

  4. 4. JSON-LD markup

    31 of 47 homepages carry JSON-LD, and 29 describe the company as an Organization or a more specific type.

  5. 5. XML sitemap

    45 of 47 sites have one; Cardless and Hyperliquid do not.

Fit check

Is An AI crawler access check on a fintech website
right for you?

A good fit if

  • The homepage text appears only after JavaScript runs, so AI crawlers get an almost empty page.
  • Content should stay out of AI training but stay visible in AI search answers.
  • Nobody has read robots.txt since launch; Pavlo Zhydkykh lists which bots the file blocks.
  • A redesign or a platform move is planned, and Pavlo Zhydkykh makes the new site readable by AI assistants from day one.

Not the right fit if

  • The goal is a guaranteed mention in ChatGPT: access makes a mention possible, not certain.
  • The site already passes all four basics and the gap is content or mentions on other sites.
  • The question is a legal one about AI training rights: that needs a lawyer, not a crawler check.
  • The pages sit behind a login, which no crawler reads either way.

Compared

AI training crawlers vs AI search crawlers

Two kinds of AI bots, two robots.txt decisions, and what the Fintech 50 chose when Pavlo Zhydkykh checked on 11 November 2026.

AI training crawlers vs AI search crawlers
CriterionTraining crawlersAI search crawlers
ExamplesGPTBot, ClaudeBot, CCBot, Google-ExtendedOAI-SearchBot, Claude-SearchBot, PerplexityBot
What they doCollect text that may train modelsFetch pages for search answers
If blockedContent stays out of future trainingThe site drops out of those search answers
Fintech 50 sites blocking3 of 460 of 46
Pavlo Zhydkykh advisesThe owner decidesKeep allowed

In their words

AI crawlers, in the words
of the AI companies and clients

Word for word, with the source linked: research, official documentation and the people Pavlo Zhydkykh has worked for.

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

OpenAIOverview of OpenAI crawlersSource ↗

“There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.”

Google Search CentralAI features and your websiteSource ↗

“The results consistently show that none of the major AI crawlers currently render JavaScript.”

VercelThe rise of the AI crawlerSource ↗

“One of the best contractors I have worked with. Really thoughtful about communication. Went above and beyond in delivering the results that our team needs. Would 100% hire again for our projects.”

Client on UpworkFramer development · Framer Specialist contract on UpworkSource ↗

“Pavlo was a pleasure to work with on this project. The web and mobile designs he created were amazing. His communication was prompt which made for a smooth project. I highly recommend his services.”

Shawntae SpencerClient, QuickQuick Pay · Web and mobile design on UpworkSource ↗

“The collaboration went smoothly and the work was delivered as agreed. Communication was timely, and the final result aligned with the initial expectations. Overall, a very positive and rewarding experience!”

Client on UpworkE-commerce website · Home page redesign on UpworkSource ↗
Pavlo Zhydkykh

About the author

Pavlo Zhydkykh

Pavlo Zhydkykh is a web designer and developer in Eastern Europe who has built websites for AI, fintech and B2B startups since 2016, designing and building each project himself on Framer or Astro + Sanity. Clients include MStudio Asia, Silicon Data, Inturai, Accolade Pro, NimbleStax and Sancts.

  • Top Rated on Upwork
  • 100% Job Success
  • 10 years since 2016
  • 50+ sites shipped
  • 90+ PageSpeed at launch
Kulparkivska St. 200, Lviv 79021, Ukraine +380 93 610 62 68 pavlozhydkykh@gmail.com

Page reviewed by Pavlo Zhydkykh · Updated

AI visibility check

  1. 1Website
  2. 2Business
  3. 3Contact
  4. 4Check

Does ChatGPT recommend your website?

Pavlo Zhydkykh asks ChatGPT, Perplexity, Gemini, Claude and Grok 20 questions your buyers ask, and shows where you show up.

Free · report in 48 hours