Is my Webflow site visible to AI crawlers?

The short version

Webflow is a static site generator with a visual editor on top. When you hit Publish it renders your pages to HTML and pushes them to a CDN. There is no server-side app deciding what to send you and no client-side framework assembling the page in the browser. A crawler that executes no JavaScript gets the same bytes a browser does.

We fingerprinted 35 candidate sites for Webflow markers, confirmed 14, then requested each with the OAI-SearchBot user-agent, executed no JavaScript, and read the raw bytes. Every confirmed site was readable.

Across this sample, tabs, sliders, accordions and CMS pagination did not hide text. The failures were a page with little text, a public staging domain, and an oversized response-header block on Webflow’s marketing site.

What we measured

For each site we fetched the homepage twice, once as OAI-SearchBot and once as Chrome, then read robots.txt, then followed a link into a CMS collection page. The commands below repeat those checks.

Webflow’s default robots.txt contains no rules at all

Three sites in the sample shipped Webflow’s untouched default, and it is a single line naming the sitemap. No User-agent block, no Disallow. Under the robots standard, a file with no rules allows everything. A fourth shipped "User-agent: *" followed by an empty "Disallow:", which is the explicit way of saying the same thing.

The default state of a Webflow site is open to AI crawlers. Custom rules can change that, so inspect the live robots.txt file for the site.

One site in the sample had a robots.txt that returned an HTML page instead of plain text. That is a soft 404, and a crawler treats it as no robots.txt at all, which happens to be harmless here. It is worth fixing anyway, because the same misconfiguration silently breaks your sitemap declaration.

# Webflow's untouched default, verbatim from three live sites:
curl -s https://yoursite.com/robots.txt
# Sitemap: https://yoursite.com/sitemap.xml

# Confirm it is really plain text and not an HTML soft 404:
curl -sI https://yoursite.com/robots.txt | grep -i content-type

Tabs, sliders and dropdowns are in the HTML

Webflow renders every tab panel, slide and dropdown into the HTML at publish time, then shows and hides them with CSS and a class toggle. A parser reading the raw response sees that content.

Six sites in the sample used Webflow tabs, 32 panels between them. Text was present in 28. The other four belonged to one site whose panels contain nothing but product screenshots, so there was no text to find. That is an image problem, not a rendering problem, and the fix is alt text rather than a rebuild.

Sliders behave the same way. One site shipped 8 slides and all 8 carried their text in the HTML. Nineteen dropdown menus across three sites all carried their link text. If your content is inside a Webflow interaction, it is already reachable.

# Count the text inside every tab panel, including the ones not shown on screen.
# A non-zero number on panels you cannot see proves they are already in the HTML.
curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://yoursite.com/ \
  | grep -o 'w-tab-pane' | wc -l

# Then read the page as a crawler does and look for the hidden panel's wording:
curl -sL -A "compatible; OAI-SearchBot/1.0" https://yoursite.com/ \
  | sed 's/<script[^>]*>.*<\/script>//g; s/<[^>]*>/ /g' | tr -s ' \n' ' ' | grep -o "text from your third tab"

CMS collection pages and pagination are server-rendered too

Ten collection pages, one from each site that had them, all returned server-rendered HTML with a real h1 and the full body copy in Webflow rich-text markup. They ranged from 327 to 5,700 words. Nothing was waiting on JavaScript.

Pagination is the part people worry about, and it survives as well. Four sites had paginated collection lists, and all four had the load-more script installed that replaces pagination with infinite scroll in the browser. It does not matter. Webflow still writes a real paginated link into the HTML, in the form of a query string with the collection list’s own hash. We followed each one and page 2 returned items that page 1 did not, on all four sites.

That is the detail worth knowing: the JavaScript enhancement sits on top of working server-rendered pagination, so a crawler that ignores the script still walks the whole collection. Check yours rather than assuming, because a custom build can remove the underlying link.

# Find the server-rendered "next page" link. Webflow names it with the
# collection list's hash, so it looks like ?a1b2c3d4_page=2 rather than ?page=2.
curl -sL -A "compatible; OAI-SearchBot/1.0" https://yoursite.com/blog \
  | grep -o 'href="[^"]*_page=2[^"]*"'

# Then confirm page 2 holds different items than page 1.
curl -sL -A "compatible; OAI-SearchBot/1.0" "https://yoursite.com/blog?a1b2c3d4_page=2" \
  | grep -o 'href="/blog/[a-z0-9-]*"' | sort -u | head

The real failure: a page built out of pictures

One site in the sample is close to invisible, and it has nothing to do with Webflow. Its homepage carries 126 words before JavaScript runs, no h1, and no h2. The body HTML is 98KB, almost all of it image markup and layout divs. Everything a reader would call content is baked into PNGs.

Webflow did its job here. It published exactly what was designed, and what was designed was a visual canvas. An assistant asked about this brand gets a title, a description and a pull quote, which is not enough to answer anything with.

The section pages underneath it are healthier, at 277 to 347 words each with real headings. The homepage is the weak response. This is the Webflow failure worth checking: a design-led page where the words live in artwork rather than HTML.

# The one number that matters. Under about 300 words on a page that looks
# full of writing means your copy is inside images.
curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://yoursite.com/ \
  | sed 's/<script[^>]*>.*<\/script>//g; s/<style[^>]*>.*<\/style>//g; s/<[^>]*>/ /g' \
  | tr -s ' \n' ' ' | wc -w

# And check you have a heading at all:
curl -sL -A "compatible; OAI-SearchBot/1.0" https://yoursite.com/ | grep -c "<h1"

The strangest thing we found: a response too big to fetch

Webflow’s own blog, at webflow.com/blog, could not be fetched at all by our scanner. curl retrieves it fine and returns HTTP 200. The page is not blocked, not cloaked, and not rate limiting anyone. It sends 17,224 bytes of response headers, and the default header buffer in Node.js is 16,384 bytes. Any client with that default gets a connection error instead of a page.

The cause is two Content-Security-Policy headers, one enforcing and one report-only, each listing well over a hundred allowed domains. Raise the limit to 32KB and the same request returns 200 with 458KB of HTML. The webflow.com homepage sends 13,925 bytes and squeaks under the cap, which is why the homepage works and the blog does not.

This is not a Webflow platform default. The other 13 sites in the sample send around 1,500 bytes of headers. Large CSP allowlists can push the total past a client header limit. That failure produces a connection error rather than a 403.

# Measure your response header block. Anything near 16KB is a live risk.
curl -sS -D - -o /dev/null https://yoursite.com/ | wc -c

# Reproduce the failure and the fix:
node -e 'fetch("https://webflow.com/blog").then(r=>console.log(r.status)).catch(e=>console.log("FAIL:",e.cause.message))'
# FAIL: Headers Overflow Error
node --max-http-header-size=32768 -e 'fetch("https://webflow.com/blog").then(r=>console.log(r.status))'
# 200

Your .webflow.io staging domain is still public

Every Webflow project keeps its free subdomain working after you attach a custom domain. We checked three and all three were live, serving the complete site, byte for byte the same content as the paid domain.

Webflow does put "User-agent: * / Disallow: /" in the staging robots.txt, which is the right call and stops well-behaved crawlers. What is not there is a noindex meta tag on the pages themselves. So the protection is entirely robots.txt, and robots.txt is a request rather than an enforcement. A crawler that ignores it, or a person who pastes the staging URL into a chat window, reaches a full duplicate of your site with no indication it is not the real one.

This is low severity and worth thirty seconds. Check whether yours is up, and if you have retired a project, unpublish the subdomain instead of leaving it serving an old version of your site forever.

# Is your staging subdomain still serving the site?
curl -s -o /dev/null -w "%{http_code}\n" https://yourproject.webflow.io/
curl -s https://yourproject.webflow.io/robots.txt

Check your own Webflow site in one command

Fetch your homepage as a crawler and read what comes back. If your headline, your value proposition and your main body copy are in there, your rendering is fine and no Webflow-specific fix will improve it.

curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://yoursite.com/ \
  | sed 's/<script[^>]*>.*<\/script>//g; s/<style[^>]*>.*<\/style>//g; s/<[^>]*>/ /g' \
  | tr -s ' \n' ' ' | head -c 1200

What to fix instead

On Webflow, reachability is solved for you and structure is not. Four of the 14 sites had no JSON-LD at all, and one had no h1. Those are the gaps worth your time, because they are what tells an assistant which company you are and what you sell, rather than leaving it to infer both from prose.

Add an Organization block with your name, URL, logo and description, add the type that matches each page, and put a real h1 on every page. Webflow supports all of this in the page settings without a developer.

These numbers are one day’s snapshot of 14 sites. Re-run the commands above after a redesign, domain move or CMS restructure. Crawlable can run the checks on a schedule and email you when the result changes.

Frequently asked