What Your Optimizely DXP Is Serving to AI Crawlers - and the Three Levers You Actually Have

Status: verified August 2026. The vendor-backed claims below link to vendor documentation; observations from my own client work are marked as such.

The observation that started this

I have watched a single AI crawler pull tens of gigabytes a month from one client’s DXP site. That observation comes from sources I can’t publish, so treat it as context, not evidence. From the same engagement: attributable referral traffic in return was effectively none, with the caveat that referral attribution for AI assistants is unreliable, since many send no Referer header at all.

One anecdote proves nothing. For the industry picture, Cloudflare publishes crawl-to-refer ratios per operator on Radar: how many pages a crawler takes for each visitor its platform sends back. Go look them up before you quote a figure from any article, mine included. They move quarter to quarter, and the ones circulating in blog posts are usually a year stale.

The gigabytes are the wrong thing to be angry about. Most of that volume was images and other static assets, and the question that decides whether any of it costs you something is a different one. Here’s the part that matters, straight from Optimizely’s consumption documentation:

“Bot traffic, that is not blocked by default Cloudflare WAF rules, is counted as a billable page view.”

Read that alongside how Optimizely defines the units. A static file is “an uploaded or deployed file, such as a PDF, image, or CSS.” A page is content that is “not a static file but has a URL and can be loaded in a browser.” And a visit is “an HTTP request that returns an HTML response where the Referer header either is absent or does not match the site hostname”, measured at the CDN edge.

So the assets are not the primary signal for this particular billing question (they may still matter for cache behaviour and origin load, but a PDF or an image is not a page view). The HTML requests are the ones to count. A page view is a page being served; a visit is the edge counting an HTML response that arrived without a matching Referer. An AI crawler can produce both.

One honest caveat, because your CFO will raise it before you finish the sentence. Optimizely’s documentation is clear that unblocked bot traffic is billable, and clear that visits are measured at the CDN edge. It is less clear about the exact plumbing of page views. The same page points you at Application Insights to view consumption, which is server-side telemetry, while visits come from Cloudflare Zone Analytics. That leaves a real question unanswered for architects: does a bot request served entirely from CDN cache (never touching your App Service or your database) still increment the billable counter? If it does, edge blocking is your only cost defence, because the application never sees the request to reject it. Get that confirmed for your contract and your tenant before you build a business case on it. Asking the question costs you an email; assuming the answer costs you credibility in the room.

Your homework before reading further: count HTML 200 responses to the top AI user-agents and divide by your contracted page view allowance. Call that what it is: an estimate of the AI-crawler share of potentially billable HTML traffic, not a billing reconciliation. Page views and visits come from different systems, and the caching question above is still open. It is still the one number that turns this from an opinion into a conversation.

You can get the raw material yourself. DXP pushes Cloudflare edge logs to an Azure Blob container, and Optimizely gives you self-service access: generate an API key in the PaaS portal with access to the General environment, then use the EpiCloud PowerShell module. Get-EpiEdgeLogLocation returns a read-only SAS link (24 hours to expiry by default); Get-EpiStorageContainerSasLink works against the cloudflarelogpush container. The logs arrive as gzipped JSON, one request per line, with ClientRequestUserAgent, EdgeResponseStatus, CacheCacheStatus and the rest of what you need.

Mind the retention. Edge logs in that container are kept for seven days. You cannot decide today to measure last month. There is no last month. Either you have already been exporting, or you take a seven-day window, say so explicitly, and treat the extrapolation as an estimate, nothing more. Set up the export first; the measurement is a week behind whenever you start.

Why you can’t just fix it in Cloudflare

Every DXP environment sits behind Cloudflare; CDN and WAF are provided by default. This leads teams to a reasonable-sounding conclusion: open Cloudflare, set AI bot rules, done. It won’t work.

That zone is not yours. Optimizely’s WAF documentation is unambiguous: Optimizely updates rulesets at the platform level and “does not typically update these rules on a customer by customer basis.” Support will action blocks for specific countries, IP addresses and ASNs on request. Rate limiting is available “only as a temporary incident mitigation measure (not as a permanent setting).” And then the sentence that defines the boundary:

“Any WAF rules beyond the previous are not supported by the platform and must be handled in code or by managing your own Cloudflare zone (orange-to-orange).”

Custom rulesets, per the same page, “cannot be defined at this time” (their words). That’s the whole toolbox.

That’s the real constraint. Not a missing feature - a missing tenancy.

Lever 1: robots.txt

Free, immediate, entirely under your control. robots.txt is editable from the CMS, and for CMS 12+ there is a well-known community admin extension, Stott.Optimizely.RobotsHandler, which handles per-site files in multi-site solutions.

The obvious objection is that robots.txt is voluntary. True: it is a request, not a control. But the major AI crawlers publish and honour their user-agent strings, and a well-formed file costs you an afternoon. If you have done nothing so far, this is where doing something starts.

Three cautions. First, audit what your existing file already says, because broad Disallow rules written years ago for search engines can block AI crawlers you actually want. Second, be deliberate about which user-agents you name; see the mixed-purpose problem below. Third, if your front end is decoupled (a Next.js app on Vercel, a SPA behind its own hosting, anything reading Optimizely through Content Graph), the robots.txt your visitors receive comes from the front end. The CMS never serves it. The admin extension is the right answer for a classic CMS 12 solution rendering server-side; it is the wrong place to look if the file actually comes from somewhere else. Fetch /robots.txt from production before you decide where to edit it.

Lever 2: Optimizely Support

Country, IP and ASN blocks, enforced at the edge, on the zone you don’t control. This is a blunt instrument, built for abuse cases. But it is genuinely enforced, unlike robots.txt, and it costs a ticket.

Two things worth doing while you have Support’s attention: ask whether any AI-bot rules are currently active on your zone at the platform level, and ask what the escalation path is if a crawler starts materially affecting your consumption. I have seen no public Optimizely statement on AI crawler handling either way, so treat this as an open question about your specific tenant. Don’t assume.

Lever 3: your own Cloudflare zone (Orange-to-Orange)

This is the real one. Optimizely documents and supports bringing your own Cloudflare zone in front of theirs, a configuration Cloudflare calls Orange-to-Orange. Per Cloudflare’s documentation: “the settings configured in your Cloudflare zone will be applied to the traffic first, and then the settings configured in the SaaS provider’s zone will be applied to the traffic second.”

Optimizely lists what this unlocks: Bot Management, Workers, and R2 Storage.

What matters most for this problem is AI Crawl Control, which is available on Cloudflare’s Free plan as well, though detection and analytics are more limited there than with Bot Management (on Free, crawler analytics are capped at a 24-hour window). It shows which AI services are hitting your content, which operator they belong to, and, since Cloudflare’s July 2026 change, what they are there for. The categories are now:

• Search - crawlers indexing your content to answer questions about it later, where you should expect referral traffic or other compensation in return
• Agent - acts in real time on a person’s behalf: chat fetch bots, browser-driving agents
• Training - crawlers taking your content to train or fine-tune a model

Those categories are not three flavours of the same thing. They have different business value to you, which is exactly why a single switch is the wrong instrument:

Traffic	What it is doing	What it is worth to you
Search crawler	Indexing for classic search results	SEO and discovery, the channel already funding the business
AI search crawler	Retrieving an answer to cite now	Visibility in AI answers, referral if the operator sends any
Training crawler	Taking content to train a model	Usually no direct return
Agent / browser agent	Acting on a real person’s request	Potentially the highest value per request: a user is waiting

You can set allow or block rules per crawler. That is the difference between a bot policy and a slogan.

But know what “free on every plan” buys you. On the free plan, AI Crawl Control identifies crawlers by user-agent string, so it sees the ones that introduce themselves honestly. Catching a crawler that lies about what it is takes Cloudflare’s Bot Management detection, available on upgraded plans. That is a reasonable place to start: the large operators do identify themselves, and they are the ones generating the volume. Just do not confuse “no unknown crawlers in the dashboard” with “no unknown crawlers.”

Before you scope this, know the constraints. Per Cloudflare’s documentation, O2O “only applies when the two zones are part of different Cloudflare accounts,” and it “does not apply when an A record is used to point to the SaaS provider’s (apex proxying).” Whether your specific apex-domain and DNS setup is compatible with the O2O topology is a question to settle with Cloudflare or Optimizely before implementation, not one to infer from a diagram. On the Optimizely side, your hostname must be on a zone ending in dxp.optimizely.com (check the Hostnames tab in the Management Portal). And Bot Management is not included with a Cloudflare plan: it is “added to Enterprise plans by your account team,” meaning a separate commercial conversation. Price it before you promise it to anyone.

One thing I have not been able to verify from documentation: how cleanly AI Crawl Control enforcement behaves for a DXP hostname in an O2O chain specifically. The ordering rule says your zone acts first, which implies it works. But implication is not a test. Prove it on one hostname before you migrate the estate.

The trap: mixed-purpose crawlers

The most consequential detail in Cloudflare’s July change is the one that gets least airtime. A crawler that serves both search and training is blocked by any setting that blocks Training, including the older “Block AI Bots” toggle. Cloudflare’s own words: “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.”

Read that list again, because it is not really a story about AI. Googlebot is how you appear in Google Search. Applebot and BingBot are how you appear in theirs. A team that reaches for “block training” as an obvious, defensive, cost-saving default can take out ordinary search indexing on the affected pages. That channel has been funding the business since long before anyone argued about model training. The risk to your presence in AI-generated answers is real, but it is the smaller half of this.

Which means the intuitive move, “block training, keep search”, is not achievable with a broad switch for every operator. Decide per crawler, with the data in front of you, and the data you need lives in AI Crawl Control, which brings you back to Lever 3.

What actually happens on 15 September 2026

Cloudflare announced on 1 July 2026 that from 15 September, defaults change: Training and Agent crawlers blocked on pages that display ads, Search still allowed. Per Cloudflare’s press release, this applies to new customers, new sites created by existing customers, and “all existing free customers that have not changed their settings by September 15, 2026.” All customers can opt out of the new defaults before that date.

Your DXP production domain is none of those things. It is an existing hostname on an existing enterprise zone, so it should not be assumed to inherit the new defaults. That assumption still needs confirming with Optimizely, because the zone is theirs, not yours.

I am labouring the point because the honest version is more useful than the urgent one. The deadline is not your deadline. What is happening is that the industry default is moving from “crawlable unless you object” to “ask first”. Platform-hosted sites will move last, because their operators sit one tenancy removed from the switch. The gap between where the web’s defaults are going and where your DXP site actually sits is the thing to put on an agenda.

With one twist worth catching before it catches you. Lever 3 puts a Cloudflare zone of your own in front of Optimizely’s, and that zone is a zone like any other. Stand it up on a free plan, leave its settings untouched, and it falls squarely inside the group the 15 September change does apply to. The deadline you just dismissed as somebody else’s arrives through the door you opened. Nothing dramatic follows from it: you review the AI settings on your own zone as part of standing it up, which you would want to do anyway. But do it on purpose. Otherwise you find out in October that the defaults decided your policy for you.

Where monetisation stands, honestly

Cloudflare’s Pay Per Crawl (charging AI crawlers for access) is real, but as of the documentation update on 28 July 2026 it is “currently in closed beta,” with access via a signup form or an Enterprise account executive. During Agents Week on 4 August 2026 Cloudflare announced Wallets and cloudflare.pay for agentic payments, with full functionality described as arriving in the coming months.

Neither should be treated as a committed part of this quarter’s business case. Watch both. Budget for neither yet.

A sequence that works

1. Start the export today. Edge logs live seven days. Until something is pulling them out of cloudflarelogpush on a schedule, every measurement you plan is a measurement you cannot take.
2. Estimate. HTML 200 responses per AI user-agent, as a percentage of your page view allowance. One number, labelled as an estimate, with the measurement window stated next to it.
3. Audit robots.txt. Fix what is accidentally blocked, state what you actually want, and check where the file is really served from. This week.
4. Ask Optimizely two things: what bot rules are active on your zone today, and how your contract counts page views against cached and uncached bot requests.
5. Decide whether O2O is justified, with the Bot Management add-on priced in, and against the number from step 2.
6. Pilot O2O on one hostname, set the AI policy on your own zone deliberately, verify AI Crawl Control behaves as documented, then decide per crawler.
7. Put the policy in writing, with a named owner. Governance isn’t fun. But “nobody decided” is the default that everyone is living with.

Sources

Optimizely

• Web Application Firewall - custom rulesets, what Support will and won’t block
• Consumption Metrics - billable page views, definitions of page / static file / visit
• Integrate Your Cloudflare Zone with Optimizely DXP - O2O, hostname requirements, what it unlocks
• Logging in Optimizely DXP - edge log retention
• Manage storage containers - Get-EpiEdgeLogLocation, Get-EpiStorageContainerSasLink
• Edit the Robots.txt file

Cloudflare

• AI Crawl Control - availability and plan-dependent detection
• New options to manage AI traffic - changelog, 1 July 2026
• Your site, your rules - category definitions, multi-purpose crawlers
• Your Content, Your Rules - press release, scope of the 15 September defaults
• Bot Management for Enterprise
• How O2O works
• What is pay per crawl - updated 28 July 2026
• Cloudflare Wallets - announced 4 August 2026
• Cloudflare Radar - crawl-to-refer ratios, live data

Community

• Stott.Optimizely.RobotsHandler - per-site robots.txt for CMS 12+

The crawler-volume and zero-referral observations come from private client data and are not independently verifiable; that is why the text keeps them round and hangs no argument on them. Everything else links to vendor documentation, verified in August 2026.