Sovereign AI for Support in CMS 13, Part 1: Bielik vs the Global Model

Disclosure. This is the work of a single author (Pino Labs). It is independent work: not affiliated with, sponsored by, or reviewed by Optimizely, Algolia, or the SpeakLeash/Bielik project.

Executive summary. This is a pre-registration. We built a 3-arm A/B/n test on Optimizely Feature Experimentation (part of Optimizely One) to answer two questions: (1) does any AI assistance beat plain keyword search on user-level resolution, and (2) can a sovereign Polish 11B model (Bielik), self-hosted on GPU, match or beat a global hosted model on the same prompt, the same retrieval, and the same corpus. The harness is built, tested and green: one verify.ps1 gate runs the full battery (26/26 self-test, 22/22 statistics, 19/19 runtime routing, judge artifact, clean net10 build). On 2026-07-20 the harness had its first one-day dress rehearsal on real endpoints, Bielik-11B on an A100 GPU (vLLM) versus gpt-4o-mini on Azure OpenAI, which validated the plumbing end-to-end and produced the first real latency, token and cost numbers (see The dress rehearsal). The traffic was simulated, however, and the corpus is a 12-document stand-in, so there is no Q1/Q2 result yet: every verdict emitted under those conditions is stamped HARNESS_VALIDATION_ONLY, and the result tables below exist before the data. The method is frozen with a lock_sha256 and a git prereg-freeze tag before the numbers exist, so the eventual result can be checked against a method committed while the outcome was still unknown; the companion repository is public at github.com/pino-labs/sovereign-ai.

In this part: The machine at a glance · The stack · Two models, one prompt · What counts as “resolved” · Five gates that can stop the run · The decision-grade gate · Opal or custom · Results · The dress rehearsal · Threats to validity · Sharp edges · FAQ · The take · Glossary · Further reading.

The experiment compares three customer-support answer engines on the same questions. Control is what we ship today: Algolia keyword search, a list of links. Global is a general-purpose LLM behind an API, concretely the Azure-hosted gpt-4o-mini snapshot 2024-07-18. Bielik is the reason this experiment exists: an 11-billion-parameter sovereign Polish model, self-hosted on a GPU we run, so inference (the prompt, the retrieved context, the generation) happens inside a deployment boundary we control. (One caveat, disclosed now and detailed in Threats to validity: retrieval still calls Algolia’s SaaS in every arm, so the user’s query does leave that boundary.) The serving surface is a support page on an Optimizely CMS 13 site; the assignment and measurement engine underneath is Optimizely Feature Experimentation.

Two distinct questions follow. Q1 is the business question: does AI assistance improve resolution enough over keyword search to justify its cost? Q2 is the architectural one: a global model is a managed dependency in someone else’s data center, priced per token, with your customers’ words flowing through it. A self-hosted model can keep inference inside a boundary you choose, which makes data-residency an architecture property you can actually audit, provided retrieval, logging and the GPU’s physical region all honor the same boundary. It also reframes cost from per-token rent to owned GPU capacity, a trade-off we chose not to model here, because a GPU running 24/7 is not automatically cheaper than tokens. All of it matters only if the model can keep up, so the operational question is whether a model you deploy and operate yourself can match or beat one you rent. We test non-inferiority first, then superiority if the sequence allows it.

Most “our AI beats X” posts skip the hard part: making the comparison fair, with the model as the only thing that differs and everything else pinned. Doing that properly surfaces failure modes that are easy to underestimate. We hit several during development, an internal audit found a few more we had missed, and a one-day run on real hardware surfaced one nobody had predicted. This article covers the method and those failure modes, plus the first real infrastructure numbers; the verdict numbers come in a later part, after the decision-grade run.

If Bielik matches Global, teams get a measured, self-hosted, in-region alternative to the hosted APIs, a claim strong enough that it needs a measurement behind it. We pre-registered the method, froze it with a hash, and are publishing it before the outcome is known.

The machine at a glance

Everything below is a refinement of this diagram.

The machine at a glance - the full harness architecture One request path inside Optimizely One: a frozen FX datafile buckets the user, ArmRouter runs one shared retrieval and one prompt for whichever arm was assigned, and only the model id (with its endpoint dialect) differs between the AI arms; every assist leaves a three-row trail in the JSONL sink.

The harness is a single request path: FX deterministically buckets the user, ArmRouter runs the same retrieval and the same prompt for whichever arm was assigned, and only the model id (and its endpoint dialect) changes downstream. Control short-circuits before the LLM and serves Algolia hits directly, which makes “keyword search” a real, measurable arm. Every assist emits a three-row trail (exposure, interaction, outcome) into an append-only sink that stands in for the production warehouse, so the dependent variable is captured the moment it happens.

The stack: how one prompt reaches three engines

Deterministic bucketing on a frozen datafile

The bucketing decision belongs to FX. We hand it a user id and pre-treatment attributes, and it returns a sticky variation:

var userCtx = opti.CreateUserContext(userId, attrs);   // attrs: only pre-treatment, type-coerced
var decision = userCtx.Decide("support_assistant");     // MurmurHash(userId) → control | global | bielik
var armAssigned = decision.VariationKey ?? "control";

Because bucketing is a pure hash of the user id against the datafile’s traffic allocation, the same user lands in the same arm on every visit without us storing a single assignment; there is no assignment table to corrupt and no cross-arm leakage when a user returns mid-experiment.

That guarantee only holds if the datafile cannot change underneath us. The default SDK behaviour is to poll the CDN and hot-swap a new revision, which would re-bucket sticky users mid-run with nothing in the logs. So we freeze it. WithDatafile(datafileJson) pins the boot-validated revision and WithAutoUpdate(false) never re-polls; shipping a new datafile then takes a deliberate redeploy.

The freeze is also enforced. At boot the service fetches the same datafile the SDK would use, compares its revision against the pinned value, and asserts the flag contract (the support_assistant flag plus all three control/global/bielik variations must be present); on any mismatch it refuses to start. Revision drift therefore surfaces as a boot failure, and a boot failure is loud, which beats a silent shift in who-saw-what that would only be discovered when the numbers looked strange. The pinned revision for the current harness freeze is 31 (re-pinned and re-frozen if the FX flag or event set changes before a live run).

Algolia as the shared retrieval: the isolation seam

Retrieval runs first, for every arm, including Control. The Control arm returns those hits verbatim with no model in the loop; both AI arms feed the identical hits into the identical prompt. This is the variable-isolation seam: if the two AI arms drew from different retrievals, Q2 would be comparing two RAG systems. ArmRouter builds the grounding context once and hashes it, so the evaluator can later verify what each model actually saw:

var prompt = BuildPrompt(question, hits);   // IDENTICAL for both AI arms
var contextHash = ComputeContextHash(hits); // same bytes the model is grounded on

A single BuildPrompt plus a recorded contextHash means any later difference in answers is attributable to the model, because what it was asked and what it was given were identical.

The corpus behind that retrieval is deliberately tiny and fictional, a 12-record knowledge base for “Izovell,” an invented mineral-wool manufacturer, frozen for harness validation (a boot-time guard asserts exactly 12 records in the live index). That is enough to prove the plumbing and the isolation seam; it says nothing about retrieval quality at production scale, which is why the decision-grade run needs a real corpus (see Threats to validity).

One client, two dialects

Both AI arms speak the OpenAI chat-completions shape, so a single ChatModelClient serves both. It picks the per-arm named HttpClient (endpoint + key) but builds the same request body for each, varying only the model id:

var payload = JsonSerializer.Serialize(new {
    model,                              // the ONLY per-arm difference
    messages = new[] {
        new { role = "system", content = options.SystemPrompt },
        new { role = "user",   content = userMessage },
    },
    temperature = options.Temperature,  // 0.2, shared
    max_tokens  = options.MaxTokens,    // 512, shared
    stream = false,
});

Generation parameters live on shared ModelOptions, never per-arm, so temperature and token budget cannot diverge between Bielik and the global model. The two endpoints differ only in dialect, and the whole dialect fits in a table: two config values (AuthHeader, ApiVersion) select it per named HttpClient at registration time:

	vLLM / OpenAI (Bielik)	Azure OpenAI (Global)
Auth	`Authorization: Bearer <key>`	`api-key: <key>` (raw header, no scheme)
Path	`chat/completions`	`chat/completions?api-version=…` on a deployment URL

Because the dialect is that thin, swapping today’s llama3.2:3b stand-in for production gpt-4o-mini is a config change. No code fork. The runtime test (19/19 passing) proves each arm hits the right endpoint with the right header and version. The one-day dress rehearsal exercised exactly that swap on live endpoints and exposed a config bug on this seam (sharp edge #5).

The JSONL sink: a local stand-in for the warehouse

Every assist appends three joinable rows (assistant_exposed, assist_interaction, assist_outcome) to an append-only JSONL sink, keyed by a per-assist InteractionId. In production these are warehouse streams; locally, this file is the warehouse:

public Task EmitAsync(OutcomeEvent ev, CancellationToken ct = default)
    => AppendAsync("assist_outcome", ev);   // ct accepted for symmetry, deliberately NOT forwarded

The durable write uses CancellationToken.None, so an aborted slow request (Bielik on a CPU stand-in) cannot drop its exposure row; that would be arm-correlated attrition, which is the bias this design has to prevent. The sink also counts its own drops and surfaces them on /health, and a lightweight FX TrackEvent is dual-written so Optimizely’s native Results and SRM checks are not dark while the JSONL plane remains the primary inference source. Record counts for the decision-grade run: pending. (The one-day dress rehearsal wrote 150 joinable exposure rows across its two passes, 75 in the clean pass, through this exact sink with zero drops on /health.)

Two models, one prompt: the rule that makes it a fair fight

The whole experiment rests on one rule. The only thing that may differ between the two AI arms is the model id. Both arms get the same system prompt, the same Algolia retrieval pass feeding the same passages into context, and the same generation parameters (temperature, top-p, max tokens, stop sequences). Global and Bielik see byte-identical inputs and diverge only in the weights that turn those inputs into an answer.

The strictness follows from how narrow Q2 is: “does an 11B sovereign Polish model match or beat a mainstream hosted model (gpt-4o-mini-class) at this support task?” The moment the prompt drifts per arm, the experiment stops measuring the model and starts measuring prompt-engineering effort, which is almost never symmetric: whatever was optimized last favors the arm it was optimized for.

A few things that would break the isolation, in rough order of how often they catch people:

Different prompts per arm. The obvious one, and it still happens when a stray if model == "bielik" branch shows up in the prompt builder. One prompt, parameterized only by the model id, no exceptions.
Different retrieval. If Global and Bielik are handed different passages (different k, different ranking, a different index snapshot, or the same index queried at different times as content shifts), then you are comparing retrieval pipelines. The Algolia pass runs once per request and its output is the shared context both arms consume.
Different generation params. Because temperature and top-p change the answer distribution as much as the weights do, a model run at temp 0.2 against another at temp 0.7 is a different experiment than the one you pre-registered.
Different latency. The subtle one. If one model is reliably slower, some users give up before the answer lands, and that abandonment shows up in the resolution rate as a quality difference when it is actually a speed effect. That is why we hold a symmetric per-arm latency cap (Global ≈ Bielik): variable isolation covers not only the bytes fed to the model but everything downstream that the user can perceive. More in the decision-grade gate.

Without a single prompt template, a single retrieval call, and a single params block shared across both arms, there is no model comparison.

What counts as “resolved” and the post-treatment trap

The dependent variable is user-level resolution rate: of the users assigned to an arm, what fraction got their issue resolved within the attribution window. User-level rather than message-level: a user who asks four follow-ups is still one observation, and counting messages would let chatty users dominate the estimate and break the independence our inference assumes.

The primary estimand is ITT-by-assignment (intention-to-treat): every assigned user is analyzed in the arm they were assigned to, and the outcome is whether that user’s issue was resolved within the window, through any channel: the assistant’s answer, the search results, or a human agent after a hand-off. The tricky case is the hand-off, when the model declines and routes the user to a human. That event is neither an automatic failure nor an automatic success: the user stays in the denominator, and the same downstream signal (did the issue get resolved in the window) is measured for them like for everyone else. (This is what resolution_job.py implements: declined users are kept, with a defined outcome; they are separated out only in the per-protocol views.)

One consequence of ITT needs spelling out, because the fallback matrix otherwise blurs it: under ITT, Q2 compares the Bielik-assigned strategy against the Global-assigned strategy, assignment including whatever the fallback chain did downstream. A timed-out Bielik call served by Global still counts in Bielik’s column. That is the deployable question, since you would ship the strategy fallbacks and all, but it is not a pure model contrast, and we won’t quote it as one. Two mechanisms keep the distinction visible: the contamination gate flags fallback imbalance between arms (heavy one-sided fallbacking would gut the model comparison and gets reported as such), and every exposure row records ArmServed next to ArmAssigned, so an as-served sensitivity that strips fallback-served interactions can be computed and published alongside the primary.

That definition rejects two tempting alternatives, for different reasons. Scoring declines as unresolved would change the question from “did the problem get solved” to “did the problem get solved without a human”, a legitimate estimand but a different one, and an unfair primary here: Control structurally cannot decline (it’s a search results page), so the penalty would land only on the AI arms, and hardest on the model that hands off responsibly. Excluding declined users is worse: the decline is a post-treatment variable, caused by the arm the user landed in and by how hard their case is, and selecting the denominator on it is textbook post-treatment conditioning; the arms stop being comparable groups and the randomization is lost. (An earlier draft called this a “collider”; the precise error is conditioning on a post-treatment variable.)

One interpretive consequence follows. Because the human backstop is available to every arm, the primary measures whether the problem got solved under a given assignment; how much the AI alone contributed is a separate quantity. A diligent human tail can compress the gap between arms. That is acceptable (it is the business question, measured conservatively) but it forbids one specific misreading: the primary must never be presented as isolating “AI resolution.” Separating the AI’s own contribution is the job of the sensitivities below.

Around that primary, we run two per-protocol sensitivities that triangulate the ITT result. Both score only non-declined interactions, so they condition on post-treatment behavior, and that is why they stay sensitivities while the verdict stays with ITT:

strict, still user-level: a user scores 1 only with an in-window resolution_confirmed and no reopen or escalation, judged from their non-declined interactions. Users whose only interactions were hand-offs drop out of this view; they are separated and counted.
weighted, same unit, softer scoring: 1 for confirmed, 0 for reopen or user escalation, and a pre-registered partial weight w for silence (no confirmation, no reopen; ambiguous evidence, so it gets partial credit). w is fixed in the lock up front.

If the headline holds across ITT, strict, and weighted, we trust it more; if it only holds under one of them, that is a finding in itself and will be reported as such.

The reopen window (window_x_days) is the clock on all of this. A ticket isn’t “resolved” the instant the chat ends, so the window will be set at the P95 reopen time from historical data, which makes “resolved” mean more than merely quiet for an hour. Until that calibration lands, the lock carries a 7-day placeholder tagged [CALIBRATE-BEFORE-RAMP], and check_lock blocks the ramp while it does. Late entrants are right-censored: users who joined too near the end of the run aren’t scored on a window they never had a chance to complete.

Metric	Denominator	Hand-off to human	Role
ITT-by-assignment	All assigned users, by assigned arm	Stays in; outcome = resolution via any channel	Primary
strict	Users with ≥1 non-declined interaction	Declined interactions excluded; declined-only users separated & counted	Sensitivity
weighted	Users with ≥1 non-declined interaction	Declined interactions excluded; silence scores w	Sensitivity

The ITT number is the verdict; the other two indicate how much to trust it.

Five gates that can stop the run

A clean estimand is necessary but not sufficient. Between assignment and a credible verdict sit several failure modes that routinely turn a “significant” result into an artifact, and we instrument each of them as a gate.

SRM: whether randomization held. The split is pre-registered at 20/40/40 (Control / Global / Bielik). Our srm_check is an offline gate of our own: a chi-square of final arm counts against that split, run once at analysis close, flagged at p < 1e-3. That is a separate thing from Optimizely’s native sequential SSRM monitoring, which watches continuously during the run; the dual-written TrackEvent keeps that live in parallel, and our gate is the analysis-time backstop on the JSONL plane. (Whether the native monitoring covers this exact 3-arm FX configuration is itself verified during ramp.) Sample-ratio mismatch is the basic health signal of any experimentation stack: if the arms don’t fill in the ratio you specified, something upstream is broken (a bucketing bug, a logging gap, a redirect eating one arm) and every downstream number is suspect. A failed SRM doesn’t get explained away; it halts interpretation until the cause is found.

Contamination: whether the data is what we think it is. The contamination_report watches several leaks at once:

% excluded (anonymous users, off-flag traffic): how big a slice we drop.
fallback rate by reason: when the pipeline falls back, why, broken out per reason, so a spike can be traced to its cause.
off-flag-leak guardrail: must be exactly 0. If a user who isn’t in the experiment flag receives experimental treatment, the arms are contaminated. There is no threshold to tune here; it must read exactly zero or the run is compromised.
malformed / degradation imbalance flags. This one matters for Q2 specifically: a model that fails at a different rate than the other introduces a confound analogous to differential latency, so we track not only failures but whether they are balanced across arms, and flag the imbalance.

Inference: no parametric shortcuts. Because the unit is the user and observations cluster (multiple interactions per user), we don’t lean on textbook two-sample formulas that assume independence we don’t have:

p-values via randomization inference: re-shuffle assignments 10,000 times with the user as the cluster and read the observed effect against the permutation null. It assumes almost nothing about the outcome distribution and respects the clustering by construction.
confidence intervals via cluster bootstrap: resample users (not rows) with replacement, so the CIs inherit the real correlation structure.
fixed-sequence gatekeeping across {Q1, Q2}: Q1 is tested first at full α; Q2 gets its formal test only if Q1 rejects (the rationale, and the sizing consequences, in the sizing gate below). The pre-registered hierarchy is what controls family-wise error.

Sizing: the trap that bakes in a false negative. Two-proportion power gives:

Q1 (AI vs Control, superiority): ≈ 7,354 users/arm (at the placeholder baseline p=0.60, MDE 2pp; recomputed if calibration moves the baseline).
Q2 non-inferiority (Bielik vs Global, δ=1pp): ≈ 25,967 users/arm (at the placeholder baseline p=0.70).

For reproducibility: both figures are sized at α = 0.05, one-sided (Q2’s superiority is one-sided too, since NI and superiority share the > direction in the fixed sequence), power 0.80, inflated by a design effect Deff = 1 + (m−1)·ICC, with placeholders m = 1, ICC = 0, both [CALIBRATE-BEFORE-RAMP], so the binding N only grows once real interactions-per-user data lands. Superiority uses standard two-proportion sizing at the stated baselines; NI is sized at worst-case true equality: n/arm = (z₁₋α + z₁₋β)² · [p_B(1−p_B) + p_G(1−p_G)] / δ². The formulas live in sizing.py, the baselines in the lock.

users per arm required
Q1  superiority       ███████████ 7,354
Q2  non-inferiority   ████████████████████████████████████████ 25,967  ← binding

The non-inferiority gate needs roughly three times the sample of the superiority test, and it is the binding constraint. But size for superiority alone and the NI test comes back inconclusive purely because it was underpowered, a false negative baked into the design that no analysis fixes after the fact (sharp edge #2 has the full account). We size to the NI gate, and superiority comes free.

One correction belongs on the record, because an external review caught it and it changed the plan: an earlier revision declared Holm across {Q1, Q2} while sizing at per-test α = 0.05, an inconsistency, since Holm-consistent sizing must plan for α/2 = 0.025, which moves Q1 to 9,336/arm and the binding Q2-NI figure to 32,966/arm. A protocol cannot be called frozen while that fork is open, so it is closed: the family is now fixed-sequence gatekeeping (Q1 tested first at full α, and Q2 formally tested only if Q1 rejects, also at full α), so the figures above are exactly the figures the plan implies. The accepted, printed cost: if Q1 fails, Q2 is reported descriptively, with no verdict (not_tested_gatekept). That hierarchy matches the product decision: if AI doesn’t beat keyword search, nothing ships and “which model” is moot. The change is recorded in the lock with the Holm numbers left visible for audit, implemented in confirmatory.py, and made before any real data existed.

Each of these five gates can stop the run, and each is reported whether it passes or fails.

The decision-grade gate

Everything above can be green and the result can still fall short of a product decision. The line between “the harness works” and “this number tells you what to ship” is enforced mechanically by the pipeline.

Prod models AND real data, or it’s stamped. A verdict counts as decision-grade only when both AI arms are running production models and the data is real. Fail either condition and the pipeline brands HARNESS_VALIDATION_ONLY onto both Q1 and Q2. The stamp travels with the result, so a synthetic or stand-in number can never be quoted bare, screenshotted into a deck, or mistaken for a finding. As of the one-day dress rehearsal the models condition has been met once (real Bielik on GPU, real gpt-4o-mini) but the data condition has not: the traffic was simulated. One green condition out of two is still a stamp, so today every result carries it.

The pre-registration freeze: lock_sha256. The experiment definition (arms, split, primary estimand, margins, windows, sizing) is locked and hashed, and every report carries the hash. If the method changes, the hash changes, and the mismatch is visible on the face of the report; the goalposts cannot be moved after seeing the data while still presenting the old lock_sha256. The freeze also enforces readiness: any calibration value still sitting at [CALIBRATE-BEFORE-RAMP] makes check_lock block, so the experiment is not ready to ramp until the placeholders are real values.

The lock in force as this draft freezes is sha256: 869cbc2924e235a513650623ec613ab378835d1719e54a9465800c53d2e68dd6, frozen at the git tag prereg-freeze in the public repository: github.com/pino-labs/sovereign-ai, file config/experiment.lock.yaml. To verify, hash the file as committed (git show prereg-freeze:config/experiment.lock.yaml | sha256sum) and compare against this paragraph. During development the lock changed twice, both times before any real data existed: an audit remediation adding two clustering-calibration placeholders (m_interactions_per_user, icc_resolution) plus the symmetric global_timeout_ms, and the multiplicity decision (Holm → fixed-sequence gatekeeping; details in the sizing gate). Those amendments happened in the private development history; the public repository begins at the frozen state, so the verifiable commitment is the tag and the hash - and any change made after the freeze will be a public diff, exactly as the lock’s own header demands (a change == a new experiment).

The latency cap: closing the confound. As flagged earlier, a symmetric per-arm latency cap (Global ≈ Bielik) bounds the response-time tail so a slow model can’t bias resolution through user abandonment. Without it, a Q2 result could be a speed artifact: “Bielik resolved less” could really mean “Bielik made people wait.” The cap is the same on both arms, so it constrains the confound without itself becoming one.

The judge is a secondary lens, and it has prerequisites. We run an LLM-as-judge scoring four dimensions (grounding, correctness, completeness, safety), weighted, with a pass threshold of ≥3.5. It explains why an arm wins or loses; the decision still rides on resolution rate alone. It also has prerequisites: it runs only after the real models are in, and not before we’ve validated it against ≥2 human annotators with Cohen’s kappa on a golden set. Until that validation exists, judge scores carry no evidential weight.

Opal or custom? The decision table

One architecture decision was about what not to use. Optimizely Opal, their AI-agent platform, is the obvious place to stand up a support assistant fast. We kept the experiment off it, and the reason is the principle that governs everything above: variable isolation. If Opal orchestrated one arm (its own RAG, its own prompt assembly, its own guardrails) while a custom path served another, Q2 would become a comparison of two systems, and an orchestration layer that behaves differently per arm destroys the comparison no matter how good it is.

A widely-shared comparison, Architecting AI in Optimizely CMS: When to Use Opal vs Custom Integration on Optimizely’s developer-community blog (community content written against CMS 12; don’t read it as official product documentation), lands on the same split, and its framing maps onto our two phases:

You need…	Opal	Custom (this harness)
Decision-grade A/B/n with one isolated variable	✗ per-arm orchestration confounds Q2	✓ one `BuildPrompt`, one retrieval, only the model id varies
Control over the ML / model choice	Limited custom-ML (Medium control, per that comparison)	High control: self-host vLLM, pin params, swap model in config
Speed-to-ship / editor & marketer productivity	✓ fastest path, managed agents	✗ you build and run the plumbing yourself
Domain-specific, controlled retrieval grounding	Managed, less directly auditable per arm	✓ frozen Algolia corpus, recorded `contextHash` per answer
*Production home for the winning* arm**	✓ Custom Tool SDK (C#) + Bring-your-own-AI	✗ the experiment rig is not the production surface
Native editor/marketer reach in the CMS	✓ first-class	✗ out of scope for a measurement harness

The two columns describe a sequence. Custom wins the controlled experiment: high control and a single isolated variable are what a decision-grade test demands, and even that comparison reserves custom for precisely this kind of domain-specific, controlled-ML work. Opal wins production, after the experiment: the winning model arrives via Bring-your-own-AI, and the validated grounding and outcome tracking are exposed through the Custom Tool SDK (C#), so production reuses the retrieval and measurement the experiment validated, and the confound the study spent so much machinery eliminating stays eliminated.

Never route the decision-grade Q2 through Opal. Opal is downstream of the result; it operationalizes the winner.

Finally, a critical-path dependency: Bring-your-own-AI enablement runs through Optimizely, and the path is unclear from the outside. Public CMP pages show a self-serve “Connect Now” flow, while field reports describe CSM-assisted enablement, and it plausibly differs by product and contract (CMP, Opal and CMS 13 are three different surfaces; don’t conflate them). The experiment can run to completion without it, but the production rollout of the winning arm is blocked until it lands, so verifying your enablement path is a long-lead item for the start of the experiment. Lead time: unknown; we haven’t asked yet.

Results

Every cell below is a placeholder because the decision-grade run has not happened yet. The method is pre-registered and frozen: lock_sha256 is set, the gates are wired, the sizing is fixed to the NI bar. When the real run lands (prod models on real data), the numbers drop straight into these cells as the lock dictates, pass or fail. Nothing here is selected after the fact, because the table exists before the data. Until then, by the rules of the previous section, all of it carries the stamp. What we can already publish - real-hardware telemetry from the one-day dress rehearsal - follows right after these tables.

Pre-registered results: to be filled from the real run

Question	Result
Q1: does AI beat keyword search? (AI pooled vs Control, one-sided)	pending
Q2: does Bielik match/beat Global? (gatekept by Q1; NI at δ=1pp → superiority, fixed-sequence)	pending; tested only if Q1 rejects. A failed NI is published with its margin

Trust gates: to be filled from the real run

Gate	Reading
SRM (offline chi-square vs 20/40/40, flag p<1e-3)	pending
Contamination (excluded %, fallback by reason, off-flag-leak must read exactly 0, imbalance flags)	pending
Sizing satisfied (NI gate binding: 25,967/arm at the placeholder baseline)	pending
Latency cap held (symmetric Global ≈ Bielik)	pending
Judge: secondary quality lens (pass ≥3.5)	pending, only after kappa validation
Decision-grade status	`HARNESS_VALIDATION_ONLY`: real models for the one-day dress rehearsal, but simulated users + 12-doc corpus, as of this draft

The gates table is a checklist: a green Q1/Q2 means nothing if SRM failed, off-flag-leak isn’t zero, or the run never hit the NI sample size, which is why those rows sit next to the headline and not in an appendix.

The dress rehearsal: one day on real hardware (2026-07-20)

The verdict tables stay empty, but the harness has now met real endpoints once. On 2026-07-20 we deployed the whole stack for a single budgeted day: Bielik-11B-v2.3-Instruct (Apache 2.0) on one A100 via vLLM, bf16, on the Azure Container Apps workload profile Consumption-GPU-NC24-A100 at ~$3.7/h in region eastus (a US region; see Threats to validity), and gpt-4o-mini (model version 2024-07-18) behind Azure OpenAI, api-version 2024-10-21. These are the exact production dialects the code was written for. We then pushed 75 requests through the real path (50 simulated users, a 24-answer golden-set eval across both AI arms, one smoke test), twice as it turned out (sharp edge #5), analyzed the clean pass, and tore the stack down the same evening. And at ~$3.7/h there is no reason to keep the GPU up overnight.

Everything below is infrastructure telemetry, and none of it is an experiment result. Simulated users can’t resolve tickets, so none of this touches Q1/Q2, and it carries the stamp like everything else in this draft. What it does is replace four assumptions with measurements.

Measured latency, by arm actually served (simulated traffic + golden-set eval combined, clean second pass only):

arm	n	p50	p95	p99	avg completion tokens
bielik (11B, self-hosted, 1× A100, vLLM)	26	2,805 ms	3,923 ms	4,040 ms	~132
global (gpt-4o-mini, Azure OpenAI)	21	1,619 ms	3,087 ms	3,764 ms	~80
control (Algolia keyword, no LLM)	11	41 ms	63 ms	63 ms	n/a
human hand-off (low-confidence gate, pre-model)	17	42 ms	83 ms	83 ms	n/a

The record flow, so the arithmetic can be audited: the day wrote 150 exposure rows in two passes of 75. The first pass is quarantined whole. 21 of its Global calls 404’d on the api-version bug (sharp edge #5) and the rest of its rows are contaminated by retry and timeout artifacts, so none of it is tabled. The quarantine rule is mechanical (every row timestamped before the service restart that loaded the corrected api-version): one boundary, no row-level judgment calls, and nothing inside the clean pass was dropped. It also survives a sensitivity check in the other direction: Bielik’s first-pass numbers (n = 26, p50 2,835 ms / p95 4,035 ms) are statistically indistinguishable from the clean pass, so quarantining changes nothing for the arm that worked; it exists because the Global arm’s first-pass “latencies” measure retries against a 404ing endpoint. The clean second pass’s 75 rows split 26 bielik + 21 global + 11 control + 17 human hand-offs = 75. Hand-off rows are sub-100 ms by design (the low-confidence gate answers before any model is called), which is why they get their own row, away from the model latency columns.

latency (ms)
control p50 ▏41
global  p50 ████████████ 1,619
bielik  p50 █████████████████████ 2,805
global  p95 ███████████████████████ 3,087
bielik  p95 █████████████████████████████ 3,923
cap ────────────────────────────────────────────────────────────▏8,000 (never hit)

Four things this day settled:

A sovereign 11B on a single GPU kept up in this rehearsal. Bielik answered grounded Polish support questions at ~2.8 s median and under 4.1 s worst-case, inside the symmetric 8 s cap with zero timeouts on either arm in the clean pass. The cap can now be calibrated from measured data; p95 × 1.2 gives ≈ 4.7 s.
Most of the latency gap is answer length. Dividing completion tokens by wall-clock puts the two arms within ~5% of each other (≈47 vs ≈49 tokens/s end-to-end). Bielik simply writes ~65% longer answers at the same temperature and token budget, a style difference the judge will weigh in on.
The isolation seam behaves as designed. On the 12-question golden set, both arms answered the same 7 questions and declined the same 5 to a human, identical sets:
```
golden set   g01 g02 g03 g04 g05 g06 g07 g08 g09 g10 g11 g12
global        ✓   ✓   ✓   ✓   →H  →H   ✓  →H   ✓   ✓  →H  →H
bielik        ✓   ✓   ✓   ✓   →H  →H   ✓  →H   ✓   ✓  →H  →H
```
The identical sets are an architectural consequence: the low-confidence gate sits on the shared retrieval, upstream of any model, so decline behavior cannot differ by arm. The rehearsal confirmed the mechanism on real endpoints.
The economics of rehearsing are trivial. The complete pass (deploy, model load, traffic, eval, analysis) burned about 45 minutes of A100 time: roughly $3. The full day, config bug included, stayed far under the $40 cap. At that price, a rehearsal like this should precede any run pointed at real users.

One thing the day did not settle: anything about resolution. 75 simulated requests say nothing about answer quality and were never intended to. The verdict still requires real users, a real corpus, and the full gate battery above.

Threats to validity

A measurement is only as trustworthy as the limitations its authors name up front, so here is what we already know is wrong or unfinished:

Real models for one day; stand-ins the rest of the time. Still the biggest limitation, though it narrowed. The dress rehearsal ran the real Bielik-11B on an A100 and the real gpt-4o-mini for a single budgeted day, proving the swap is the config change we claimed. But the stack is torn down after each day, development still runs on stand-ins, and the rehearsal’s users were simulated. Q2 therefore remains HARNESS_VALIDATION_ONLY: we can now show Bielik’s latency, but not Bielik’s verdict, and any Bielik-vs-Global resolution number quoted today describes the harness.

Control is keyword search, and keyword search only. The Control arm is Algolia keyword search because keyword search is the incumbent users actually have today. It is not NeuralSearch, which sits behind an Elevate gate we don’t have enabled. Q1 therefore answers “does AI beat keyword search?”; the stronger question, “does AI beat the best available search?”, stays open. If your incumbent is already a strong semantic search, our Q1 lift will overstate what you’d see.

The corpus is 12 fictional documents. The Algolia index behind every arm holds the 12-record, fictional Izovell knowledge base described earlier. It is good enough to validate the harness; production behavior is a different matter. Retrieval quality, judge scores and resolution dynamics on a corpus this small do not transfer to a real knowledge base; the decision-grade run needs the production corpus indexed before ramp.

“Sovereign” carries two caveats in this rig. Retrieval calls Algolia’s SaaS in every arm, so the user’s query leaves the self-hosted boundary even when the model doesn’t. And the rehearsal GPU ran in Azure eastus, a US region permitted for the PoC via an explicit policy exemption, because that’s where consumption-GPU quota was available. What we can defend today is self-hosted inference under an open license (Bielik’s weights are Apache 2.0, though the Hugging Face download sits behind a light acceptance gate, a contact-consent checkbox as of this draft; an open license means you run the weights on your terms, which falls short of owning them). End-to-end residency is a production-deployment property: an EU region pin, a decision on in-boundary retrieval, and a logging path that honors the same border. The wording here is deliberately conditional and gets a privacy/legal review before any production claim.

The latency confound is bounded but still present. The symmetric cap stops gross asymmetry from biasing resolution, but it does not make the two models equally fast within the cap, and residual response-time differences below the cap could still nudge behavior at the margin. A sufficiently small measured effect could therefore still be partly a speed effect.

The judge isn’t annotator-validated yet. The LLM-as-judge is wired and weighted, but it has not been validated against human annotators; no Cohen’s kappa on a golden set yet. Until it has ≥2 annotators and an acceptable kappa, every judge number is provisional and stays secondary; an unvalidated judge will not influence the resolution-rate verdict.

Calibration values are still pending. Parts of the lock still hold [CALIBRATE-BEFORE-RAMP] placeholders, which is precisely why check_lock currently blocks ramp. The baselines that feed sizing and the calibration constants need real values before any of this is ready to scale.

None of these make the experiment unsound; they mark the boundary of what it can currently say. When the GPU swap lands, the judge clears kappa, and the calibration values go in, most of this section gets shorter, and it will be updated when that happens.

Sharp edges ranked by blood actually drawn

Ranked by the damage each one actually caused during development. Threat-model severity did not enter into it.

1. The bare-number trap (audit finding F1): nearly shipped a synthetic result as real. The most consequential of the audit findings. The decision-grade gate that downgrades a verdict to HARNESS_VALIDATION_ONLY on stand-in models was wired for Q2 only, which meant the Q1 number (“AI beats keyword search by +X points”) could be quoted bare off a synthetic run, with no marker, looking exactly like a real finding. A clean-looking +11.8pp from a generator we told to produce an effect is easy to mistake for a result. Fix: the gate now covers Q1 and Q2, every synthetic row carries a "Synthetic": true marker that propagates into the report, and the formatter refuses to print an ungated verdict.

2. The non-inferiority sizing trap: a false negative baked into the sample size. Our sizing math powered the Q2 superiority test (~6,400 users per arm at Q2’s p=0.70 placeholder baseline, a different test and baseline than Q1’s ~7.4k in the gates section) and we’d assumed it covered non-inferiority too. It does not: proving Bielik isn’t meaningfully worse at a 1-point margin needs roughly ~26k per arm. Shipped at the smaller number, the study would “fail to show non-inferiority” purely for lack of power and wrongly conclude the sovereign model can’t keep up. This was the most consequential statistical bug we found, and one that produces no error at all, just a wrong conclusion. Fix: the sizing layer now computes the NI requirement explicitly (worst-case true-equality) and the binding ramp number is Q2_required = max(...).

3. The freeze that wasn’t frozen (audit finding F2). We called the method “pre-registered” and “frozen,” and the audit asked what enforced that. There was no hash on the lock file, and the repo wasn’t even under version control, so “frozen” was an intention. Nothing enforced it, and an unenforced freeze is exactly what gets amended late to make a borderline result look clean. Fix: every report now carries a lock_sha256 of the pre-registration, the baseline rates moved into the lock (they were hard-coded in the sizing module), and the whole thing is committed under a git prereg-freeze tag, so “did the rules change after you saw the data” is a question anyone can answer with git.

4. The mislabeled golden set (audit finding F3). Our LLM-as-judge (a secondary measure) grades answers against a golden set of questions with known-correct knowledge-base documents. Seven of twelve questions pointed at the wrong KB doc: five referenced docs that don’t actually answer the question, two referenced topics absent from the corpus entirely. A judge calibrated against a wrong answer key is worse than no judge. Fix: corrected the expected document IDs (absent topics set to empty), bumped the judge artifact version, re-ran the validator clean; caught and fixed before any scoring ran.

5. The api-version that was a model version (found on real hardware, 2026-07-20). Found during the dress rehearsal itself. Azure OpenAI wants an ?api-version= query parameter; our config pinned it to 2024-07-18, which is not an API version at all but gpt-4o-mini’s model-snapshot date, copied into the wrong field with a comment noting they “match” (yes, really). Azure answers an unknown api-version with 404, so on the first live pass the Global arm failed 100% of its calls while Bielik ran fine. All 21 of them. This is precisely the differential failure mode the contamination gate watches for. Two things worked as designed: the fallback matrix degraded every failed call to Control (users would have seen plain search results, no error pages), and the fallback-by-reason telemetry read global_http_404=21, pointing straight at the cause within minutes. Fix: a real api-version (2024-10-21), restart, clean second pass; the affected first-pass rows were quarantined by timestamp. A config bug that kills one arm asymmetrically corrupts Q2 directly, and only per-reason, per-arm failure telemetry catches it before it reads as a “quality difference.”

6. The latency confound. A reliably slower arm makes users abandon before the answer lands, which reads as worse answer quality when it’s really speed. A symmetric per-arm cap bounds it without fully removing it; full treatment in The decision-grade gate and Threats to validity.

7. The Opal trap. Orchestrating one arm through Optimizely’s Opal would add its own retrieval/prompting/routing variance and turn Q2 into a comparison of pipelines. The experiment stays custom (a community “Opal vs Custom” comparison on Optimizely’s developer platform reaches the same split); full treatment in Opal or custom.

8. Simulated users. The models are now proven on real hardware (see The dress rehearsal), but every user so far is synthetic, so every number today carries the stamp by design (edge #1). The method is auditable and the rig has now run on live endpoints, but there is not yet a citable result; see Threats to validity.

FAQ

Questions as people actually type them.

“Isn’t an 11B model just worse than a big hosted model? Why even bother?”

Possibly. That’s what the measurement is for. “Bigger model wins” is just a prior, and on a narrow, retrieval-grounded support task, where the model mostly summarizes documents it was handed, raw parameter count matters less than you’d think. Nobody knows until the run happens, including us.

“Why is the sample so big? ~26k per arm sounds like overkill.”

It’s the price of the question. Proving something is better is statistically cheap; showing it’s not meaningfully worse at a tight 1-point margin is expensive, and clustering by user inflates the bill further. The large sample is what makes a null result interpretable at all.

“What does ‘decision-grade’ actually mean here?”

A verdict only counts if it ran on production models AND real data. Anything else (stand-in models, synthetic data, a forced dev override) gets stamped HARNESS_VALIDATION_ONLY and cannot be quoted as a finding. The gate is enforced in code. Right now everything is below that bar, and the report says so on every line.

“Is Bielik actually private/sovereign, or is that marketing?”

It’s architectural, and conditional. Inference is self-hosted: the prompt, the retrieved documents and the generation stay on a GPU you run, with no per-token egress to a third-party model API. Two caveats, disclosed in Threats to validity: the user’s query still goes to Algolia for retrieval in every arm, and residency is decided by where your GPU physically runs (our rehearsal box sat in a US region, which a production EU deployment must fix). “Sovereign” here means openly licensed weights (Apache 2.0; the HF download does sit behind a light acceptance gate), self-hosted inference, an auditable boundary, a property you can go and verify yourself. Whether it’s good enough is the separate question this experiment answers.

“You haven’t run it yet, so why publish anything?”

Because publishing the method before the result is what makes the eventual result trustworthy: you can check that we didn’t move the goalposts, because the goalposts are hashed and committed. A frozen method that can still turn out wrong is more credible than a polished result you can’t audit.

“You ran real models on a GPU for a day, isn’t that ‘the run’?”

No. The rehearsal made one of the two decision-grade conditions true (real models) while the other stayed false: simulated users, 12 fictional documents. It bought the things a rehearsal is supposed to buy. Real latency and cost numbers, proof the swap is a config change, the isolation seam holding on live endpoints, and one genuine bug fixed for ~$3 before it could surface mid-experiment. It cannot buy a resolution verdict, because simulated users don’t resolve tickets. The stamp stayed on every number, which is the gate working as intended.

“What happens if Bielik wins or loses?”

If it matches or beats Global, the winner ships to production through Opal’s C# Custom Tool SDK, and the story becomes “you have a measured, self-hosted, in-region alternative to a rented model.” If it loses, that’s a publishable finding too: “the sovereign option isn’t there yet, here’s by how much.” Either way the method is the same, which is why it was frozen first.

The take

We are publishing the method before the result. A result is only as believable as the method committed to before anyone saw it; ours is hashed and committed under the prereg-freeze tag, so when the numbers arrive the method can be diffed against what was frozen.

The same test applies to sovereignty. Most of the sovereign-AI conversation happens on slides, at the level of principles, and a clustered, pre-registered non-inferiority test against a global model on identical inputs settles almost none of it, because almost nobody runs one. If Bielik wins, the claim finally has a measurement behind it. A loss is publishable too. It would put a number on how far “owned” still sits from “rented,” and people rarely publish that number. Either way, what makes the outcome publishable is the gate that keeps stamping HARNESS_VALIDATION_ONLY on our own numbers until they qualify.

Glossary

ITT (intention-to-treat): Analyzing users by the arm they were assigned, even when they effectively got a different one, so dropouts don’t bias the result.
SRM (sample-ratio mismatch): A guardrail check that real bucketing matches the intended split (e.g. 20/40/40); a mismatch means something is broken upstream.
Non-inferiority: A test that asks “is the new thing not meaningfully worse than the old thing within a margin,” rather than “is it better.”
Variable isolation: Holding everything constant (prompt, retrieval, corpus, questions) so the only difference between two AI arms is the model itself.
Decision-grade: A verdict that ran on production models and real data; anything less is stamped HARNESS_VALIDATION_ONLY and can’t be quoted.
RAG (retrieval-augmented generation): Giving a model relevant documents to ground its answer, so it does not have to answer from memory alone.
LLM-as-judge: Using a language model to score answer quality against a golden set; here a secondary signal with no say in the verdict.
Feature Experimentation: Optimizely’s experimentation platform; provides deterministic, frozen bucketing via a versioned datafile.
Randomization inference: Getting a p-value by re-shuffling assignments many times (permuting clusters, with rows kept together), without assuming a textbook distribution.
Fixed-sequence gatekeeping: Testing hypotheses in a pre-registered order, each at full α; a later hypothesis is formally tested only if the earlier one rejects. Controls family-wise error through the hierarchy alone, with α left unsplit; this plan’s choice for {Q1, Q2}.
Holm correction: A way to control false positives when testing multiple hypotheses, less conservative than plain Bonferroni; this plan’s original choice, replaced by gatekeeping (see the sizing gate).
Sovereign LLM: A model with openly licensed weights (e.g. Bielik, Apache 2.0, from the SpeakLeash project) that you host and run yourself, keeping inference (though not necessarily every auxiliary service around it) inside infrastructure you choose.