Let a buyer-intent question \(q\) (for example, "What is the best project management tool for a small agency?") be submitted to a generative engine \(e\) (ChatGPT, Gemini, Claude, Perplexity). The engine's decoding process is stochastic: sampling temperature, retrieval variability in web-connected modes, and continual model updates mean the returned answer is a draw from a distribution over texts, not a deterministic lookup.
For a brand \(b\), define the mention indicator of a single generation:
$$ Y_{b,q,e} \sim \mathrm{Bernoulli}\!\left(p_{b,q,e}\right), $$where \(p_{b,q,e}\) is the probability that engine \(e\), queried with question \(q\) in its default, unpersonalized configuration during a defined measurement window, produces an answer that names brand \(b\). The estimand is explicitly conditional: on the question text, the engine and whatever model version it serves during the window, and the absence of user-specific context. It is not a claim about any individual user's personalized answer. Everything downstream (competitive position, trend, the effect of interventions) is a function of \(p\), which is never observed directly. Any honest product in this category is an exercise in estimating \(p\) and communicating its uncertainty faithfully.
Most tools in this category ask each question once and report the result as a score. A single Bernoulli draw has variance
$$ \operatorname{Var}(Y) = p(1-p), $$maximized at \(p = 0.5\), exactly the contested middle where competitive questions live. The one-draw estimator \(\hat p \in \{0, 1\}\) can only ever report "always" or "never". Using the (uncorrected) Wilson score construction of Section 3, a single observed mention yields a 95% interval of approximately \((0.21,\, 1.00)\), and a single non-mention yields \((0.00,\, 0.79)\). One observation is compatible with the majority of possible truths. Presenting it as a stable score is not a simplification; it is a misrepresentation of the underlying process.
Repeated generation shrinks this uncertainty at the usual \(\sqrt{n}\) rate. OTW Signal collects \(n = 8\) generations per question per engine, chosen as the point where the marginal reduction in interval width per additional dollar of inference cost flattens. The generations are issued as separate, stateless requests and are treated as conditionally independent given the question, engine, and measurement window; we discuss the plausibility and limits of that assumption in Section 8. At \(n = 8\) the worst case (4 mentions of 8) gives a Wilson 95% interval of \((0.22,\, 0.78)\), a half-width of about \(0.28\): wide enough to stay honest, narrow enough to separate "AI names you most of the time" from "AI rarely names you", which is the decision-relevant distinction for a business. Aggregating over a battery of \(m \geq 4\) questions further tightens brand-level estimates (Section 5).
With \(k\) mentions in \(n\) draws, the naive (Wald) interval \(\hat p \pm z\sqrt{\hat p(1-\hat p)/n}\) fails precisely where our data lives: it degenerates to zero width at \(\hat p \in \{0, 1\}\), which are common outcomes at \(n = 8\), and its coverage is badly below nominal for small \(n\) (Brown, Cai, and DasGupta 2001). We therefore report the Wilson score interval (Wilson 1927): the set of \(p\) not rejected by the score test, with closed form
$$ \frac{\hat p + \frac{z^2}{2n} \;\pm\; z\sqrt{\dfrac{\hat p (1 - \hat p)}{n} + \dfrac{z^2}{4 n^2}}}{1 + \frac{z^2}{n}}, \qquad z = 1.96. $$The Wilson interval has three properties that matter here: it never degenerates at the boundaries (observing \(k = 0\) of \(8\) yields "somewhere between 0% and about 32%", not "exactly 0%"); its coverage is close to nominal even at small \(n\); and it is contained in \([0,1]\) without ad hoc truncation. The Clopper-Pearson exact interval would also be defensible but is known to be conservative (wider than necessary) at our sample sizes; Wilson is the standard recommendation for this regime (Agresti and Coull 1998; Brown, Cai, and DasGupta 2001).
One production detail matters for exact agreement between this paper and the product: customer-facing displays apply the continuity-corrected Wilson variant (Newcombe 1998, method 4), which is strictly wider than the uncorrected interval and gives coverage closer to the exact interval at \(n = 8\). The correction changes numbers only in the conservative direction: 0 mentions of 8 displays as "0% to 40%" rather than the uncorrected "0% to 32%". All intervals shown in the product, including the pooled and competitor-leaderboard intervals, use this one variant, computed by a single shared, unit-tested implementation.
On interpretation: a 95% confidence interval is a statement about the procedure, not a probability statement about \(p\) itself. The procedure that generated the interval covers the true rate in approximately 95% of repeated measurement runs; any single reported interval either contains the true rate or does not. In customer-facing copy we render this as "the range of rates supported by these observations", which is informal but does not assert a posterior probability we have not computed.
The indicator \(Y\) requires deciding whether a returned text mentions the brand. This is less trivial than substring search. Our detection layer is deliberately precision-oriented, because false positives corrupt scores in the flattering direction, the direction users are least likely to challenge:
(?<![A-Za-z0-9]) and (?![A-Za-z0-9])), so "Asana" does not fire inside "Asanas".Define the brand-and-engine level estimand as the average mention probability over the fixed question battery \(B = \{q_1, \dots, q_m\}\):
$$ \bar p_{\,b,e} \;=\; \frac{1}{m} \sum_{q \in B} p_{b,q,e}. $$The battery is constructed to span distinct buyer intents (category search, comparison, local variants, use-case specific phrasings) and is shown to the customer with the report, so the estimand is fully specified: it is the average rate over these questions, in this window, on this engine. We deliberately do not claim it estimates the rate over the population of all questions real buyers ask; that population is not sampled, and generalizing to it would require a sampling model of buyer phrasing we do not possess.
The estimator pools counts, \(\hat p_{e} = \sum_q k_{q,e} \,/\, \sum_q n_{q,e}\). When per-question sample sizes are equal (the design target, \(n_q = 8\)), pooling is algebraically identical to the unweighted mean of per-question rates, so "equal weighting" and the pooled formula agree exactly. When engine failures produce unequal \(n_q\), the pooled estimator implicitly weights questions by completed generations; per-question counts ship with every report, so a reader can recompute the unweighted mean if the distinction matters for them.
The displayed interval is the Wilson interval on the pooled counts. Its validity for the battery-average estimand deserves justification rather than assumption. Because the battery is fixed (not sampled), the variance of \(\hat p\) under conditional independence is \(\frac{1}{m^2}\sum_q p_q(1-p_q)/n\), and by Jensen's inequality (concavity of \(x(1-x)\)) this is at most the pooled-binomial variance \(\bar p(1-\bar p)/(mn)\) that the Wilson construction assumes. The pooled interval is therefore weakly conservative for the fixed-battery estimand: heterogeneity across questions makes it wider than necessary, not narrower, and the continuity correction applied in the displayed version widens it further, preserving the conservative direction. The anti-conservative failure mode would arise only if we claimed the interval covered a superpopulation of questions; we do not make that claim, and per-question intervals are published alongside the pooled one so between-question heterogeneity is visible rather than hidden.
The customer-facing composite score combines this measurement (0 to 60 points, the dominant component) with a deterministic technical-readiness audit of the site itself (0 to 40 points: crawlability, structured data, entity clarity, and related checks). The two components are reported separately as well as combined, and engines that return no valid answers during a run are excluded from aggregation rather than silently counted as zeros, which would deflate scores with outage noise rather than signal.
Raw mention rates confound three causes: how strong the brand is, how difficult the question is (a generic "best tool" question names market leaders; a niche use-case question names specialists), and how liberally a given engine names brands at all (engines differ systematically in list length and hedging). To separate these, we fit a cross-classified logistic model with additive brand, question, and engine effects. It is Rasch-family in spirit (Rasch 1960; Lord and Novick 1968), structurally closest to a many-facet Rasch formulation (Linacre 1989), but it is not the classical two-factor Rasch model, and we describe it by its actual form. Each generation is modeled as
$$ \Pr\left(Y_{b,q,e} = 1\right) \;=\; \sigma\!\left(\theta_b \;-\; \delta_q \;+\; \gamma_e\right), \qquad \sigma(x) = \frac{1}{1 + e^{-x}}, $$where \(\theta_b\) is latent brand visibility strength, \(\delta_q\) is question difficulty, and \(\gamma_e\) is engine leniency. The linear predictor is invariant to location shifts among the three parameter sets, so identification requires constraints; we anchor \(\sum_q \delta_q = 0\) and \(\sum_e \gamma_e = 0\), leaving \(\theta\) on a common scale whose location is defined relative to the battery and engine set. Parameters are estimated by penalized maximum likelihood with an L2 (ridge) penalty, equivalently the MAP estimate under independent Gaussian priors; we use the frequentist description since we report point estimates, not posterior distributions. The penalty stabilizes estimates for brands observed on few generations, shrinking extreme cells toward the center rather than letting \(k = 0\) send \(\hat\theta \to -\infty\).
Status and evidence. The latent model is a research layer, not the production scoring path: customer reports and published Index rankings are computed from raw counts and Wilson intervals as described above, and the latent model's outputs are reported alongside them. In a preliminary internal evaluation on the first Index dataset (30 brands, 4 engines, 8 generations on each of 4 questions per engine), the penalized model achieved lower held-out log-loss than a smoothed per-cell empirical-rate baseline under question-level cross-validation, and its brand ordering was strongly concordant with raw pooled rates, diverging mainly where engine coverage differed across brands. We label these findings preliminary: the dataset is one category in one window, and we will publish the evaluation protocol and results with a future Index edition rather than ask readers to take the summary on faith.
The quarterly OTW AI Visibility Index applies the same machinery at category scale: a fixed public question battery per category, identical sampling across all brands, engine responses collected in the same time window to control for model-version drift, engines that return no valid answers excluded run-wide, and ranks computed from raw pooled mention rates and published with the underlying counts so any reader can recompute both the rates and the intervals. No brand, including ours, can pay to alter a rank; our own product's score (0% at first measurement) is published on the same terms.
| Threat | Nature | Mitigation |
|---|---|---|
| Temporal drift | Engines update continuously; \(p\) is time-indexed | Timestamped runs; same-window collection for benchmarks; subscription product re-measures weekly |
| Prompt-battery bias | Estimates are conditional on the chosen questions | Battery spans distinct buyer intents; equal weighting; battery published with results; \(\delta_q\) modeled explicitly |
| Residual dependence | Generations are treated as conditionally independent given question, engine, and window; shared server-side state, caching, or correlated retrieval could induce positive correlation, which would make intervals narrower than the truth | Stateless, separately issued requests; no conversation history; the assumption is stated rather than hidden, and battery-level aggregation limits the damage any one correlated cell can do |
| No personalization | Real users' answers condition on history and location | We measure the default, unpersonalized configuration, the common component underlying personalized answers; local-intent questions carry locality explicitly in the question text |
| Detection error | Alias collisions, negated mentions | Precision-oriented rules (Section 4); raw transcripts shipped for audit |
| Position effects | Being first in an answer is worth more than being tenth | Current estimand is mention probability; rank-weighted variants are under evaluation and will be reported separately, never silently blended |
| Small-sample noise | \(n = 8\) per cell is deliberate, not large | Uncertainty always displayed; pooling across questions; shrinkage in the latent model |
Every customer report ships the raw engine answers used in its computation, linked from each number. The Index publishes per-brand counts. The estimand, the sampling design, and the interval method are documented here precisely so that a skeptical reader can recompute any score we publish. We consider this the minimum standard for selling measurement, and the absence of this standard elsewhere is the reason this document exists.