Skip to main content
Read this before modelling anything from search data. Each rule exists because breaking it once produced a confident, internally consistent, statistically significant wrong answer. The instances described are real; the specifics are generalized so the patterns transfer. The same content is available mid-session as the resource aga://guides/analysis-guardrails. For the instrument bias underneath these patterns, see GSC data fidelity.

​
De-trend before reading any age curve

Rule: when the population-level metric has a strong time trend, an “X vs age” curve measures the trend, not age. Index each unit-day against the same-day population average: 1.00 = the average live unit that calendar day. Instance. On a site with orders-of-magnitude impression growth over the window, the raw days-since-publish curve rose monotonically and peaked around day 90 — content improving forever. False, twice over: the oldest cohort’s day 90 landed in the site’s strongest month (calendar confound), and only a fraction of articles were old enough to have a day 90 at all (survivorship). De-trended, the same data showed a peak at day 1, a drop to under half of peak within the first week, a trough in the second week, and a slow recovery after. Same data, opposite shape.

​
Balanced panels; report N at every point

Rule: if the set of units contributing to each x-value changes, the curve conflates shape with composition. Recompute on balanced panels — the same units observed for the full horizon — cross-check at multiple horizons, and report N at every point. Instance. Cohort sizes can collapse across the window as fewer units reach the longer horizons. Recomputed on only the units observed for the full horizon, and cross-checked at shorter horizons with larger N, the shape (spike → trough → recovery) held at every horizon — which is what makes it believable. The same discipline applies to rate comparisons: compare cohorts at a matched horizon, not lifetime — a lifetime comparison that read as underperformance became parity at a matched horizon.

​
Censoring at the data edge

Rule: the last bucket of every cohort line is partially observed and trends to zero mechanically — those points could not be anything else. Track observed/potential exposure per cell and label censored cells. Instance. A chart showing “every cohort decays to ~zero” reversed once censoring was marked: with the data ending mid-month, no recent cohort had completed its longer windows, so their late buckets could not be anything but near-zero. Recomputed with exposure tracked, the oldest cohort rose with age. The headline finding flipped on this one correction.

​
Placebo tests gate every event study

Rule: run the estimator on a date where nothing happened. If it shows an effect, the estimator is dead — no exceptions. Instance. An estimator reported a large, “statistically significant” negative effect with tight confidence intervals across several post-windows — and reproduced essentially the same effect on a placebo date where nothing happened. The adjustment had failed to absorb regression-to-the-mean and manufactured the result; discarded. The design that was kept returned placebos scattered around zero, so its real-date estimate (null, with a wide interval) is trustworthy. The placebo row is what separated the two. Without it, a dramatic significant finding would have gone to the client.

​
Medians on zero-inflated data: report both populations

Rule: with heavy zero-inflation, a median across all units measures the share of zeros, not the central tendency of the active ones. Report mean AND median, for all units AND active units, and label which population each statistic describes. Instance. A median page’s impressions collapsing across equal windows read as “pages going dark”; among pages actually active, the median declined far less. Mean/median divergence is itself a finding — the mean rising while the median falls toward zero means a few winners compounding while the typical unit flatlines. That gap only appears if you compute both.

​
Distributions of individual ratios, never ratios of aggregates

Rule: never compare an individual unit’s ratio to a ratio of aggregate means. Aggregate a distribution of individual ratios instead. Instance. Comparing each treated page against a ratio of control aggregate means let a handful of large, growing control pages inflate the denominator — the estimate swung from dramatically negative (“almost no events positive”, completely spurious) to null once individual ratios were aggregated against matched controls. A single aggregation choice was the entire difference.

​
Net vs gross churn; absent from a window is not absent from the world

Rule: a net change in a count conceals two-way churn, and absence from a window is not absence from the world. Before claiming units disappeared: widen the window, separate gross flows, and quantify the impressions attached. Instance. “Dozens of pages stopped being shown” was more leaving and some entering — two-way churn behind a net count. And only a small minority of the leavers had zero impressions across a much longer window, carrying a small share of the basket’s impressions; the rest were active, just not in that short window. The short window overstated disappearance several-fold.

​
Validate every proxy against ground truth

Rule: a proxy for an unobserved variable must be checked against the real thing on a sample before it carries an analysis; report the mis-assignment rate. Instance. First-impression-date as a publish-date proxy mis-assigned nearly an entire cohort — the “new” cohort was older back-catalogue content finally surfacing — and silently dropped a large share of the corpus, because pages that never earned an impression had no proxy value. Mature content relabelled as new mechanically produces a decay curve. In the same analysis a second proxy WAS validated and held: no articles showed impressions before their recorded publish date, so publishDate itself was trusted.

​
Dilution vs decay

Rule: a falling per-unit average with a growing unit count is dilution, not decay — different cause, different fix. Decompose into total vs fixed-basket vs head/tail before concluding anything is deteriorating. Instance. Per-page averages fell across the whole corpus while a fixed basket was flat — a wave of new pages contributing almost nothing diluted the average. That is dilution, not decay. Existing pages were not losing traffic; new pages were not yet earning any. Both statements are true simultaneously, and only the decomposition distinguishes them.

​
Test competing interpretations as hypotheses

Rule: when a stakeholder proposes a competing interpretation, test it with data instead of defending the original reading. Instance. “The corpus is narrowing” vs “visibility is spreading to more queries”: tested directly, pages earning impressions grew substantially and newly published pages brought in hundreds of distinct queries the existing library had never touched. The challenge was correct — and the claim was withdrawn.

​
See also