The business story
Why this system exists
At a startup I ran growth analytics for, the funnel numbers told a 5x growth story — until segmentation showed that roughly half of all tracked events were the engineering team's own localhost and dev-environment traffic. Quoting the raw numbers to anyone — investors, the founder, the team — would have been reporting fiction. The same audit found a revenue-related event that had silently stopped firing: what looked like a conversion collapse was a dead instrument.
What I actually did (the original work)
A full tracking-inventory and data-quality audit of the product's analytics: enumerating every event, segmenting internal versus real traffic, reconstructing real user journeys, dating the instrumentation breaks, and publishing dated reports the team could act on — with the rule that a metric which can't be trusted gets flagged as unmeasured, never quoted as zero or passed along inflated.
The judgment calls
- Segment before quoting — always. No number leaves the report without the internal-traffic filter applied and stated.
- "Zero" and "unmeasured" are different facts. A step whose event died mid-month has an unknown rate, not a 0% rate. Confusing the two sends teams off to fix products that aren't broken.
- Findings carry evidence, not just verdicts. Every flag above shows the numbers that triggered it, so a skeptic can check the diagnosis instead of trusting it.
- Small anomalies are left alone. The detector requires volume far outside the population median before calling an account internal — flagging noise teaches people to ignore flags.
What went wrong along the way
The first pass at the original audit trusted the platform's own "active users" number and nearly shipped a growth summary built on it. The habit this demo encodes — diagnose the data before reading the data — exists because of that near-miss, and it later caught a second, unrelated tracking regression within days of it appearing.
What I'd change today
Detection here is heuristic (env labels, volume outliers, precursor-vs-target survival). The production version should also fingerprint office IP ranges and CI user agents at ingestion, and alert on instrumentation death within hours — a dead event found in a monthly audit already cost a month of unmeasured data.