Bayesian Thinking for Product Metrics: A Practical Engineering Intro

Why Point Estimates Mislead Product Decisions

A/B discussions often collapse into one conversion number, but this hides uncertainty and overstates confidence. Bayesian framing helps teams reason about ranges, not just single values.

Step 1: Represent prior belief explicitly

from dataclasses import dataclass

@dataclass
class BetaPrior:
    alpha: float
    beta: float

Step 2: Update posterior from observed outcomes

def update_beta(prior, successes, failures):
    return BetaPrior(
        alpha=prior.alpha + successes,
        beta=prior.beta + failures,
    )

Step 3: Compare variants by sampled probability

import random

def sample_prob_beats(a, b, n=5000):
    wins = 0
    for _ in range(n):
        va = random.betavariate(a.alpha, a.beta)
        vb = random.betavariate(b.alpha, b.beta)
        wins += 1 if vb > va else 0
    return wins / n

Pitfalls

• Using posterior probability as guaranteed uplift.
• No prior documentation for future experiment audits.
• Ending experiments early without decision thresholds.

Verification

• Posterior updates match known toy examples.
• Decision threshold is defined before experiment start.
• Reports include uncertainty ranges, not only point estimates.