Causal Inference

Causal inference is a set of methods that estimate a cause-and-effect from data collected without randomisation.

A randomised test does one job: it makes groups comparable, so any difference can be pinned on the change. But in many situations randomisation is impossible, unethical, or already too late:

The whole field rests on one question: what would have happened to these same users if the change had not occurred? That "parallel world" is the counterfactual, and you can never observe it directly — the same customer either got the feature or didn't.

So every causal method is really a different way to build a believable counterfactual: find a group or period that honestly shows "what would have happened anyway." A good analysis = a good counterfactual. A bad counterfactual = a beautiful but false number.

Everyone knows the phrase, yet it catches people constantly. Three mechanisms that make an observed association lie:

Something drives both the "treatment" and the outcome at once. A product-analytics classic: "users of feature X retain better → the feature boosts retention." But feature X is chosen by the most engaged users — they'd have retained anyway. Engagement is the confounder, and it inflates the effect.

Groups differ from the start because of how they entered the analysis. Buyers of an extended warranty return items less often — not because the warranty protects them, but because more careful shoppers buy it.

A doesn't cause B; it's the other way round. "Customers who contact support often churn more → support drives churn." Actually, churning customers contact support more because they already have problems.

For each: when it works, what it needs, and where it breaks. No formula derivations — what matters is when a method is honest and when it deceives.

Idea: compare the change in the affected group with the change in a similar untouched group over the same period. The common trend cancels out.

When: a change hit one group/region, a similar untouched one exists, and before the change both moved in parallel.

Limit: rests on the "parallel trends" assumption — without the intervention the groups would have moved identically. Checked on pre-period history. If trends diverged, it lies.

Idea: if "treatment" switches on at a sharp threshold (discount above 5,000; status above N points), customers just above and just below the cutoff are almost identical. The jump in the outcome at the boundary is the effect.

When: there's a clear threshold rule. Very convincing, close to an experiment.

Limit: measures the effect only for those near the threshold — it may differ for everyone else. Needs lots of data around the boundary.

Idea: find an "instrument" — something that randomly nudges people toward treatment but doesn't affect the outcome directly. Through it, isolate the causal part.

When: there are hidden confounders and you can find a plausibly random nudge (e.g. an arbitrary delay in feature rollout).

Limit: a good instrument is hard to find, and its validity can't be fully proven — it's a matter of argument, not proof. The most fragile of the methods.

Idea: for each "treated" user find a statistically similar "untreated" one and compare pairs, balancing observed differences.

When: lots of user data and confidence that the key differences are measured.

Limit — critical: it balances only what you observe. A hidden factor (motivation, intent) remains and spoils the result. Often creates a false sense of rigour.

Idea: when one large unit is affected (a city, a country), build a "synthetic copy" from a weighted blend of untouched units that reproduces its pre-period history. The gap afterwards is the effect.

When: one big treated unit, many donor candidates, a long history.

Limit: needs a long, stable pre-history; sensitive to external shocks that also hit the donors.

The most used method in practice — and simple enough to show in full. Here's the idea as a formula, then runnable Python.

Four mean values of the metric — by group (treated / control) and period (before / after):

The first bracket is how the affected group changed. The second is how the background would have changed even without the intervention (the control estimates it). Their difference is the clean effect.

In practice DiD is run as a regression — it gives the standard error, p-value and confidence interval directly:

Here Treated = 1 for the affected group, After = 1 for the post period. The Treated × After term only "switches on" in the bottom-right cell — so its coefficient β₃ captures exactly the extra change that can't be blamed on the common trend or on the initial gap between groups.

Self-contained — needs only numpy. Data is generated inside (a true effect of +2.5 pp) so you can run it as-is and watch the method recover it. Swap the generated arrays for your own.

# Difference-in-Differences — needs only numpy.
import numpy as np
rng = np.random.default_rng(42)

# Synthetic: retention (%) in two groups, before and after a launch.
# The treated group has a true +2.5 pp effect baked in after launch.
def cell(base, trend, effect, after, treated, n=500):
    return base + trend*after + effect*(after*treated) + rng.normal(0, 3, n)

c_before = cell(30, 1.5, 2.5, after=0, treated=0)
c_after  = cell(30, 1.5, 2.5, after=1, treated=0)
t_before = cell(28, 1.5, 2.5, after=0, treated=1)
t_after  = cell(28, 1.5, 2.5, after=1, treated=1)

# Method 1: the 2x2 formula
d_treated = t_after.mean() - t_before.mean()
d_control = c_after.mean() - c_before.mean()
did = d_treated - d_control
print(f"Change in treated: {d_treated:+.2f} pp")
print(f"Change in control: {d_control:+.2f} pp (background)")
print(f"DiD effect:        {did:+.2f} pp")

# Method 2: same estimate via regression (adds SE, t and 95% CI)
y = np.concatenate([c_before, c_after, t_before, t_after])
n = len(c_before)
treated = np.concatenate([np.zeros(2*n), np.ones(2*n)])
after   = np.concatenate([np.zeros(n), np.ones(n), np.zeros(n), np.ones(n)])
X = np.column_stack([np.ones_like(y), treated, after, treated*after])

beta, *_ = np.linalg.lstsq(X, y, rcond=None)
resid = y - X @ beta
sigma2 = (resid @ resid) / (len(y) - X.shape[1])
se = np.sqrt(np.diag(sigma2 * np.linalg.inv(X.T @ X)))[3]
print(f"\nRegression effect = {beta[3]:+.2f} pp")
print(f"  SE={se:.3f}, 95% CI = [{beta[3]-1.96*se:+.2f}, {beta[3]+1.96*se:+.2f}]")

Causal inference is powerful, and that's exactly why it's dangerous: it returns a confident-looking number even when the assumptions don't hold. Keep in mind: