ModelLab

A/B Testing Platform for ML Models

A production experimentation platform for ML models with a rigorous statistics engine spanning frequentist and Bayesian analysis, consistent-hash routing, and champion/challenger promotion.

Methods & contributions

• Frequentist core — Welch's t-test with Satterthwaite df, two-proportion z-test (pooled test SE, unpooled CI), and observed power via the noncentral t.
• Bayesian analysis — Beta-Binomial and Normal-Normal conjugate posteriors with P(B>A), expected loss, and HDI.
• Sequential testing — O'Brien–Fleming and Pocock alpha-spending, plus Bonferroni and Benjamini–Hochberg corrections.
• Routing & variance reduction — consistent-hash traffic assignment, CUPED, SRM detection, and Wilson / Clopper–Pearson intervals.

Data flow

traffic → consistent-hash bucket → variant
        → metrics → frequentist + Bayesian engine
        → sequential monitoring → champion/challenger promotion

Technologies