The statistics of online experiments

Introduction

Online experiments are widely used for product development in the tech industry. As a result, a large literature on practices and increasingly advanced methods has developed over the past 20 years. Much of this literature is summarised in this review article by Larsen and coauthors, and the standard textbook treatment on online experiments is Kohavi, Tang, and Xu. There is also a thriving small community of practitioners who discuss new ideas on LinkedIn. In addition, aspects of the statistical foundations of online experiments are covered in excellent textbooks online (here, here, here) and in print (e.g. here, here, here), as well as in publicly available class notes (here, here). Finally, there is an excellent repo that collects resources on causal inference here.

What I could not find is a reference that covers the statistical foundations of online experiments concisely but without shortcuts (e.g. by including step-by-step proofs of key results). This is what these notes aim to provide.

I created them mainly for my own reference, thinking that they might evolve over time as I deepen my understanding of key concepts and have time to add further content – we’ll see how that goes. In the meantime, if you find them helpful, find errors, or have any suggestions, please get in touch by writing to experimentstats.playpen453@passmail.net.