MBBK’S “STUDY OF VARIATIONS”

DORON WITZTUM AND YOSEF BEREMEZ

 

Abstract:
A paper of McKay, Bar-Natan, Bar-Hillel and Kalai (MBBK) in the Statistical Science (1999) made the extraordinary claim that the result of Witztum, Rips and Rosenberg (WRR), which was published in the same journal in 1994, was obtained by deceit. The main statistical work presented in MBBK’s paper is the "study of variations", aimed at proving that WRR’s result was obtained through “tuning” of their data.
In reply, we argue that MBBK’s case is fatally defective, indeed that their results merely reflect on the choices made in designing their “study of variations”, collecting the data and presenting the results. We present extensive evidence in support of that conclusion. In particular, we report on many experiments of our own, in which we applied their “study of variations” to several lists of data, some of them “tuned” and some of them not “tuned” – and the results are the exact opposite of the expectation of MBBK’s thesis.

 

CONTENTS:

•   Introduction  1
•   Logical and statistical flaws in MBBK’s thesis  3
•   Mistaken and misleading sampling in MBBK’s “study of variations”  9
•   Misrepresentation of data in MBBK’s “study of variations”  27
•   Experiments that refute the “study of variations”  39
•   An alternative model: the evolution of the “study of variations”  48
•   Conclusions  57
•   Appendix  57
•   Acknowledgments  58
•   Bibliography  58
•   Supplement I: Sec. 7 of MBBK’s paper (“The study of variations”)
•   Supplement II: Appendix C of MBBK’s paper

 

 

 

Back to top