June 29, 99
Concerning the Statistical Test that was Published
in our Paper in Statistical Science
by Doron Witztum
1. The Accusations of McKay et al:
In their article, "Solving the Bible Code Puzzle" which is
scheduled to appear in the May 99 issue of Statistical Science, McKay,
Bar-Natan, Bar-Hillel, and Kalai (MBBK) write the following about the
statistical test that Witztum, Rips and Rosenberg (WRR) utilized:
"To correct the error in treating P1-4 (that is, P1,
P2, P3 and P4) as probabilities, Diaconis
proposed a method that involved permuting the columns of a 32X32 matrix,
whose (i,j)th entry was a single value representing some sort of
aggregate distance between all the appellations of rabbi i and
all the dates of rabbi j. This proposal was apparently first made
in a letter of May, 1990 to the Academy member handling the paper, Robert
Aumann, though a related proposal had been made by Diaconis in 1988. The
same design was again described by Diaconis in September (Diaconis, 1990),
and there appeared to be an agreement on the matter. However, unnoticed
by Diaconis, WRR performed the different permutation test described in
Section 2." (Section 3)
Further on (Section 10), they name the test that was published in our
article "the test invented by WRR," as opposed to "
the permutation test of Diaconis."
Even before this, Dr. McKay had written the following letter, which
was published in "Galileo" (issue #27, March-April 98):
"A Grave Mistake:
My intention is to correct a grave mistake in Doron Witztum's
letter to the editor ('A Refutation Refuted', Galileo #26, p. 75).
He claims that the experiment involving the bible code that he published
in Statistical Science utilized a test that was devised by Prof. Diaconis.
This assertion is false and extremely misleading. Documents, copies of
which I have in my possession, demonstrate that he overlooked Diaconis'
test and set up another test, the result of which was hundreds of times
better. The data had already been in his possession for three years. All
of this has already been confirmed by individuals who are knowledgeable
in this matter, among them his colleague in the research and the article.
Mr. Witztum, however, continues to tell the same lie. This testifies to
his integrity." (translated from the Hebrew.).
Dr. McKay's letter (which, by the way, is typical of his style) should
surprise us for its impudence. Dr. McKay can assume that the readers of
Galileo have no idea what the experiment was that was agreed upon
with Prof. Diaconis. He should not, however, allow himself to assume that
the readers of Galileo have faulty memories. Here is a quote of
my exact words that were published in that issue of Galileo:
"After a great success in measurement for the second list as well,
Prof. Diaconis suggested that we use a new method of measurement, and
try it out on the second list. This is what we did, and the surprising
results of the experiment...." (translated from the
Now let us compare my statement to what Dr. McKay himself wrote
(together with Dr. Bar Natan and Prof. Bar Hillel, in their article in
Galileo #25, p. 53):
"Prof. Persi Diaconis, world renowned mathematician and statistician...
suggested another method to them, which they used in the paper that
was published in Statistical Science." (My emphasis. This quote
was also translated from the Hebrew.)
Amazing? - But this isn't all. We will now unfold before the reader
with entire course of events, and clarify what in truth the documents
in the possession of MBBK prove.
2. The truth about the statistical test that was published in our
paper in Statistical Science
Our paper, "Equidistant Letter Sequences in the Book of Genesis,"
was submitted for publication to PNAS by Prof. Robert J. Aumann,
professor of mathematics at the Hebrew University of Jerusalem and member
of the American National Academy of Sciences. Within the framework of
this attempt, he had discussions and correspondence with the referee,
Prof. Persi Diaconis. In September 90, the stage at which they concluded
their discussions, they were both at Stanford University, and they exchanged
letters as follows:
Prof. Persi Diaconis sent the following letter to Prof. Aumann on September
Professor Robert Aumann
Department of Economics
Mail Code 6072
Stanford, CA 94305
I am glad to report we are in agreement about the appropriate testing
procedure for the paper by Rips et al. A permutation test is to be performed.
There are four basic sets of data/test statistics, I will call them additive,
multiplicative, with and without Rabbi. For each there is a 32X32 table
of distances. It is my understanding that for each such table, one million
permutations will be performed. For each permutation SIGMA
be computed. This gives one million numbers/table. Again for each the
number SIGMA ti will be
located. If it is within 1/4000 of the smallest table sums, that test
is judged a success. If one of the
four tests is successful, the whole experiment is.
In case of ties, the interval of ties will be broken at random. If half the
proportion of such breaks
amount to better than 1/4000, that table is successful. Otherwise not.
I hope that the authors agree to make their findings public no matter
what the outcomes. Please let me know when you need from input from me.
A number of things are not clear in this letter. For example: 1) It
isn't clear what the intention is in the adjective "additive,"
vis a vis one of the statistics. 2) It isn't clear what the "distances"
are that make up each table. 3) It seems that he is referring to four
different tables, and it isn't clear at all what he is referring to. 4)
It isn't even clear which list of names/dates is to be used in the suggested
In order to clarify all of these, Prof. Aumann wrote the following letter
to Prof. Diaconis on September 7, 90:
Professor Persi Diaconis
Department of Statistics
Stanford, CA 94305
Thanks for your good letter of September 5, about the paper submitted
by Rips et al. to the PNAS.
Since it's important to clarify the precise rules of a statistical test
before performing it, allow me to set down here a few points of clarification.
The same 1,000,000 permutations may be used for each of the four basic
tests. The million will consist of the identity permutation plus 999,999
others. All million will be different from each other.
The sample to be examined is that of their "second experiment"
(Table 3 of their submission). For each of the four basic tests, the
exact same procedures as reported on in their paper (Tables 5 and 7) will
be done for each of the 1,000,000 permutations. (Incidentally, "bunching"
or "twenty percent" might be a more suggestive name for the
test you call "additive").
The precise tie-breaking rule (agreed on by phone today) is this: Out
of the million permutations, let there be s that are ranked smaller than
the identity, and t with which it is tied (excluding itself). Then the
test is successful if and only if s+(t/2) < 250.
Again, with many many thanks for all your help on this,
Bellow his name, Prof. Aumann added in handwriting:
"given to Persi by hand in Sequoia hall, September 9, 1990, 2:50
PM. He looked it over and approved."
That is, the details in Prof. Aumann's letter were approved by Prof.
When Prof. Aumann came to Jerusalem, he presented the above agreement
We thereupon wrote our paper anew, exactly according to the details
of the agreed upon experiment, in which "question marks" replaced
"results". During the course of 1990-1991, Prof. Aumann sent
this new paper to Prof. Diaconis and to other referees (all of them members
of the American National Academy of Sciences). They were all asked
to comment on the described experiment, and to establish a success
threshold. Prof. Diaconis didn't claim that we "overlooked
his test and set up another test". The experiment itself was performed
at the end of '91 after Prof. Aumann established, with the help of Prof.
Diaconis and two other referees, the statistical seed needed
to run the permutation test. The results were now incorporated into the
paper in the place of the question marks. All of this was reported to
the referees, who were asked to write a referee's report.
It is thus clear that the exact description of the experiment, as detailed
in the Statistical Science paper, was inspected by Prof. Diaconis and
the other referees before the experiment was performed.
To conclude this chapter, I will quote from a letter that Prof. Aumann
later wrote to Prof. Bar Hillel on January 17, 97, in which he describes
the chronology of our research in its various stages. At what he calls
Stages J and K, he writes:
"J. The details
of a formal test are agreed between Diaconis and Aumann (I'm trying to
avoid pronouns, because they often lead to confusion).
K. The formal test turns
out significant at a level of 16 out of a million. (That is, the best
result of the four statistics is 4 out of a million, and then Bonferoni.)"
3. Have MBBK seen the documents presented above?
Yes! On September 9, 93, Prof. Bar-Hillel received from Prof. Aumann
"all of my correspondences with Prof. Persi Diaconis concerning the
work of Rips et al" (Prof. Aumann's words on the accompanying letter).
If so, why are MBBK hiding the relevant information, and in its stead,
raising accusations that have no basis? It seems that they have no trust
whatsoever in their other claims. At any rate, if we might use Dr. McKay's
own words quoted above from his letter to Galileo, "This testifies
to their lack of integrity."