Bs"d, Sivan 5761 (June 2001).
Designed to Distract
(On McKay and Kalai's Response to our article
"New Statistical Evidence for a Genuine Code in Genesis")
By Doron Witztum

McKay and Kalai's response, titled "Replication of the famous rabbis experiment – a reply to Doron Witztum" [1], purports to answer our critical article [2] " New Statistical Evidence for a Genuine Code in Genesis". But it is merely a failed attempt to distract people from McKay et al's deceit which we discussed [2] and which they have still not explained.

I.         On the Working Procedures of McKay et al:

In our article [2] we detailed McKay et al's manipulations concerning their "replication" "according to Dr Emanuel". Here, we will briefly mention some main points. First, let us discuss the preparation of "list b" and "list c" which were intended to "mimic" the original second list of Prof. Havlin.

  1. McKay et al asked Emanuel to prepare appellations for 35 personalities and he did so and sent them the appellations. Thus McKay ordered appellations for this group of personalities, which constitute the total of several possible choices, in order to select only some of these appellations a posteriori.
  2. McKay et al indeed chose 33 rabbis from this list a posteriori.

    These two stages (A and B) absolutely contradict the procedure they reported in Statistical Science:
    "1.          A list of rabbis was drawn from Margaliot's encyclopedia by applying WRR's criteria for their second list, while correcting the errors they made. Our list differed from WRR's in dropping two rabbis and including three others. One rabbi who fits the selection criteria could not be included because he appears incorrectly in WRR's first list.
    2.         Emanuel was shown the spelling rules and table of appellations for WRR's first list as they first appeared in WRR (1986). He then compiled a parallel table of appellations for our list of 33 rabbis, attempting to follow the rules and practices of WRR's first list." (Emphasis mine).
    Note how in their response they attempt to implicate Emanuel in their misdemeanor [1]:
    "Nevertheless, after the experiment was finished, Emanuel approved the description of it that appeared in our paper."
    This is baseless: 1. Emanuel explicitly said several months after the publication of their article, that he never read it. 2. He was extremely surprised when he learnt that two rabbis from his list of appellations had been "dropped".

  3. The criterion for choosing the 33 rabbis was invented ad hoc and it contradicts the criterion they publicized on earlier occasions.
  4. Concerning the dates: Contrary to their response [1], they did not instruct Emanuel to check or emendate all the dates (see later section IV). They only did so concerning one rabbi (or a few).
  5. Concerning the appellations: Many peculiar decisions were made in "list b" (and "list c") which even Emanuel could not explain. Considering that the instructions given him concerning the dates were through "hints" ([2] chap. I sec. 1(B)), who knows how, and how often such "hints" were given concerning the appellations?
  6. McKay et al indicated two mistakes in Emanuel's "list b" and asked Emanuel to correct them. But these were not all the mistakes they detected. In section II we will see from their own words that there were at least two more mistakes in their opinion (involving the name "Ayash"). But they chose to correct only what suited them (See [2], chap. 3).
Paragraphs A-F above describe the direct manipulation of lists b and c.
But it should be mentioned that this was preceded by preparatory spadework:
  1. At an earlier stage, McKay et al ordered appellations for "list a" from Emanuel. They ordered them for only the 32 rabbis of Havlin's second list.
  2. Note that from the collection of appellations Emanuel gave them at this stage ("list a") it was possible to estimate which appellations would remain in the next stage for the 32 rabbis, after Emanuel would use the spelling rules and would work according to Havlin's first table of appellations and names. This is because McKay et al also understood Havlin's table and the spelling rules (in fact they knew them better than Emanuel, even suggesting corrections to his work).
  3. No protocol dictated that the experiments be continued on further lists, like b or c. Spadework like this would allow one to act unrestrictedly in the next stage. For example:
    • If the significance expected according to #2 is about 0.00002 (many appellations would be expected to survive in the next stage) - one can choose to simply not continue to the next stage.
    • If the significance expected according to #2 is about 0.002 (about half of the number of the appellations in Havlin's list could be expected to survive) - one can use steps A-F above, to "sufficiently" downgrade the significance.

II.         The "Ayash" Episode:

When I realized that McKay et al had secretly erased the appellations of R. Aharon of Karlin and R. Yehudah Ayash from Dr Emanuel's appellations lists ("list b" and "list c", see [2] chap. 1) without Emanuel's knowledge, I phoned Emanuel [3] and asked him to send me the original "list b" or "list c". But, acting on McKay et al's instructions, he refused.
So I tried to find out which were the missing appellations. In the course of our conversation:
Interestingly, for "list a" which was bound by absolutely no grammatical rules Emanuel wrote the name "Ayash" as follows: "שאייע" and "שאייאע". But for "list b" (and "list c") he was supposed to spell the names according to the spelling rules of WRR's first experiment – so write McKay, Kalai et al in their article in Statistical Science:
"2.         Emanuel was shown the spelling rules and table of appellations for WRR's first list as they first appeared in WRR (1986). He then compiled a parallel table of appellations for our list of 33 rabbis, attempting to follow the rules and practices of WRR's first list." (Pg. 163)
Thus Emanuel was supposed to do the final spelling according to the given rules and not according to his private opinion. Therefore, I concluded that Emanuel must have written those names as follows: " שאיע " and "שאיאע", because that is how the given spelling rules dictate. Thus I finally reported in [2] that "A conversation with Emanuel helped us to deduce the missing appellations".

Even more, McKay et al did the same thing themselves. In an article [4] describing their second list for War and Peace they write:
"The independent expert mentioned above used for the a-priori experiment the forms שאייע and שאייאע, but for our "experiment" here we follow the 'grammatical" dictum that Witztum has set in the case of Oppenheim, i.e. that must not use a "double yud" in transliterating foreign names, so we write שאיע and שאיאע." [Note that in [5] they admit that this spelling rule was already utilized in Havlin's first list.]
In other words, they write here that their expert, Dr Emanuel, used "שאייע" and "שאייאע" for his first a priori experiment ("list a"), and that according to the spelling rules it is written "שאיע" and "שאיאע".

Note that in their reply McKay and Kalai stress:
"Conclusion: Witztum did not use the spellings provided by Emanuel."
And then they hypocritically add:
"We also wonder why Witztum's article fails to mention that he changed the spelling."
They say this knowing full well that the only spelling publicized by Emanuel was in "list a", that is, before the spelling rules were applied. But after the spelling rules were applied the spelling had to change as they explicitly write themselves!
It is amazing that they basically claim in their defense that Dr Emanuel used his own initiative in the spelling and ignored the spelling rules which he had been requested to use, by… McKay et al!

(Note that, according to them, Emanuel's mistake was his reliance on spelling in an unvowelized text while the requested spelling was of a vowelized text. See further details about this in [6]).

III.         Concerning the Dialog with Dr Emanuel.

McKay and Kalai complain about my dialog with Dr. Emanuel, even though Kalai himself (and others who worked with MBBK) contacted Prof. Havlin and queried him on the phone concerning his work and his methods. Let us remind the reader:
  1. Havlin publicized his explanations to his lists after they raised questions about them. They, however, never asked Emanuel to prepare a list of explanations.
  2. They only publicized Emanuel's lists long after they were prepared.
  3. When I contacted Dr Emanuel and asked for his reasoning he said that so much time had passed that he no longer remembered.
  4. McKay et al concealed the "game" they had played with Emanuel's lists (see [2], chap. 2), so they were obviously angry when my direct contact with Emanuel exposed their fraud.
  5. In "Concerning McKay's Response to our article 'Of Science and Parody'" [6], we added more information from what Emanuel said which made it clear that the work of "Anonymous" (and McKay) relied on nonprofessional books. This is obviously embarrassing for McKay and it is understandable why he prefers that these things remain unknown.
In conclusion: McKay et al's procedures concerning Emanuel's lists were both procedurally and ethically wrong. When this was exposed, McKay and Kalai reacted not like scientists but rather like frustrated lawyers –– and resorted to mud throwing. We ignored their baseless and unsubstantiated insults.

IV.        Concerning their treatment of the dates:

Concerning their treatment of the dates McKay and Kalai claim as follows:
"The issue of dates is not much different. In their own experiment, WRR replaced or deleted some of Margaliot's dates on the basis of historical evidence. We did not originally ask Emanuel to do the same, but of his own accord he started to make comments about the inaccuracy of some of the dates in the Margaliot encyclopedia. Emanuel is an expert on such historical questions, so we then asked him to check all the dates. This gave us a compilation of dates of the best possible historical accuracy without the need for any subjective choice of our own."
But:

A.        Reminder:
In our article [2] (chap. 1) we already discussed the strange procedure which led to addition/omission/replacement of a few dates, a process which makes their date changes "contaminated" and useless for any experiment.
1.        This is the sequence of events according to Dr Emanuel quoted in [2]:
"…At a certain stage I understood that this matter interested them. Perhaps they asked it for one particular rabbi and I went and did the same for them all…"
Compare this to the what McKay and Kalai say:
       "…so we then asked him to check all the dates".

2.        Note that nowhere in their Statistical Science paper did McKay and Kalai ever mention that they asked Emanuel to check the dates. Only after the publishing of our article [2] was the following claim created:
       "…so we then asked him to check all the dates".

B.       Chronology of their dates' treatment:
  1. On 7 November '96, Dr Bar-Natan et al wrote to Prof. Aumann outlining a proposal to have all the dates relating to our experiment examined by an independent expert. We basically accepted this proposal, and in a letter of reply to Prof. Aumann (14 November '96) we added three minor remarks. But Bar-Natan et al retracted from their proposal.
  2. We consider this the reason they never had the audacity to ask Dr Emanuel to check the dates: It would be too obviously improper to retract from a mutually agreed examination and instead do it unilaterally.
  3. After Dr Emanuel examined some dates they showed some interest; but they did not instruct him to check them all. This left them the freedom to use:
    1. All date changes,
    2. Or, some of them,
    3. Or, none of them.
Conclusion: It is hard to imagine a more defective procedure.

V.        Concerning the accuracy of their dates' changes:

Let us reread the end of the above quotation of McKay and Kalai:
"Emanuel is an expert on such historical questions, so we then asked him to check all the dates. This gave us a compilation of dates of the best possible historical accuracy without the need for any subjective choice of our own."
But:
  1. True, Dr Emanuel is a historian. But Mckay et al should have made clear that his expertise lies in the Middle Ages, a period encompassing only three out of the 32 personalities in the second list.
  2. McKay and Kalai claim that they achieved:
    "a compilation of dates of the best possible historical accuracy". But this is their mere personal opinion and since they understand little of these matters –– it is worthless.
On the other hand one can demonstrate that there are mistakes in their "corrections".

An example:
McKay et al claim that the death date of R. Yehudah Hasid (no. 15 in the second list) is in doubt. Therefore they erase it. What creates this doubt? They briefly explain in their article in Statistical Science ([7], in the References, entry Gedaliah):
The source of his death date (relied on by WRR) is in the book of R. Gedaliah of Siemiatycze, "Pray for the Peace of Jerusalem" [8]. But the date there contradicts the date in another book called "The Travels of Rabbi Moshe Yerushalmi" [9].

Their conclusion that this creates a doubt is nonsense, as we will soon see. Therefore, we cannot believe that this "correction" was done by any qualified historian.

Because this case is obvious we will quote the sources and let the reader judge for himself.

1.         The book "Pray for the Peace of Jerusalem" of R. Gedaliah of Siemiatycze ("source A"), a close disciple of R. Yehudah Hasid, traces R. Yehudah's life in detail from when he arrived in Jerusalem:
In that year the first of Heshvan was on a Thursday, and therefore Monday was the fifth of Heshvan.

2.         But in the book "The Travels of R. Moshe Yerushalmi" (published in the year 5529, from now on "source B"), we find a different description:

According to this source he died on Friday, 2nd of Heshvan 5461. Therefore McKay et al claim (in Emanuel's name) that there is a contradiction between source A and source B, and therefore the date should be considered doubtful.

3.         The basic task of an historian is to weigh his sources. Let us compare them:

Source A: Source B:
How can a second hand testimony from a later period (decades later) contradict the detailed and reliable testimony of a witness (who both saw and heard)?
In conclusion, the weight of "source B" cannot compare with "source A" and cannot be considered a valid contradiction.

4.        Because it is hard to imagine that this "correction" was made with the sanction of any authentic historian, doubt is cast not only on beginning of McKay and Kalai's statement:
       "This gave us a compilation of dates of the best possible historical accuracy…"
but on its end as well:
       "…without the need for any subjective choice of our own."
Especially considering what we said above in section IV.

VI. The Standard of their Work:

McKay and Kalai declare:
"In our replication we tried very hard to apply WRR's rules correctly. One necessary exception was that Rabbi David Ganz had to be excluded because he was already in WRR's first list. (WRR excluded him for the same reason.) Otherwise, neither Witztum nor anyone else has found an error in our selections."
See our article [2] chap. 1 where we unequivocally prove that:
  1. MBBK ordered appellations for 35 personalities from Emanuel who prepared and delivered them.
  2. MBBK, a posteriori, chose 33 personalities from this list.
  3. Their criterion for this choice was ad hoc, and contradicted the criterion publicized by MBBK on earlier opportunities.
  4. Thus MBBK achieved the worst results possible.
Conclusion: It took a lot of nerve to write the above quoted assertion.

VII.          What Did Our Measurements Prove About Emanuel's List?

As we demonstrated in [2] and as we repeated and added in the preceding sections, the changes in the composition of the list and the dates are "contaminated" and unusable. The only possible use for Emanuel's appellations is in investigating the claim that WRR's success stemmed from "freedom" in choosing appellations.
       In [2] chap. 2, I described how we repeated WRR's experiment with just one difference: We used Dr Emanuel's appellations instead of Prof. Havlins's. The experiment succeeded to the probability of 0.0036. An additional measurement showed that this was consistent with the original experiment's result if one takes the size of the sample into account. (Concerning their claim that we altered the spelling of "Ayash", see above section II that this is not so: We utilized the necessary spelling rules as they themselves explicitly wrote. If they claim that Emanuel erred and misapplied the rules, this should obviously have been rectified).
       Thus we proved that the success of the original experiment was not because of "freedom" in choosing appellations: Emanuel's list was completely unbiased towards WRR; if there was any bias it was to MBBK's benefit.

VIII.         The Criticism against our Consistency Test:

McKay and Kalai raise several arguments against our consistency test described in [2] chap. 2:

A.        They claim: There is no mathematical rationale in comparing a randomly created sub-group to a group assembled by an expert. The latter has "strong internal structure" which affects its behavior in a way that a randomly assembled sub-group "cannot match".

But:
We think Emanuel's list has no "internal structure" which could influence ELS appearances, especially in proximity to specific dates. Therefore we see no need to answer this claim until McKay and Kalai give some demonstration of this "internal structure" in Emanuel's list.

B.        They claim: That Emanuel's appellations are, on the average, the most important and should therefore succeed more, according to our theory.

But:
  1. The assumption of McKay and Kalai that Emanuel chose the most important appellations is baseless. Emanuel omitted many major appellations. He gave no reason for his decisions and claimed that he could not remember his rationales. For more about this see [2] chap. 3.
  2. According to their theory, that there is no code phenomenon, these appellations are not "more important". So why does Emanuel's list succeed (in ratio to group size) as would be expected if there is a phenomenon?

C.        They claim: That the complement of Emanuel's list of appellations (that is Havlin's group minus Emanuel's group) succeeds 250 times better than Emanuels' list, even though both lists are similar in size.

But
:
  1. For accuracy's sake, the complement of Emanuel's list succeeds about 90 times better than Emanuel's list. By the way, the difference in size is: "Emanuel" has 63 word pairs and the Complement has 73 pairs.
  2. The question should be: Is the result for the Complement reasonable as compared with that for a randomly created sub-group of the same size?
    We repeated our test exactly as it is described in [2] chap. 2, and concluded that the probability of receiving such a result (or better) for a randomly created sub-group of Havlin's group, of the same size, is rather large: p=0.3.

D.        They claim: That they examined the odds of the data in claim C, according to our test, and received very small probability, less than one in 65,000. (By the way, even without repeating their measurements it is easy to see that there is a simple mistake in their calculation [10]).

But:
  1. What is the source of the result presented by McKay and Kalai and why is it different from our result in the previous paragraph?
    It turns out that they measured something completely different: They measured the probability of a randomly created sub-group of the Complement, with 63 pairs, succeeding the same as Emanuel's group (or less). This is a statistical mistake because it involves conditional probability.
    We will illustrate this with an extreme case: We have two mutually exclusive groups of word pairs. Group A has 1,000,000 pairs and group B has 1,000,001 pairs. We are given that:
    (a) The success of B is twice as that of A.
    (b) No single pair of B can raise the success of the rest of the group by a factor of two.
    Is there any sense in the following test: What is the probability that a sub-group of B, having 1,000,000 pairs, would have equal (or worse) success as A?
    It is clear in advance (according to (a) and (b)) that this cannot happen.
    But according to McKay-Kalai one would "measure" a "probability" of one in a million!
  2. Thus all that their "measurement" proved was what is obvious with no measuring at all: That the Complement group is stronger than Emanuel's group. The correct question should be what is the probability that a sub-group of this size and strength could be randomly chosen from Havlin's group. The answer, as stated before, is p=0.3.
References:
  1. McKay, B. D. and Kalai, G. (2001). "Replication of the famous rabbis experiment – a reply to Doron Witztum". Available at http://cs.anu.edu.au/~bdm/dilugim.
  2. Witztum, D. (2000). New statistical evidence for a genuine code in Genesis.
  3. A telephone conversation with Dr S. Emanuel, 31 Dec. '99, about 13:00.
  4. Bar-Natan, D. and McKay, B. D. (1999). Equidistant letter sequences in Tolstoy's "War and Peace". http://cs.anu.edu.au/~bdm/dilugim/WNP.
  5. Anonymous (1999). Equidistant letter sequences in Tolstoy's " War and Peace": Witztum's "refutation" refuted. (Assertion 4) http://cs.anu.edu.au/~bdm/dilugim/WNP.
  6. Witztum, D. (2001). Concerning McKay's Response
  7. McKay, B. D., Bar-Natan, D., Bar-Hillel, M. and Kalai, G. (1999). Solving the Bible Code puzzle. Statist. Sci. 14 No. 2 150-173.
  8. Yaari, A. (1946). Travels in the Land of Israel, Tel-Aviv, Israel, 328-329.
  9. Ibid, 448.
  10. The mistake: According to them they calculated 65,000 random sub-groups applying the given procedure in [2] (chap. 2, #3). But it is easy to calculate that according to this there are only less than 14,000 different choices!