Ó Doron Witztum, ‘98

In our article, "Equidistant letter Sequences in the Book of Genesis," which appeared in Statistical Science (Vol. 9 No. 3, Aug '94), we showed that there exists an intentional encoding of expressions in the form of equally spaced letters in the book of Genesis. We accomplished this by measuring the tendency of convergence between the names of famous rabbinical scholars and the dates of their birth and/or death (day and month). This tendency proved so strong that the chances of it being a random phenomenon are extremely minute, according to one measurement only 4 in a million.

The Nov. ‘97 of Galileo (an Israeli popular-science magazine) carried a critique of our research. The authors of the critique, Prof. Maya Bar-Hillel, Dr. Dror Bar-Natan and Dr. Brendan McKay, claim that the results of our experiment were achieved through manipulation, taking advantage of the "flexibility" which exists in the guidelines within which we were working, or by breaking these rules altogether. They base their accusations on a research project of their own, which they publicized via the Internet. In this work they claim to have produced, artificially, a level of "success" similar to ours in the novel War and Peace, by openly manipulating the data, while remaining "within the guidelines mentioned above, or by violating them to the same extent."

In our response published in the same forum, a summary of which will be presented later on in this article, we showed that:

1. The "success" in War and Peace was achieved solely through violation of the guidelines. Therefore there is absolutely no significance to their work. (They also admit that without guidelines one can produce whatever result one wishes).

2. On the other hand, they were not successful in identifying a single instance in which we violated the guidelines in our original research.

Paradoxically, their failure to produce a "success" while staying within the rules acts as solid evidence against their claim that within the established guidelines there is "enough choice to generate comparable significance levels in War and Peace."

* * *

A brief review of the stages leading up to the publication of our original article will facilitate an understanding of the discussion which follows:

A. Preliminary experiments indicated a tendency for conceptually related expressions appearing in equally spaced letters in the book of Genesis to converge on one another. A methodology was developed for evaluating the probabilities of these convergences.

B. We decided to examine this phenomenon with regard to convergences between the names and appellations of famous Rabbinic scholars and their dates of birth or death, as these appear in the form of equally spaced letters in the text. To this end a list of scholars was compiled using objective criteria. The list of these scholars was established a priori, by an expert bibliographer, Prof. Shlomo Zalman Havlin, who was, at the time the list was prepared, head of the Department of Bibliography and Library Science at Bar Ilan University. Guidelines for spelling and for indicating the Hebrew date were also established a priori by the linguist Yaakov Auerbach, of blessed memory. These experts operated without any prior knowledge of which names or appellations would "succeed" in the experiment, and they were certainly unfamiliar with our methodology for evaluating convergences.

C. After this list proved very successful in Genesis, and the results were published in a preprint, a well-known statistician, Prof. Persi Diaconis, requested that we carry out a new experiment using the exact same methodology as in the first experiment. This time, too, a list of names and appellations was prepared a priori by Prof. Havlin, employing the same principles and guidelines which he used in compiling the first list, and the dates were written in exactly the same format as had been previously established.

D. When the second list also proved very successful, Prof. Diaconis suggested that we use a new form of measurement, and that we apply it to the second list. An agreement was reached on exactly which test to perform. This test was carried out, and achieved the astonishing result cited earlier in this article.

[A detailed report on the fixed guidelines according to which Prof. Havlin operated, can be found here in Document 1. A description of the procedures and stages leading up to the publication of the Statistical Science article can be found here in Document 2: "Bar Hillel and Bar Natan Ask; Witztum and Rips Respond."]

Since that time many other experiments have been carried out, which also indicate the existence of an intentional encoding of expressions in equally spaced letters in Genesis (some of these will appear in Doron Witztum's forthcoming book, currently in progress). All of these experiments were carried out on the basis of objective and a priori lists (lists which, by the way, did not require the assistance of an expert consultant).

Any reasonable person will understand that the success of a list of word pairs in the book of Genesis, compiled according to pre-established guidelines, indicates that expressions have been intentionally encoded in the text.

On the other hand, by compiling a list without guidelines one can produce an artificial "success" in any text: There is enough latitude to allow for the presentation of successes and the concealment of failures.

The authors of the critique are well aware of this. Therefore their accusation is that the guidelines which were established for the sake of constructing the first list were flexible enough to allow for manipulation.

They attempt to base themselves upon a work which was presented via the Internet on September 20, '97, under the title "Equidistant Letter Sequences in Tolstoy's War and Peace," which they cited in the Galileo article.

This work, as they explain it, consists essentially of constructing a new list of names and appellations, which "succeeds" in War and Peace and fails in the book of Genesis. Without a doubt they invested a tremendous amount of effort in constructing this list. It was an effort which involved searching through many volumes and which took up a great deal of computer time for making the calculations. This list was prepared over the course of many months, and the published version was not their first version. They attempt to justify it with a list of twenty-four detailed assertions. They claim to have prepared their list (to use their own words) "by purposefully constructing our own list of appellations, staying within the WRR-stated rules or breaking them by about as much as they did" -- In short, they claim to have done "the same thing."

In the Galileo article, too, they sum up by saying that: "With the exception of the 'fine print', that is in the realm of small changes, all of which are no less justified, consistent and legitimate than those of Witztum and Rips, it is possible to replicate in the novel War and Peace, the same astonishing discovery which was made in Genesis."

Anyone who reads their words superficially is liable to come away with the impression that they did indeed accomplish their goal. But the more cautious reader knows that it is never a good policy to sign a contract or commitment before one has examined precisely what is stated in the "fine print." And indeed, a closer examination reveals that the changes made by the authors are: not minor, not justified, not consistent, and not legitimate. There is, in fact, no resemblance between what they did and what was done in the original experiment.

In our response mentioned above we show that their claim to have carried out their manipulations "within the guidelines laid down in the Stat. Sci. article" are completely without foundation. Their work consists of nothing other than flagrant and unjustified deviations from those guidelines. Therefore, their attempt to demonstrate that there is excessive latitude in the guidelines is an utter failure.

In our refutation we address in detail each and every one of their assertions, assertions which were intended to justify the many deletions and additions they made to the original list (which consisted of a total of 52 appellations appearing in Genesis in equally spaced letters). We highly recommend that the reader examine this document: He will be floored to discover the extent to which their claims are without foundation.

In brief:

1. They did not succeed in indicating even one appellation that was included by Prof. Havlin in violation of the rules! Thus there is no justification for the many deletions they made from the list, and all of them should be considered null and void.

2. They did succeed in indicating a total of 5 appellations (one doubtful) which Prof. Havlin omitted from the list: Out of these five appellations, two do not appear in the book of Genesis in equally spaced letters, and therefore their omission had absolutely no affect on the results.

It turns out that adding in the 3 remaining names to the list improves the results of the original experiment by a factor of 1.5!

3. All of their other additions constitute deviations from the rules. For example, 8 of the appellations are not pronounced, 3 of them deviate from the spelling conventions, etc. (see the Introduction to the response, Section 2).

Far more astounding is the fact that they added 8 additional names which are complete fabrications. For example, "HaRaavad Sheini", "Raavad Sheini", "Yitzchak Levi", etc. (ibid).

4. Furthermore, they found it necessary to omit rabbis from the list and substitute others in their place arbitrarily, without any justification at all: neither according to the original experiment, nor even according to the amended list which they themselves proposed a year ago [see Document 2].

Now you may be wondering: How is this possible? Didn’t the authors of the critique write that "the deletions and additions were made according to research" carried out by "Dror Bar-Natan and Brendan McKay, with the assistance of Prof. Menachem Cohen from the faculty of Judaic Studies at the Bar Ilan University"? The answer is that their work was not carried out by a professional bibliographer. For some reason they neglected to mention in their article the fact that Prof. Cohen is from the Biblical Studies department. He is a professional from a different field. Furthermore, you may have received the impression that Prof. Cohen himself prepared the list. But this was not the case at all - the list was prepared by Mr. Aryeh Levitan, who had no prior experience in the field of bibliography, and whose mother tongue is not even Hebrew. This is what was responsible for the many errors in their assertions.

In the limited space that has been allotted for this response, we have had to ignore many inaccuracies in the critique. But it is impossible not to treat (if only briefly) a crucial point raised by the authors in their critique. That is the question of the integrity of our research: Was the original research conducted honestly and sincerely? They make the following analysis: "Every researcher in every investigation is obliged to make various decisions of various sorts regarding his procedures." In the original investigation there were also choices of this kind to be made. If the choices had been made in an a priori fashion, that is to say, without knowing how they would affect the results, it would be reasonable to assume that the number of "beneficial" choices (those which improved the results) would be about the same as the number of choices which were not "beneficial" (i.e. those which had a deleterious affect on the results). They continue their analysis: "Yet, surprise surprise, it turns out that in almost every case, if not in every case, their supposedly blind choices paid off." Their conclusion: The chances that the original research was carried out honestly and sincerely are "weak in the extreme."

Yet a closer examination of their analysis reveals that it was carried out in a way which was defective in the extreme. When one examines their analysis of our choices one reaches exactly the opposite conclusion!

Any analysis, done in order to reveal whether there was any bias in the decision-making process, must adhere to the following conditions:

1. Obviously the one conducting such an analysis must guard himself against bias. Therefore he must make himself a list, a priori, of the "choices" he is going to investigate, before he investigates whether the choices were "beneficial" or not.

Then, in order for his conclusions to be persuasive, he must present all of the "choices" which he investigated. Otherwise, there is no value whatsoever in his declaration that "such-and-such choices were 'beneficial' and such-and-such choices were 'not beneficial'."

2. The evaluation must be made only for those choices which were not dictated by the nature of the research. For example, when Galileo decided to investigate whether Jupiter has moons using a telescope instead of a microscope, this was a decision which was dictated by the needs of the investigation. Similarly, it would be somewhat ludicrous to level a charge of bias against a researcher investigating which company produces the most popular steak for not having chosen to conduct his survey among a village of vegetarians.

The authors of the critique failed to fulfill either one of these conditions. A full analysis of their "analysis" is presented here in Document 4. Here let it suffice to say that on two previous occasions the authors of the critique presented lists of the "choices" they investigated. In the winter of '95 Prof. Bar-Hillel presented a list of 13 choices at a convention held by the Center for the Research of Rationality (of the Hebrew University). She claimed that every one of our 13 choices were "beneficial." But mathematician Prof. Robert J. Auman presented in response a totally different analysis: Of the 13 choices only one was "beneficial." The rest were either choices dictated by the needs of the research (for example, the "choice" to use correct dates rather than incorrect ones!), or choices which were "not beneficial." One example of a choice which was "not beneficial" was the choice not to use the form of the Hebrew date áà áúùøé(b’alef b’Tishrei). To our sorrow Yaakov Auerbach, is no longer with us to explain his reasons for making this choice, but one thing is clear: If we had used this form it would have improved the results both for the first list as well as for the second. Therefore it was "not beneficial" to have omitted it. How strange it is that this example was cited in the critique as an example of a "beneficial" choice!

Another example: The second list included Rabbis whose entries in Margalioth's Encyclopedia were from 1.5 to 3 columns. They claimed that we erred in measuring the column lengths, and in their opinion we should have omitted and added certain personalities. It turns out that had we operated according to their recommendations, the results would have improved! At the request of Prof. Auman we performed this calculation using the permutation test, as well, and the results improved from 4 in a million to one in 5 million!

On April 3, '97 Dr. Brendan McKay published on the Internet a report of an investigation of our research. In this report he lists 20 choices which he investigated (principally regarding the method of measurement). Even if we ignore the fact that some of the "choices" he mentions were completely imposed by the requirements of the research, it turns out that at least 11 choices out of 20 (by his own calculation) were "not beneficial" for us. One was particularly "not beneficial": If we had made a slightly differently decision the results for the first list would have improved by a factor of approximately 1,000, and the results for the second list would have improved by a factor of more than 1,000!

It seems rather strange that of all these investigations, which they themselves conducted, there is no mention in their critique! If one takes a look at our "document 4", he will discover other decisions pointed out by the authors of the critique in their criticisms of Prof. Havlin's guidelines and the application of them, which turn out to have been "not beneficial" for us.

To summarize:

The assertion of the authors of the critique that our success derived solely from the "level of flexibility" in Prof. Havlin's guidelines - has not been demonstrated. Yet we must thank psychologist Prof. Maya Bar-Hillel and her colleagues for the tremendous efforts they have invested in their attempts to invalidate our research, because there is a great deal we can learn precisely from their total failure. First of all, it serves as solid evidence against their assertion that within the established guidelines there is "enough choice to generate comparable significance levels in War and Peace."

Furthermore: A detailed analysis of the very "choices" which they point to shows that the original research was conducted without any bias and in an a priori manner.

Strangest of all was their decision to use parody as their method of attack. Parody is incapable of answering the question: Does there exist a hidden code in the book of Genesis or not. On the other hand, they chose not to use a more appropriate scientific methodology to test their claim - that is to allow an independent authority to prepare a new list of names and appellations for the 32 personalities on the second list, using Prof. Havlin's guidelines! - This is what we suggested a year ago, and to this day we would be very happy to discuss the necessary procedures.