BS"D
The Failure of the University Committee
Established to Examine Gans – Inbal Experiment
By Doron Witztum
Part I: The Experiments
A. An experiment performed by Harold Gans to examine ELS rabbi/community matches in Genesis, concluded with statistical results that were highly significant. Data collection for the experiment was accomplished through a mechanical procedure by means of a precise algorithm (the Inbal Algorithm).
For the scientific publication of Gans, Inbal and Bombach's (2006) paper, click here.
For general background to this work, see here (pages 16-18).
For a detailed Hebrew description of both the algorithm and data collection, see here.
B. Following criticism of the algorithm and its implementation, a university committee headed by Professor Yisrael (Robert J.) Aumann was established in 1996, with Professor Hillel Furstenberg and Professor Isaak Lapides as participants. The committee also included a representative of the experiment’s proponents, Professor Eliyahu Rips and a representative of its opponents, Professor Dror Bar-Natan. The last two did not participate in the final stages of the committee's discussions.
The objective of the committee was “to look into the results reported by Gans in [G].” The committee began examining the algorithm and how it was implemented, but after several meetings the original goal was abandoned and the committee moved onto something different: To design and perform a new experiment that would examine the ELS matches of rabbi/community names in Genesis using new data. In fact, two separate experiments were designed.
For this purpose, a protocol was prepared, establishing in advance all necessary steps to prepare data and calculate its results. The protocol was written, and signed at the end of the committee's discussions on August 6, 1998 by Aumann, Furstenberg and Lapides.
To read the committee's discussions - contact the Center for the Study of Rationality at the Hebrew University, Feldman Building at Givat Ram, Jerusalem. The committee undertook to make the material (documented by professionals) available for public scrutiny.
C. After lists of data were prepared, a committee representative gave them to Professor Eliyahu Rips to do the calculations. At first glance, it was clear there were mistakes. [For example, on the first page of one list of data Venice was misspelled (as "וינוציא" instead of "ויניציא") and on the first page of the second list of data the name of the Spanish town טולידו (Toledo) was confused with the name of another Spanish town טודילא (Tudela)!].
We immediately drew the attention of the committee to the fact that there were trivial errors in the data, and made it clear that we would not take part in any experiment or calculation until the collected data was first verified as being appropriate for a scientific experiment.
An exhaustive examination made at that time found not only trivial errors, but discovered that the experiments were fatally flawed because of significant deviations from the protocol and because of serious errors and carelessness in preparation of the data. We concluded that any experiment using this data would be scientifically worthless.
D. When we informed the committee chairman of our decision, he insisted on proceeding with the experiments regardless. Even though he agreed that the data included serious flaws, he argued that he nevertheless felt obliged to complete the project as specified in the protocol that had been prepared in advance. When we retorted that an experiment based on such severely flawed data would be worthless, he admitted that it might indeed be necessary to start again from scratch. However, he insisted that he would publicize a committee report on the experiments even if we refused to participate, relying on Professor Bar-Natan’s calculations. The chairman recommended that we agree to conduct the experiment and hand him our calculations, and said that if we did so we would be given an opportunity to explain the experiments’ flaws.
Only by agreeing to his suggestion were we enabled to apprise the public (in the Committee report) of the simple fact that the experiments lacked scientific validity. Besides our firm position that it was inappropriate to conduct a flawed experiment and publicize it as scientifically valid, we are puzzled at the using of “conforming to protocol” as a chief rationale to continue with the experiment and publicize its results. In fact, the protocol was not conformed to in the course of the experiment and there were actually significant deviations from it.
E. After this agreement we made the necessary calculations for the two lists of data. In both cases the results were insignificant.
The members of the Committee disagreed whether the experiment based on flawed data had any significance: Professors Aumann and Furstenberg signed the results, while Professor Lapides refused to.
The Committee's Report was published by the Center for the Study of Rationality in Discussion Paper # 364. (It can be read in file DP364.pdf). Professor Lapides' minority opinion was also published by the Center for the Study of Rationality in Discussion Paper # 365. (It can be read in file DP365.pdf). It includes Professors Aumann and Furstenberg's point of view. Our criticism was also published by the Center for the Study of Rationality in Discussion Paper # 365 (it can be read in the original file of DP365.pdf).
In our opinion:
• The results of the committee’s experiments reflect nothing but the failure of their data preparation. In particular, these results provide no implication concerning the existence or non-existence of Torah Codes.
• As we have always stressed, according to our thesis only experiments based on accurate data have a good chance of detecting codes successfully, while those based on erroneous data are likely to fail.
Acknowledgment: We thank the members of the Committee, Professor Yisrael (Robert J.) Aumann, Professor Hillel Furstenberg, and Professor Isaak Lapides, for the considerable effort and time they invested over the years they spent in the committee. We are confident that they honestly attempted to clarify the validity of the allegations against Gans' work. Sadly, due to the circumstances that arose, their considerable investment led to no scientific conclusions.
Part II: The Significance of the Experiments' Results
We will now clarify the meaning of the committee's work and how the opponents of Torah Codes research presented it. This section will discuss three issues,
A. The significance of the committee's experiments.
B. The significance of the experimental results.
C. The propaganda of opponents to Torah Codes research.
A. What is the significance of the committee's experiments?
Gans' original experiment (denoted as [G]) had two components:
1. The thesis – Including an exact definition of the investigation’s subject.
2. Data collection.
As mentioned in part I, the committee originally decided to examine the technical component 2. But instead of doing this it chose to conduct experiments of its own.
- One experiment, [F], was dissimilar to component 1 (the thesis) and therefore it had no relevance to [G].
- The second experiment, [R], was similar to component 1 (the thesis) and therefore it could be considered as a replication of [G]. The new aspect of [R] was component 2 (data collecting).
Part E of the Committee report explained the logic behind experiment [R]:
"... If there is indeed a phenomenon of codes, then one would expect it to follow certain general rules or practices. Some idea of these rules were gained from the WRR experiment on dates, and from other previous experience (on matters unrelated to the localities in question), and it would be legitimate to incorporate these ideas in testing the localities. This is the logic behind the Replicative test."
What can we learn from the results of experiment [R]?
Experiment [R] had three possible outcomes:
(a) If the data turned out to be very similar to [G] – it would indicate that there was no wiggle room in [G]’s data collection.
(b) If a significant part of the data was different and the experiment still succeeded – it would indicate that although there was wiggle room in [G]’s data collection, this was not the reason for the experiment's success. (In other words, success was not dependent on specific variants or spellings of community names).
(c) If a significant part of the data was different and the experiment failed – it would indicate that there was wiggle room in the data collection and that [G]’s success depended on specific data.
Part E of the Committee report said, before making the calculations, what it would mean in the committee's opinion if there was a great disparity between the results of [R] and [G]:
"The result G of Gans' experiment was adduced in Part D above for comparison with the result R of the replicative test. A great disparity between R and G may raise eyebrows. But it should be noted that if indeed there is a code phenomenon, then G might a priori be considerably smaller, or larger, than R. For example, this could happen if the expert had used spelling conventions that are systematically different from those used by the putative encoder. "
Here they mention the obvious fact that since the claimed phenomenon is encrypted words and phrases, language and spelling are critically important. If there is indeed an encryption, it is logical that it exists according to certain language conventions and spelling rules. Data that does not obey these language conventions and spelling rules would obviously fail.
According to this analysis, even result (c) of the experiment (or result (b) with less success than [G]), would teach nothing about the rigor of Gans' original experiment and certainly nothing about the encryption or non encryption of other matters!
B. What is the significance of the experimental results.
As we know, the outcome of [R] was (c) [see (c) in the previous paragraph].
If the committee members concluded before the experiment that they were incapable of reaching any conclusions regarding the rigor of Gans' original experiment as they wrote themselves (see end of previous paragraph) – even if the data was correct – the situation now became far worse as the data collection for the experiment was greatly flawed. Much of it (dozens of cases) was missing or erroneous.
Because of this, the committee had no way of knowing whether the difference in the prepared data came about because of wiggle room or because of the mistakes and omissions.
Therefore, the committee never succeeded in clarifying the technical question it had set out to investigate – whether wiggle room exists even if data is collected scientifically and rigorously!
C. The propaganda of opponents of the Torah Codes research
The opponents of the Torah Codes research deliberately distort the significance of the committee's conclusions. Professor Dror Bar-Nathan, the committee representative of the Torah Codes opponents wrote as follows [We took this from his site on 29 Nissan 5770 (April 13, 2010). The site says that the material was first posted on September 4, 2002 and updated on May 14, 2003. He hurriedly posted it before the committee's report was published in June 2004]:
Anyway, here are the results of the analysis:
The Fresh Experiment |
The Replicative Experiment |
463,406 /1,000,000 |
578,075 /1,000,000 |
Some of the readers of this page may not be sufficiently immersed in this subject to understand what these numbers mean. Ok, here's the meaning, plain and short:
A Complete Failure of the "Codes"
|
In noisy red he announces the "complete failure of the 'codes.'" In his view, this is the meaning of the committee's experiment.
One may well ask:
- Is Professor Dror Bar-Natan unaware of the simple arguments outlined in the previous paragraphs?
- Moreover, is Professor Dror Bar-Natan unaware that there were fatal deficiencies in the data collection and that results obtained from incorrect data are worthless?
Let's look at Professor Bar-Natan's viewpoint before the calculations were made.
The minutes and report of the committee reveal that:
1. Professor Bar-Natan strongly opposed determining a threshold for the experiments' success. He was unwilling to sign that the experiment was successful even if its significance was higher than one in a million.
2. Professor Bar-Natan was only interested in lists of data themselves for everyone to analyze according to his understanding.
3. Professor Bar-Natan demanded that data lists be posted in public in order to be subjected to strict scrutiny:
"... My only remaining interest in the "Gans Committee" is that the data it had collected will be made available to the public for scrutiny."
(From the committee's report, Appendix 5)
4. Of his own initiative, Professor Bar-Natan stated his opinion that there is no relevance to a negative outcome resulting from incorrect data.
Viewpoints 1, 2 and 4 were expressed at meetings of the committee. Here is a quote (translated from Hebrew) from the minutes of the committee for examining the work of Gans from June 29, 1998:
"Yisrael Aumann: So I understand you correctly that the threshold is one out of ten thousand?
Dror Bar-Natan: This is not the threshold for anything. And if there was a result of one in a million, I would be even more impressed… what can I say? But again, it is absolutely clear regarding the other direction, say the result was completely negative, and this was because our expert with his word processor constantly wrote zayin everywhere he needed to write daleth, as these letters are quite similar.
I do not know why I brought this up. No firm conclusions can result from this, there is no use trying to take [them] out from here. So let us be satisfied with what we can take out, what we have a chance of taking out is lists that came out under controlled conditions and after that one can analyze, each person according to his understanding..."
Regarding viewpoint 1, it should be noted that logically speaking if strong significance is insufficient to indicate encryption, all the more so, a lack of significance would not imply a lack of encryption!
Therefore, according to the position of Professor Bar-Natan before the experiment, the lack of statistical significance that was obtained does not indicate "a complete failure of the 'codes'" – even if the data were correct!
Regarding viewpoint 2, it should be noted that we did indeed analyze the data. Readers are directed to our published CRITIQUE OF THE REPORT SUBMITTED BY THE COMMITTEE ESTABLISHED TO EXAMINE GANS’ EXPERIMENT, in particular section F (click here).
But Professor Bar-Natan analyzed nothing and was satisfied with the mere numerical results - contrary to his stated position before the experiment.
Regarding viewpoint 3, it should be noted that we put the data collection process under strict scrutiny. Readers are again directed to our published CRITIQUE (click here).
But Professor Bar-Natan criticized nothing. He does not even inform the reader that there was criticism of the data, even though it was present before the results were calculated. He provides a link to Discussion Paper # 364, but does not direct the reader to criticism published in Discussion Paper # 365 of the Center for the Study of Rationality, and does not even mention the existence of Discussion Paper # 365. Thus he prevents his readers from knowing that the committee members acknowledged the existence of errors in the data, and that one of the three committee members refused to sign the experiments' results, considering them of no scientific worth.
Regarding viewpoint 4, it should be noted that we brought firm evidence that there were many errors and omissions in the data and that fatal deficiencies resulted in the experiments. Readers are directed to our published CRITIQUE (click here).
But Professor Bar-Natan ignored all this - contrary to his stated position before the experiment.
Dishonest Trick
Our criticism of the commission's work published in Discussion Paper # 365 with the agreement of the committee chairman (see above, Part I, Section D), contained an abstract and a large appendix of 157 pages detailing all the errors, omissions and flaws. We took the trouble to take dozens of photos from the sources on which the expert of experiment [R] relied, and time and again it turned out that the source he relied on was different from (or the opposite of) what he said. Anyone not checking this out for himself would find this hard to believe. Therefore the photographed documentation is of utmost importance.
Our criticism of the committee's work published in Discussion Paper # 365 was put on the Center for the Study of Rationality's site on July 2004 as DP365.pdf. In 2005, Professor Maya Bar-Hillel, the close colleague of Professor Bar-Natan in the war against Torah Codes research, was appointed as head of the Center for the Study of Rationality. She vetoed the inclusion of our comprehensive appendix in DP365.pdf. Professor Israel Aumann, chairman of the committee protested this and when she brought the matter to a vote before the executive committee of the Center for the Study of Rationality, she was supported by all its members except Professor Aumann.
Thus, the appendix of file DP365.pdf was deleted without our knowledge. Moreover, changes were made to our abstract without notifying us, changes that disrupt its comprehension.
By chance, we heard of the deletion months afterwards when Professor Robert Haralick entered their site and was astonished to find that the file had undergone a crash diet and shrunk from 178 to a mere 21 pages. Further clarification raised the sad findings mentioned above.
Doubtless, this astounding trick of Professor Bar-Hillel
prevents the public from knowing
that Bar-Natan's boast
A Complete Failure of the "Codes"
IS FALSE!
Back to top
