Amar wrote a wonderful blog post last week, discussing the utility (or more correctly, the futility) of using patterns in historical financial data as the basis for taking financial decisions about the future.

But patterns are important. Almost every scientific discovery of any consequence from the dawn of our history has been made because some curious person, saw a pattern in his experiences, could not explain why the pattern existed and went searching for an explanation. Of course, this search wasn’t always straight forward, it often led to wrong turnings, dead ends, mystical and mythical beliefs. But if the pattern was not noticed or was ignored even when it was, human society wouldn’t be where it is today (of course, some might say, where we have reached isn’t necessarily the right place with the destruction, we have wrought on ourselves and our universe – but that is a different debate).

Why then do patterns also fail us, every so often as Amar has pointed out, so spectacularly?

The key reason why the likes of Varah Mihir or Galileo or Einstein succeed in making wonderful discoveries from patterns and despite all of your mutual fund manager’s “Back testing”, the fund cannot reproduce the promised results lies in what I said in bold above – they went searching for an explanation first before they proclaimed that the pattern meant something and can be used to predict what may happen in future with some degree of confidence.

But what does searching for an explanation entail?

The first step always is to determine whether there is a prima-facie reason to believe that the observed pattern is not observed by pure chance.

The problem of how to distinguish a genuine observation of some effect from something caused by random chance is a very old one. It’s been debated for centuries by philosophers and by statisticians. 70 odd years ago the statistician Fisher came up with a test of “Statistical Significance” which seemed to be the perfect tool to resolve this dilemma. Fisher created a technique to calculate the probability of making our observations by random chance. That technique isn’t saying that indeed the observation is pure chance. What it calculates is what can be expected if there were no real effects.

This probability estimate is called the p-value and a low p-value (generally less than 0.05) is assumed to mean that the observation is “very unlikely” to be pure chance.

However, even that is not sufficient reason to conclude that the observed pattern represents some real phenomenon. Because all that we would have learned so far is:

1. There is a close correlation between the values of two variables A and B
2. The calculation of p-values suggests that the probability of observing this correlation by pure chance is low.

We still have to discover whether A causes B and if it does, then exactly how to determine the value of B given the value of A.

Here is where things start to become interesting (and difficult). Because there is NO WAY that from the data alone, we will ever be able to determine unequivocally that A cause B. In order to determine if A causes B, we have to be able to answer 3 questions –

1. Correlation: Do A and B seem closely related from the historical data that is available? This is what we have answered so far.
2. Intervention: If I deliberately make A happen, will I always (or sufficiently frequently) see that B also happens.
3. Counter-factual: If A is absent, will I see B happen?

The questions of Intervention and Counter-factual CAN NOT be answered from historical data alone. Both of which need to be answered only by conducting experiments in the future and to even design such experiments, first we have to formulate a theory or hypothesis – i.e., we have to first construct an explanation of WHY we observe what we have already observed and WHY we will observe the same things in the future. Then we have to conduct carefully designed experiments and show that the results support this theory/hypothesis/explanation. And even when a large number of experiments are conducted and shown to support the theory/explanation by successfully demonstrating the answers to the questions of Intervention and Counter-factuals, the theory is only accepted as provisionally valid!

When therefore we are presented with a model constructed from historical patterns alone, the first question which we have to ask is – Do you have an explanation of WHY we have observed what we have observed. And the second question we have to ask is – Where are the answers to the questions of Intervention and Counter-factuals? Without all those, we do not have even a provisionally valid basis which can be used to predict future outcomes with any degree of confidence!