How Guessing Market Direction Can Be PREDICTABLY Bad for Your Financial Health

"The game is called probability guessing. . . . [S]ubjects are shown a series of cards or lights which can have two colors, say green and red . . . appear[ing] . . . with different probabilities but otherwise without a pattern. . . . The task of the subject, after watching for a while, is to predict whether each new member of the sequence will be red or green. . . . Humans usually try to guess the pattern, and in the process we allow ourselves to be outperformed by a rat. . . ." (excerpt from Leonard Mlodinow's The Drunkard's Walk--How Randomness Rules Our Lives, 2008)

In a stock market context, the probability guessing game described above would read like this: In any given year, the stock market either rises (green) or falls (red). If we look over the past six decades, from 1950 through 2009, we find that during the sequential decades (1950s, 1960s, and so on) the S&P 500 Index rose in 8, 6, 7, 9, 8 and 6 out of the 10 years (using data from Yahoo! Finance). In other words, during a "typical" decade annual stock market returns are "green" about 7 or 8 out of the 10 years, and "red" about 2 or 3 out of the 10 years. Based on these historical data, we can infer that the stock market tends to rise during any particular calendar with a probability of about 75%, and fall with a probability of about 25%. As investors, we, of course, would like to try to predict whether this year (or next year, or any future year for that matter) will be green or red.

For us investors, the million-dollar question is: Should an investor attempt to "time" the market, by investing in stocks during years the market is more likely (in the investor's opinion) to rise and staying out of the market during other years when the market is more likely (again, in the investor's opinion) to fall?

Obviously, if the investor truly has enough information, foresight or precognition to know with a high degree of certainty when the market will rise or fall, then market-timing makes perfect sense and will lead to higher returns. However, what happens if the investor only believes that he knows but actually does not, so that for all practical purposes the investor is really faced with the 75% green versus 25% red probabilities described above? Is any harm done by guessing?

Analogous to the general guessing game Mlodinow mentions in his book, let's consider two strategies:

1. Buy-and-Hold Strategy: Since the market rises during 75% of the years, one could just go long the market by buying an exchange-traded fund tracking the S&P 500 Index (or buying individual stocks), without attempting to time the market at all. A buy-and-hold investor can expect to generate positive returns 75% of the years but must also accept the unavoidable "fact" that the market will typically fall 25% of the time. In this "simpleton" strategy, an investor's long-run win percentage (i.e., the percentage of years the investor's portfolio will show positive returns) is expected to be 75%;

2. Market-Timing Strategy: A presumably more "sophisticated" investor will, through some combination of fundamental and technical analysis and application of his general intelligence and market wisdom, come up with a convincing explanation for why the market is more likely to rise (or fall) during any particular year. Believing he can distinguish beforehand (i.e., predict) which years are among the 75% "green" years when the market will rise and, likewise, which years are among the 25% "red" years when the market will fall, such an investor will want to go long 75% of the time and stay out of (or go short) the market 25% of the time.

If the bright and sophisticated market-timing investor has an "edge" over the the naive and unthinking buy-and-hold simpletons, then he will end up being right more than 75% of the time and will show higher long-run returns. At the other extreme, if it turns out that the market-timer only believes he has an edge but actually does not, one would think that his edge would just vanish and there should be no penalty for guessing, right?

Well, you might think that guessing carries no penalty, but that's actually wrong! Quite counter-intuitively, investors should expect lower returns when they guess. Here's why.

Let p be the (stationary) probability that the the market will rise in a given year, i.e., p = 0.75, representing the 75% "green" probability. Supposing that the market-timer's guesses do not give him any significant edge, his overall win percentage is given by a straightforward weighted-probability calculation:

Market-Timer's Win Percentage
= (Portion of time the market-timer goes long) x (Probability that market rises)
+ (Portion of time the market-timer stays out of market) x (Probabiility that market falls)
= p x p + (1 - p) x (1 - p)
= p² + (1 - 2p + p²)
= 2p² - 2p + 1.

On the other hand, the Buy-and-Hold Investor's Win Percentage is just p, as we saw earlier. Consequently, we may write that the expected potential downside of the market-timing strategy versus the buy-and-hold strategy is the difference:

(Buy-and-Hold Investor's Win Percentage) - (Market-Timer's Win Percentage)
= p - (2p² - 2p + 1)
= -2p² + 3p - 1
= 2(p - 0.5)(1 - p),

where the last expression is the factored-form equivalent of the quadratic polynomial in the previous line.

From the factored-form expression, we can easily see that whenever p is in the "physical" range (i.e., consistent with the probabilities indicated by market history for a wide variety of investment time windows) from 0.5 to 1.0, a buy-and-hold investor is expected to outperform any market-timer who is really just guessing without appealing to any special knowledge of market direction. In particular, when p = 0.75 (which is the historical win-percentage for a sequence of annual returns), the Market-Timer's Win Percentage becomes 2(0.75)² - 2(0.75) + 1 = 0.625, or 62.5%, which is 12.5 percentage points worse than the Buy-and-Hold Win Percentage of 75%.

Therefore, to the extent that a market-timer is "only guessing" (and who can really be so certain?) about market direction, he is (presumably unknowingly) effectively "shooting himself in the foot," following a self-destructive path of degrading his expected returns by staying out of the market 25% of the time (by the way, shorting the market 25% of the time would make matters even worse). Despite his seemingly sophisticated ways, this market-timer can actually be expected to underperform the simpleton buy-and-hold investor in the long-run.

Lesson: Don't attempt to "time the market" unless you are absolutely certain that your market-timing strategy actually works, since your expected downside from "believing without knowing" far exceeds your time spent strategizing, not to mention your trading costs and commissions consumed.