Can sentiment predict market direction?
Question: Is investor sentiment a useful indicator of market direction? If possible, I would like to trade on sentiment to make money.
Blogger Sentiment
Popular measures of investor sentment are reported in Barron's each week. For brief background reading, try the Investopedia article entitled "Investors Intelligence Sentiment Index." The article cites an academic study published in 2000 by Ken Fisher and Meir Statman that concluded: "We found the relationship between the sentiment of newsletter writers as measured by the Investors Intelligence survey and future S&P 500 returns to be negative but not statistically significant." As is the case with most (if not all?) fundamental and technical indicators, the prospect of using a simple sentiment index to trade and consistently realize excess profits does not look very encouraging.
However, instead of giving up so easily, let's have a look at a newer sentiment index that is being tracked by Ticker Sense, a financial blog of Birinyi Associates, run by former Salomon executive Laszlo Birinyi. Beginning in July of 2006, Ticker Sense has been reporting at the start of each trading week market sentiment figures resulting from a poll sent out to participating bloggers the prior Thursday. Bloggers state whether they are bullish, bearish or neutral on the S&P 500 for the upcoming 30 days. The chart below shows this sentiment data for the past 12 months.
We can develop a qualitative feel for how useful this type of sentiment data might be in a trading context by plotting the S&P 500 index alongside the weekly difference between the bull and bear sentiment percentages. The chart below shows that for the past couple of months blogger sentiment has correlated favorably with the directional movement of the S&P 500 index--the market moves down with negative bull-bear sentiment in early August, up with positive bull-bear sentiment from about mid-August through the market's recent highs in early October, and down on slightly negative sentiment last week. Hey, this is beginning to look promising. . . .
"Batting Average" Test
One barometer for gauging how helpful blogger sentiment can be in predicting market direction is to perform a "batting average" calculation. A baseball player's batting average is a number between zero and 1.000 (or zero and 1000, if we ignore the decimal point), indicating the ratio of hits to at-bats during a season. For example, during 2004 when Seattle Mariners all-star player Ichiro set a new all-time major league baseball record with 262 hits, his batting average in his 704 season at-bats was 262/704 = 0.372.
In an analogous fashion, we can define a sentiment "batting average" as being the time-average of the relevant (bull, bear or neutral) sentiment percentages corresponding to the actual up, down and flat market outcomes observed during all of the trials in a specified testing period:
Sentiment "Batting Average" = Sum(Xi)/N,
where
Xi = bullish (bearish, neutral) sentiment percentage at time i-1 if market return ends up in bullish (bearish, neutral) range at time i,
and i runs from 1 to N, inclusive, where N is the number of trials in the testing period.
(Note: Since the Ticker Sense sentiment poll is updated at a weekly frequency, for our analysis we pair each week's sentiment data with only the following week's market movement. In effect, sentiment predictions are "refreshed" each week, even though bloggers participating in the poll are asked for their opinion about the next 30 days.)
Here's a numerical example: If a particular week's sentiment poll gives bull-bear-neutral sentiment percentages of 50-30-20 and the market ends up falling into the bearish range the week after the poll is taken, this particular trial contributes an X-value of 0.300 to the average. Obviously, the highest batting average is attained when the actual market movement always matches the strongest prevailing sentiment (or highest percentage) among the three possible sentiment "states" (bull, bear and neutral).
Two extreme cases help to clarify the meaning of our sentiment "batting average": If all bloggers participating in the poll had complete clairvoyance, the sentiment percentages would always be 100-0-0 (bullish), 0-100-0 (bearish) or 0-0-100 (neutral), and each trial would always contribute an X-value of 1.000, resulting in a perfect time-average of 1.000. At the other extreme, if the bloggers were always completely split without any predominating opinion, the sentiment percentages would be 33-33-33 (rounded), and each trial would always contribute an X-value of 0.333, regardless of whether the market rises, falls or remains flat. Importantly, this batting average of 0.333 is also the expected outcome when collective opinion, whether skewed or split, is no better than a random guess at determining market direction.
In order to apply our batting-average methodology to the blogger sentiment data, we also need to define what we mean by bullish, bearish and neutral outcomes, and ideally these three "states" should be equally likely, so that no particular outcome is favored over the others. To take into account the possibility of "trending" regimes, rather than referencing static return ranges I use a 26-week moving window ending just prior to each weekly trial:
"Bullish": Above 67th percentile return of immediately prior 26 weeks
"Neutral": Between 33rd and 67th percentile return of immediately prior 26 weeks
"Bearish": Below 33rd percentile return of immediately prior 26 weeks
Though there is some variation from week to week, for the S&P 500 over the past year the 26-week moving averages of the 33rd and 67th percentile weekly returns have hovered around -0.3% and 1.2%, respectively, giving sentiment ranges approximately as follows:
"Bullish": Weekly return above 1.2%
"Neutral": Weekly return between -0.3% and 1.2%
"Bearish": Weekly return below -0.3%
The graph below shows the week-by-week contributions to the overall average. For example, for the week ending October 5, bullish sentiment (from the poll sent out Thursday of the prior week) was 50% and the market traded up, thereby contributing an X-value of 0.500. Note, however, that the overall average from the past one year is the smaller figure of 0.319, which is slightly worse than the 0.333 expected in the case of random guessing. In other words, the batting-average calculation suggests that blogger sentiment will probably not be very useful in predicting market direction.
Simulated Trading Test
Another way to visualize the sentiment data is to look at scatterplots of the bull-bear sentiment difference (at time i-1) versus S&P 500 returns for the following week (at time i). I divide the past year into two 26-week periods to enable us to run two simulated trading tests on the sentiment data.
For the first 26-week period, from October 2006 through April 2007, there is a somewhat negative correlation between sentiment and returns, suggesting that sentiment could be slightly contra-indicative of market direction.
However, over the next 26-week period, from April 2007 through October 2007, the correlation is essentially zero, indicating no apparent relationship between sentiment and subsequent returns.
We can trade on the sentiment data by using the weekly bull-bear sentiment differences to weight one-week trades on the S&P 500 index (or futures), going long when the difference is positive (net bullish) and going short when it is negative (net bearish). Trade size equals the absolute value of the bull-bear sentiment difference, so that we take larger trading positions when sentiment is more extremely bullish or bearish. The graph below shows the simulated outcome of such trading over our two 26-week periods. The initial 26-week trading period produces a loss, while the later 26-week trading period begins in negative territory but goes slightly positive from early August to early October when sentiment, as mentioned above, matches market direction.
Is There Hope?
Overall, with the batting-average test failing to show all-star-like performance above the 0.333 expected value of random guessing, and the trading test producing what appears to be little more than a random walk, it is difficult to place any confidence in blogger sentiment as a useful predictor of market direction.
But, perhaps we should not overlook how traders, investors and bloggers (myself included) all learn as we go, thereby opening up the possibility that the blogger sentiment index could be an example of a self-improving dynamic system engaged in an adaptive learning process. The most recent two months certainly show promise, and sentiment data from last Thursday's poll (bull-bear-neutral: 50-33-17), with bloggers turning bullish following last week's 4% sell-off, again matches the S&P 500's interim performance so far this week (up from Friday's 1501 close)--though, of course, it is too early to tell how the remainder of the week will turn out.
What, then, should we believe: the negative overall read from the past year, or the more promising performance of sentiment over the past few months? Rather than allowing our hopes to rise too high, I suggest keeping in mind a market truism: if there really is any predictive power in a well-publicized indicator, opportunistic traders will soon exploit this information, thereby squelching any excess returns that might have been available. Maybe, then, the trick is to be quick--exploit the winning "streak" before it vanishes! Good luck trading!
Blogger Sentiment
Popular measures of investor sentment are reported in Barron's each week. For brief background reading, try the Investopedia article entitled "Investors Intelligence Sentiment Index." The article cites an academic study published in 2000 by Ken Fisher and Meir Statman that concluded: "We found the relationship between the sentiment of newsletter writers as measured by the Investors Intelligence survey and future S&P 500 returns to be negative but not statistically significant." As is the case with most (if not all?) fundamental and technical indicators, the prospect of using a simple sentiment index to trade and consistently realize excess profits does not look very encouraging.
However, instead of giving up so easily, let's have a look at a newer sentiment index that is being tracked by Ticker Sense, a financial blog of Birinyi Associates, run by former Salomon executive Laszlo Birinyi. Beginning in July of 2006, Ticker Sense has been reporting at the start of each trading week market sentiment figures resulting from a poll sent out to participating bloggers the prior Thursday. Bloggers state whether they are bullish, bearish or neutral on the S&P 500 for the upcoming 30 days. The chart below shows this sentiment data for the past 12 months.
We can develop a qualitative feel for how useful this type of sentiment data might be in a trading context by plotting the S&P 500 index alongside the weekly difference between the bull and bear sentiment percentages. The chart below shows that for the past couple of months blogger sentiment has correlated favorably with the directional movement of the S&P 500 index--the market moves down with negative bull-bear sentiment in early August, up with positive bull-bear sentiment from about mid-August through the market's recent highs in early October, and down on slightly negative sentiment last week. Hey, this is beginning to look promising. . . .
"Batting Average" Test
One barometer for gauging how helpful blogger sentiment can be in predicting market direction is to perform a "batting average" calculation. A baseball player's batting average is a number between zero and 1.000 (or zero and 1000, if we ignore the decimal point), indicating the ratio of hits to at-bats during a season. For example, during 2004 when Seattle Mariners all-star player Ichiro set a new all-time major league baseball record with 262 hits, his batting average in his 704 season at-bats was 262/704 = 0.372.
In an analogous fashion, we can define a sentiment "batting average" as being the time-average of the relevant (bull, bear or neutral) sentiment percentages corresponding to the actual up, down and flat market outcomes observed during all of the trials in a specified testing period:
Sentiment "Batting Average" = Sum(Xi)/N,
where
Xi = bullish (bearish, neutral) sentiment percentage at time i-1 if market return ends up in bullish (bearish, neutral) range at time i,
and i runs from 1 to N, inclusive, where N is the number of trials in the testing period.
(Note: Since the Ticker Sense sentiment poll is updated at a weekly frequency, for our analysis we pair each week's sentiment data with only the following week's market movement. In effect, sentiment predictions are "refreshed" each week, even though bloggers participating in the poll are asked for their opinion about the next 30 days.)
Here's a numerical example: If a particular week's sentiment poll gives bull-bear-neutral sentiment percentages of 50-30-20 and the market ends up falling into the bearish range the week after the poll is taken, this particular trial contributes an X-value of 0.300 to the average. Obviously, the highest batting average is attained when the actual market movement always matches the strongest prevailing sentiment (or highest percentage) among the three possible sentiment "states" (bull, bear and neutral).
Two extreme cases help to clarify the meaning of our sentiment "batting average": If all bloggers participating in the poll had complete clairvoyance, the sentiment percentages would always be 100-0-0 (bullish), 0-100-0 (bearish) or 0-0-100 (neutral), and each trial would always contribute an X-value of 1.000, resulting in a perfect time-average of 1.000. At the other extreme, if the bloggers were always completely split without any predominating opinion, the sentiment percentages would be 33-33-33 (rounded), and each trial would always contribute an X-value of 0.333, regardless of whether the market rises, falls or remains flat. Importantly, this batting average of 0.333 is also the expected outcome when collective opinion, whether skewed or split, is no better than a random guess at determining market direction.
In order to apply our batting-average methodology to the blogger sentiment data, we also need to define what we mean by bullish, bearish and neutral outcomes, and ideally these three "states" should be equally likely, so that no particular outcome is favored over the others. To take into account the possibility of "trending" regimes, rather than referencing static return ranges I use a 26-week moving window ending just prior to each weekly trial:
Though there is some variation from week to week, for the S&P 500 over the past year the 26-week moving averages of the 33rd and 67th percentile weekly returns have hovered around -0.3% and 1.2%, respectively, giving sentiment ranges approximately as follows:
The graph below shows the week-by-week contributions to the overall average. For example, for the week ending October 5, bullish sentiment (from the poll sent out Thursday of the prior week) was 50% and the market traded up, thereby contributing an X-value of 0.500. Note, however, that the overall average from the past one year is the smaller figure of 0.319, which is slightly worse than the 0.333 expected in the case of random guessing. In other words, the batting-average calculation suggests that blogger sentiment will probably not be very useful in predicting market direction.
Simulated Trading Test
Another way to visualize the sentiment data is to look at scatterplots of the bull-bear sentiment difference (at time i-1) versus S&P 500 returns for the following week (at time i). I divide the past year into two 26-week periods to enable us to run two simulated trading tests on the sentiment data.
For the first 26-week period, from October 2006 through April 2007, there is a somewhat negative correlation between sentiment and returns, suggesting that sentiment could be slightly contra-indicative of market direction.
However, over the next 26-week period, from April 2007 through October 2007, the correlation is essentially zero, indicating no apparent relationship between sentiment and subsequent returns.
We can trade on the sentiment data by using the weekly bull-bear sentiment differences to weight one-week trades on the S&P 500 index (or futures), going long when the difference is positive (net bullish) and going short when it is negative (net bearish). Trade size equals the absolute value of the bull-bear sentiment difference, so that we take larger trading positions when sentiment is more extremely bullish or bearish. The graph below shows the simulated outcome of such trading over our two 26-week periods. The initial 26-week trading period produces a loss, while the later 26-week trading period begins in negative territory but goes slightly positive from early August to early October when sentiment, as mentioned above, matches market direction.
Is There Hope?
Overall, with the batting-average test failing to show all-star-like performance above the 0.333 expected value of random guessing, and the trading test producing what appears to be little more than a random walk, it is difficult to place any confidence in blogger sentiment as a useful predictor of market direction.
But, perhaps we should not overlook how traders, investors and bloggers (myself included) all learn as we go, thereby opening up the possibility that the blogger sentiment index could be an example of a self-improving dynamic system engaged in an adaptive learning process. The most recent two months certainly show promise, and sentiment data from last Thursday's poll (bull-bear-neutral: 50-33-17), with bloggers turning bullish following last week's 4% sell-off, again matches the S&P 500's interim performance so far this week (up from Friday's 1501 close)--though, of course, it is too early to tell how the remainder of the week will turn out.
What, then, should we believe: the negative overall read from the past year, or the more promising performance of sentiment over the past few months? Rather than allowing our hopes to rise too high, I suggest keeping in mind a market truism: if there really is any predictive power in a well-publicized indicator, opportunistic traders will soon exploit this information, thereby squelching any excess returns that might have been available. Maybe, then, the trick is to be quick--exploit the winning "streak" before it vanishes! Good luck trading!
posted by Lloyd Sakazaki at 10:04 AM
6 Comments:
Comments to Llyod’s post.
One ought to understand whether he is investing or speculating. Benjamin Graham defines, in his famous book Security Analysis, “An investment operation is one which upon, through analysis, promises safety of principal and a satisfactory return. Operations not meeting these requirements are speculative.”
If one is speculating, he/she needs to understand the impact of their decision – understand his/her pay-off. I recently read a book Fooled by Randomness written by Nassim Nicholas Taleb. I find something interesting in his chapter of skewness and asymmetry.
Event Probability Outcome Expectation ( Pay-Off)
A. Market goes up 70% Up 1% 0.70
B. Market goes down 30% Down 10% -3.00
Total -2.30
Assuming you are willing to put $10,000 to bet market up next week. If you win, your reward is $ 100, though probability going down is only 30%, if you lose, your losses cost you $ 1,000, not to mention trading charges. Even though event A is far more probable than event B but it is not a good idea to so because the payoff is skewed towards big absolute losses. A person will suffer far more worse if he is borrows to speculate. He needs to cover his transaction costs, interest on borrowing and losses. I am always puzzle why people willing to risk their capital for a small gain but expose themselves to big losses.
This is where rule # 1 never lose money and margin of safety makes sense and has guided many long-term successful investors.
Timothy Vick in his book, How to pick stocks like Warren Buffett, says holding period determines likelihood of gains. In his study, if one randomly buys S&P stocks, longer holding periods will increase likelihood of making money tremendously. The shorter the holding period, returns are much more random. In his study period from 1989 to 1998, the average probability of gains between for 1 month, 3 month, 1 year and 5 year is 48%, 62%, 71.38% and more than 80% respectively.
One needs not go into complicated models, formulas, charts, etc to be a successful investor. One just needs to buy stocks below its intrinsic value. Intrinsic value is hard to calculate but crucial measure of business value, it is discounted present value of the cash that can be taken of a business during its remaining life.
Charlie Munger was jokingly said he never see Warren Buffett performs any discounted cash flow calculation. Buffett replied “Yeah right”, you actually don’t have to because the stocks are so cheap that they are screaming at you. I guess it is the same here; you don’t have to spend many of your waking hours guessing whether other people are bullish or bearish. Simplicity, simplicity, simplicity.
10 November 2007
My answer would be Yes and No. Market sentiment will affect its direction. For example now, sentiments are bad, people are bearish and not willing to hold positions overnight. That's why Wall Street has drop ten over percent since 2008 due to subprime worries. On the other hand, there are value investors who want to pick up the stock at bargain prices when stocks hit a new low. This will in turn push up the market. Well, it all depends which buy/sell positions are stronger.
Great story as for me. It would be great to read a bit more concerning this theme. Thanx for giving that info.
Sexy Lady
Escort London
very nice post sir.i just love it.thanks for sharing this with us.
for two days free trail pack please contact on 0731-4295950
Fantastic Blog sir.thanks for sharing with us.
for two days free trial pack in equity and commodity please contact us on 0731-4295950
Hello everyone, Are you looking for a professional trader, cryptocurrency/forex and binary manager who will help you trade and manager your account with good and massive amount of profit in return. you can contact Mr. Bernie for your investment plan, for he helped me earned $8,600 usd with little investment of $850 usd. Mr Bernie Doran you're the best trader I can recommend for anyone who wants to invest and trade with a genuine trader, he also helps in recovery of loss funds..you can contact him on his whatsapp: +1(424)285-0682 Email:(berniedoransignals@gmail.com)I advice you shouldn't hesitate He's great.
Post a Comment
<< Home