Predictability of Bull and Bear Markets: A New Look at Forecasting Stock Market Regimes (and Returns) in the Us

The empirical literature of stock market predictability mainly suffers from model uncertainty and parameter instability. To meet this challenge, we propose a novel approach that combines the documented merits of diffusion indices, regime-switching models, and forecast combination to predict the dynamics in the S&P 500. First, we aggregate the weekly information of 115 popular macroeconomic and financial variables through an interaction of principal component analysis and shrinkage methods. Second, we estimate one-step Markov-switching models with time-varying transition probabilities using the diffusion indices as predictors. Third, we pool the forecasts in clusters to hedge against model risk and to evaluate the usefulness of different specifications. Our results show that we can adequately predict regime dynamics. Our forecasts provide a statistical improvement over several benchmarks and generate economic value by boosting returns, improving the certainty equivalent return, and reducing tail risk. Using the same approach for return forecasts, however, does not lead to a consistent outperformance of the historical average.