Academic and practitioners’ literature is a source for many investment and trading ideas. However, it is often criticized for producing ideas that do not work in practice. A problem we have seen repeatedly is that the published papers, more often than not, only present in-sample results and disregard out-of-sample ones. This is an important distinction that is often overlooked.
In-sample results are generated by using the same data to estimate a model and then using the estimated model to make predictions. This approach can produce seemingly good results, but the model may not perform well when applied to new data. Out-of-sample results are generated by using a part of the data to estimate the model and then using the estimated model to make predictions for the remaining data. This approach provides a more accurate assessment of the model’s predictive power.
Reference [1] presented an overview of the literature and it confirmed the problem of in-sample overfitting in asset management research. It also provided insights into the quality of research conducted by practitioners,
I believe p-hacking is less of a problem in asset management than in academia—in particular, less of a problem in the proprietary research that is the foundation for a product. The reasons are simple. First, in the presence of a performance fee, the asset management company’s research needs to be optimized in a way that maximizes the chances of repeatable performance (see Harvey and Liu (2018)). This means the asset manager does not choose the best-performing backtest, because likely it resulted from overfitting and luck. If the manager were to launch a backtest-overfitted strategy, it would likely fail and generate no performance fees. The second reason is reputation. Academic tenure has no equivalent in asset management. If an asset manager’s products disappoint because of overfitting, the firm’s investors will flee. This market mechanism naturally minimizes the overfitting. That said, asset management companies still produce a substantial amount of low-quality research. Similar to the academic research, investors need to be skeptical.
In short, in-sample overfitting is less of a problem in proprietary research than in academic research. Regardless, investors should be skeptical of both types of research results.
Let us know what you think in the comments below.
References
[1] Harvey, Campbell R., The Pitfalls of Asset Management Research (2022). https://ssrn.com/abstract=4078138
Further questions
What's your question? Ask it in the discussion forum
Have an answer to the questions below? Post it here or in the forum
