Academic and practitioners’ literature is a source for many investment and trading ideas. However, it is often criticized for producing ideas that do not work in practice. A problem we have seen repeatedly is that the published papers, more often than not, only present in-sample results and disregard out-of-sample ones. This is an important distinction that is often overlooked.
In-sample results are generated by using the same data to estimate a model and then using the estimated model to make predictions. This approach can produce seemingly good results, but the model may not perform well when applied to new data. Out-of-sample results are generated by using a part of the data to estimate the model and then using the estimated model to make predictions for the remaining data. This approach provides a more accurate assessment of the model’s predictive power.
Reference  presented an overview of the literature and it confirmed the problem of in-sample overfitting in asset management research. It also provided insights into the quality of research conducted by practitioners,
I believe p-hacking is less of a problem in asset management than in academia—in particular, less of a problem in the proprietary research that is the foundation for a product. The reasons are simple. First, in the presence of a performance fee, the asset management company’s research needs to be optimized in a way that maximizes the chances of repeatable performance (see Harvey and Liu (2018)). This means the asset manager does not choose the best-performing backtest, because likely it resulted from overfitting and luck. If the manager were to launch a backtest-overfitted strategy, it would likely fail and generate no performance fees. The second reason is reputation. Academic tenure has no equivalent in asset management. If an asset manager’s products disappoint because of overfitting, the firm’s investors will flee. This market mechanism naturally minimizes the overfitting. That said, asset management companies still produce a substantial amount of low-quality research. Similar to the academic research, investors need to be skeptical.
In short, in-sample overfitting is less of a problem in proprietary research than in academic research. Regardless, investors should be skeptical of both types of research results.
Let us know what you think in the comments below.
 Harvey, Campbell R., The Pitfalls of Asset Management Research (2022). https://ssrn.com/abstract=4078138
What's your question? Ask it in the discussion forum
Have an answer to the questions below? Post it here or in the forum
Gold prices were flat during early Asian hours on Wednesday, as traders positioned themselves for inflation data that could impact the U.S. Federal Reserve’s policy stance. FUNDAMENTALS * Spot gold was unchanged at $2,035.39 per ounce, as of 0031 GMT. U.S. gold futures were steady…
CALGARY, Alberta, May 09, 2023 (GLOBE NEWSWIRE) — Calfrac Well Services Ltd. (“Calfrac”) (TSX–CFW) is pleased to announce the voting results of the election of directors at its annual and special meeting of shareholders held today. Each of the nominees proposed as a director, including…
Oil prices slipped in early trade on Wednesday after industry data showed a surprise build in U.S. crude stocks, while investors awaited U.S. inflation data for April that could give direction for the Federal Reserve’s next rate decision. Brent crude dropped 16 cents to $77.28…