Can Hypothesis Testing Reduce Data Mining Risks?

Subscribe to newsletter

A significant challenge in designing trading strategies is the data mining problem, which arises from the vast amount of data available and the potential for spurious correlations. With an abundance of historical market data, traders may inadvertently identify patterns or relationships that appear significant but are merely coincidental. This can lead to overfitting, where a strategy performs well on historical data but fails to generalize to new market conditions.

To mitigate this issue, rigorous testing procedures, such as out-of-sample testing and cross-validation, are essential to validate the effectiveness and robustness of trading strategies and guard against data mining biases.

Reference [1] presents a method for minimizing data mining risks using hypothesis testing without requiring out-of-sample data. Specifically, it employs the false discovery rate (FDR) method to address this issue. The authors pointed out,

In this paper we study if the use of simple technical trading rules can outperform buying and holding bitcoins. We attempt to do this by first selecting outperforming rules, from a set of 75,360 possible trading rules, then combining them in different portfolios and finally assessing their performance out-of-sample after ’realistic’ transaction costs. Compared to earlier research, which generally concludes that trading rules can outperform a buy-and-hold strategy in the bitcoin market, we apply much more restrictive conditions (transaction costs, out-of-sample performance, data mining corrections) and search over a higher amount of technical trading rule classes, parameterizations and trading frequencies. Unlike in Hudson and Urquhart (2021), we find that our rules still can significantly outperform the buy-and-hold strategy out-of-sample, mainly risk-return wise.

In short, the study evaluated 75,360 simple technical trading rules. The best-performing rules are selected after factoring in transaction costs using a multiple hypothesis procedure. Portfolios are then formed by combining the selected rules, and their out-of-sample performance is shown to be superior to Buy and Hold’s.

We find it interesting that through in-sample hypothesis testing alone we can significantly reduce data mining risks and achieve favorable out-of-sample results. We remain open-minded and continue to monitor developments in this area of research.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Niek Deprez1, Michael Frömmel, Are simple technical trading rules profitable in bitcoin markets?, International Review of Economics & Finance, Volume 93, Part B, June 2024, Pages 858-874

Further questions

What's your question? Ask it in the discussion forum

Have an answer to the questions below? Post it here or in the forum