Thank you!

@Joakim, you are welcome. I can think of following:

1. Changing history window lengths
   
2. Changing model parameters (sensitivity)
   
3. Changing universe of stocks (select different subsets)

I would be interested in hearing other's thoughts on ways to identify overfitting.

@Satya,

Thanks for the reference. In the context of Q framework where availability of equity price and fundamental data are limited ( I think prices begin in 2002 and fundamental in 2004), measurement of probability of overfit may be quite a challenge. For example, one is working on a 10 year data and does a 80/20 train / test split with the test (OOS) set is the most recent available data. The results of the insample backtest show a Sharpe ratio of 2.0 while the OOS results show a Sharpe of 1.0, one cannot outright conclude that there is significant overfit just based on these metrics. One can improve judgement of overfitting by taking the linear statistics of the 2 year OOS in the mean/variance sense and search historically for the most similar 2 year period and then compare it's Sharpe ratio. The search can also be done in a non linear way through nearest neighbor algo in high dimensions. The main point is trying to measure apples to apples or as close as one could to make a more meaningful judgement of overfitting given the non stationarity of data over time. Just my two cents...

Thomas’ notebook using odd/even quarters to train/test might help with this I reckon, right?

You might also want to consider whether the vast numbers of constraints used in the optimization process are helpful in the context of fitting. Also some of the algorithms with a vast array of factors.

People may have been toying around with these myriad of knobs. Or they may not.