@James, I totally agree with you. There could be over-fitting here. As a second step, one can run expanding window regressions to test the performance of an alpha factor. Other OOS testing would be to keep a separate window for modeling and validation. I will try that.
I didn't use the exact procedure for mini-contest. In mini contest, I would just research an individual factor, add it linearly with other factors (if significant) but I have never liked the approach. That approach doesn't give any visibility into predictive ability of new factor in presence of other factors. In a separate thread, I requested Q to create something like this in their Alphalens module but there wasn't any feedback on it so thought of creating one myself (it didn't turn out to be difficult at all).