I've been trying to build strategies and I realize that there is great importance in out-sample testing, back-testing for an extended period of time, and so on.

But what I'm really wondering is how to determine when a strategy's statistical significance is gone or if it was statistically significant to begin with. Now the current means of calculating statistical significance from my understanding is by using the t-statistic (Sharpe ratio (for the year) * sqrt(N years)) which is fairly reasonable. If you have a few years with a high sharpe, then that's great. However, I've noticed some strategies, including those posted on the forums, that have a decent sharpe ratio but plateau or have negative performance for a year and don't rebound that much or rebound for a little but don't exhibit the same return distribution.

So, what are some ways to know that a strategy has gone bad? Do you sample from the current period and look at the log return distribution compared to the return distribution of recent years? (ANOVA test) Do you have a rolling t-statistic? Do you cut out when the t-statistic drops below a point where there is no longer a 5% or 1% p-value-level statistical significance?

And when it cuts out, do you keep an eye on the strategy and when the rolling t-statistic becomes favorable again, you jump back into the strategy?

Since data-mining bias has run rampant in finance (my opinion and, from what I see, opinions of many others too) because everyone wants to try to find the best strategy, make money and there's really one data set that, at the end of the day, determines whether or not you make money, how does one deal with this?

Because any given person is clearly going to observe the output of multiple back-tests since it's not an every day occurrence that someone fresh to algo trading spits out code and miraculously develops a strategy with a 2+ Sharpe the first time around.

Edit: Currently taking a look at the lecture on hypothesis testing, I suppose my question more or less lies in how you determine statistical significance has disappeared and whether or not it makes sense to get back into a strategy that, for a period of time, was no longer statistically significant.