Tets parameters that should work at any time, and against other set of stocks, If you use the last 3 years to test, try the previous 3 years etc.

One of my first strategies had almost 14.000% gain in a 7 year minute test, but was loosing money if I ran it for just the last year. My current build have a gain on all years, along with 3, 5, 7 and the last ten years. I'm fairly happy that time is not essential in making it work. In 10 year test it even yields a tripple digit Sharpe. Sadly I never cared to deal with shorting, so I have too high beta for the contest, as shorting would lower the beta.

Ow, and make sure the algorithm work even if the parameters are sub ideal. If you deal with 0.01% or other rather arbitrary constant values of things to get a profit, its highly unlikely to make a profit in the long run.

I would add that I think it's important to know why your strategy works. Who are you buying shares from, and why are they selling them cheaply?

I think most people advise against constantly tweaking strategies, since if you are always changing your strategies, then effectively you're participating in a meta-strategy of tweaking over time, but this meta-strategy is un-testable; you're back to discretionary trading, but with a bunch of useless trappings of systematic trading.

Be skeptical; after you read a paper, if it makes sense, try to replicate it to disprove the thesis. Strive to understand the drivers of the alleged returns. If you cannot disprove it, and it has a reasonable basis, then start paper trading it.

This seems to be the opposite of what most people do, perhaps because it's thankless and depressing. It's much more gratifying to do massive multidimensional optimizations of the many parameters of an arbitrary technical analysis combination, and then post amazing backtests and get widespread kudos.

@ Raghavendra,

One thing to keep in mind is the nature of the data you are using. For Quantopian, minute bars are provided, but as I understand, they are constructed by sampling a stream of tic data, which can be very noisy, with outliers. If you go to http://www.quantcon.com/ and have a look at Ernie Chan's talk, he touches on this topic. So, try some smoothing to see if it helps your algorithms; don't assume that you need to use raw price and volume data to make trading decisions.

I would second Simon's statement about not constantly tweaking strategies so you can see how they perform in different market conditions over time. While my strategy isn't suitable for the fund (long only/beta >.3), I've learned a lot since it started in late January.