There should be a theoretical basis behind all your algorithms. The basis should give you a risk premium. Everything else is just fluff. If your algorithm simply does well because of overfitting stop losses, then you're setting yourself up for trouble.

Minh - I definitely agree with that. Assume I have a theory/strategy that I am using, and as a part of that I feel like stop-losses are a good risk-management tool. I'm not overfitting just by trying to figure out what my stop loss percentage should be, am I? Thanks for your input.

Nothing wrong with trying some things and going with what works best. Seems reasonable.

I generally use a few rules of thumb to check (though not ensure) that I'm not fooling myself and 'overfitting' a particular parameter. These aren't exactly statistically sound but more 'seat of the pants'.

1. Small parameter changes should result in small output changes. Vary the parameter a little bit. If the output results change a lot be suspect.
   
2. Directional parameter changes should result in consistent directional output changes. Vary the parameter up then down. If the output consistently follows the direction (or inverse direction) then good. If the output direction changes then be suspect.
   
3. Inverting the parameters should invert the results. This may not be applicable to all parameters (maybe not the stop loss described above) but try flipping the logic. If a high factor value generates high returns and a low factor value generates low returns, then good. If a low factor value also creates high (or even medium) returns then be suspect.
   
4. Parameters should be independent of timeframe. Try backtesting across different time frames. If the same parameter choices come up as optimal then good. If parameters work in some times and not others then be suspect.
   
5. Parameters should be independent of the assets. Try backtesting across different independent samples of assets. If the same parameter choices come up as optimal then good. If parameters work with some samples and not others then be suspect. (See below for a custom classifier to use to generate random samples using a pipeline. Set a filter for 'universe 0' then do the same but filter for 'universe 1'. The output results should generally be the same for both samples.)

from quantopian.pipeline import  CustomClassifier  
import numpy as np

class RandomUniverse(CustomClassifier):  
    inputs = []  
    window_length = 1  
    dtype = np.int64  
    missing_value = 9999  
    universes = 2  
    seed = 1  
    size = 10000  
    np.random.seed(seed)  
    random_buckets = np.random.randint(universes, size=size)  
    def compute(self, today, assets, out):  
        # get the asset ID into a range 0-9999  
        adjusted_ids = np.remainder(assets, self.size)  
        out[:] = self.random_buckets[adjusted_ids]

Use the above code instead of the one in the notebook. The same code can be adapted to run within an algorithm too.