Before Quantopian I was into auto-tuning to some degree (eventually will revisit via Zipline I suppose). Picture 20 to 30 variables like thresholds and lookback window values for indicators like sma, macd, whatever. If I had tried to run every possible combination it would have taken years to run all the way thru. So instead used an infinite loop and dictionary something like below. First it would run everything with the base values. Next, pick the first parameter, say, ['sma']['fst'] and change its base by the given increment upward, from 6 to 7 in that case. Do the entire run again. If the output is higher, increment that upward one more time. If lower, try from 6 to 5 downward etc.

I was only deviating from the base two increments up or down. Once a higher run value was obtained the base value would be set so others would be using that better value. At the end of each run in the loop, the result and all variables were saved/printed for the record (later changed to only print when the result was higher). Checking a 24/7 machine and finding the result is up another 2.7% is awesome.

tweaks = {  
    'sma': {  
        'fst': {  
            'base': 6,  
            'increment': 1,  
        },  
        'slw': {  
            'base': 35,  
            'increment': 3,  
        },  
    },  
    'rsi': {  
        'fst': {  
            'base': 7,  
            'increment': 1,  
        },  
        'slw': {  
            'base': 82,  
            'increment': 4,  
        },  
    },  
    'threshold': {  
        'sell_above_buy': {  
            'base': 1.29,  
            'increment': .03,  
        },  
    },  
}

Thanks for the info Gary, yeah that's a brute force way we could do it for determining optimial fixed parameters during a backtest, but I'm thinking more along the lines of auto-adjusting parameters for a live algorithm.

some kind of hill-climbing algorithm algo with sensitivity analysis. If anyone else has ideas I'd love to hear :)

Jason, I've written two blog posts on this a while back: http://blog.quantopian.com/category/optimization/

Technically, it's not an easy problem as you would need to rerun the algo on past data to optimize going forward (see walk-forward optimization). So the easier first step would be to not do auto-adjusting but rather classical optimization on all data. Alternatively, we could look into reinforcement learning which does not require re-running of past data to evaluate different options.

Hi Jason (and Thomas),

Here's a library of potential interest to you:

http://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize

As a simple use case, here's a computation of the L1 median (https://www.quantopian.com/posts/collab-request-on-robust-median-reversion-strategy):

def l1_median(x):  
    """  
Computes L1 median (spatial median) using scipy.optimize.minimize_scalar

:param x: a numpy 1D ndarray (vector) of values
:returns: scalar estimate of L1 median of values
"""
    a = float(np.amin(x))  
    b = float(np.amax(x))  
    res = minimize_scalar(dist_sum, bounds = (a,b), args = tuple(x), method='bounded')  
    return res.x

def dist_sum(m,*args):  
    """  
1D sum of Euclidian distances

:param m: scalar position
:param *args: tuple of positions 
:returns: 1D sum of Euclidian distances
"""
    s = 0  
    for x in args:  
        s += abs(x-m)  
    return s  

Per Thomas' comment, it'd be nice to be able to iterate over backtests to do the optimization, as new data comes in (e.g. every day/week, adjust the algo parameters automatically). I'm wondering if it'd be possible to import zipline into the online backtester/trading platform? Then, data could be fed into zipline (e.g. from the history API) for optimization. One potential issue is the time limit on calls to handle_data, presently set to 50 seconds. All of the number crunching would have to be done within that timeframe (or the optimization interrupted every 50 seconds, storing intermediate values).

Grant

Thanks for the info guys. since in my QuantShim framework has objects for storing custom history, I'm using that right now to retroactively determine roi for various parameters, in a way mentioned by Gary (stepping interval)

I have an idea to use something like newton-raphson for hill climbing but I think I better check out the scipy package grant mentioned first :)

Hello Thomas & Jason,

An approach would be to do Quantopian/Interactive Brokers (IB) paper trading in parallel with real-money trading. Then, one could vary parameters at will when paper trading, and apply the resulting optimal parameters to the real-money trading. In my mind, this would allow an empirical estimation of the response surface without the risk (and trading cost) of tweaking a money-trading algo.

Other options would be to use a Quantopian simulated paper trading algo (15 minute delay) or a backtest (1 day delay), but there is the risk that there would be a loss of accuracy over the IB paper trading approach.

Is there any way to share data between algos in Quantopian? If not presently, is this something that could be enabled, or is it basically a non-starter idea?

Also, is the IB paper trading delayed? And how accurate is it, compared to real-money trading?

Grant

Another angle, I suppose, would be to use external data and zipline, and then feed the updated parameters into the running algo via fetcher. I haven't used fetcher--would this be feasible? --Grant

Grant, the fetcher is feasible. You can import your own signal / parameter values via a csv, the catch here is that the fetcher will import your csv as a time series.

def initialize(context):  
    fetch_csv('http://yourserver.com/cpi.csv', symbol='cpi')

def handle_data(context, data):  
    # get the cpi for this date  
    current_cpi = data['cpi']['value']

    # plot it  
    record(cpi=current_cpi)  

Hi Grant,

At present algo's can't share functions among themselves, you would need to copy/paste the code between your algorithms. The suggestion has surfaced a couple times and we'd like to add it, but we're not quite there yet.

For your second question, IB paper trading is not delayed, it receives the same data feed as the real-money algorithms. Only Quantopian paper trading, which is free, receives a 15 minute delayed price feed.

Thanks Alisa,

I'd also posted a question above, "I'm wondering if it'd be possible to import zipline into the online backtester/trading platform?" Is it feasible? Could zipline be run within the online backtesting/trading platform?

Grant

You can import some of the Zipline modules, but not all, into the IDE. We prevent some of the imports for security reasons since we're running your code on our servers.

I don't have the full list of modules handy, but a useful snippet you can add from zipline is the trading calendar. For example:

from zipline.utils.tradingcalendar import get_early_closes  

I agree that having some hyperparameter tuning functionality built into Zipline/Quantopian would be extremely helpful. Doing so elegantly might be a little tricky but still quite do-able I believe. I don't have any actual results to back this claim up but I think the gaussian process-based techniques mentioned in Thomas's blog post offer the most promise here. When it comes to backtesting, efficiency becomes crucially important and genetic algorithms, for example, take forever to converge if they ever do.