anilb: Quantopian IDE backtest is very slow for my simple code

Back to Community

edited Jun 21, 2018

Quantopian IDE backtest is very slow for the code below, even for dates that are 6 months apart. Whats the way out? How can I speed it up?

import numpy as np  
import math as math

def initialize(context):  
    context.security = symbols("AAPL","NFLX")  
def getMovingVol(data, security, n):  
    m=n*420  
    price_hist = data.history(security, 'price', m ,'1m')  
    a=price_hist.mean()  
    diff=price_hist-a  
    diff_sqr=diff*diff  
    diff_sqr_sum=math.fsum(diff_sqr)  
    diff_sqr_sum_avg=diff_sqr_sum/m  
    diff_sqr_sum_avg_sqrt=math.sqrt(diff_sqr_sum_avg)  
    return(diff_sqr_sum_avg_sqrt)

def handle_data(context, data):  
    for security in context.security:  
        if security in data:  
            mvvol2 = getMovingVol(data, security, 2)  # 2 day mavg  
            mvvol3 = getMovingVol(data, security, 3)  # 2 day mavg  
            mvvol7 = getMovingVol(data, security, 7)  # 1 week mvg  
            mvvol30 = getMovingVol(data, security, 30)  # 1 month mvg  
            mvvol90 = getMovingVol(data, security, 90)  # 1 qtr mvg  
            record(Vol_2=mvvol2,  
                  Vol_3=mvvol3,  
                  Vol_WK=mvvol7,  
                  Vol_MN=mvvol30,  
                  Vol_QTR=mvvol90)

4 responses

Josh Payne

Jun 21, 2018

At first glance, looks like you might be invoking handle_data on every minute of the simulation. Using schedule_function to invoke that logic less frequently will likely improve performance. More information can be found here: https://www.quantopian.com/quantopian2/best_practices

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Bryan Richardson

Jun 21, 2018

Anil,
Like Josh said, handle_data uses minute data, which is really slow. If all you need is EOD OHLC data then something like this might work.
This way you aren't dealing with minute data at all. Also, notice that this retrieves the longest period of data once for each of your symbols instead of getting the data for each of your moving average periods. Get the data once and then slice it as needed. Should be faster.
Of course you probably already know that 7 trading days is not a week. 5 days is. 252 trading days in a year....
EDIT: copy/pasted the wrong code before. This should work.

import numpy as np  
import math as math  
import pandas as pd

def initialize(context):  
    context.security = symbols("AAPL","NFLX")  
def getMovingVol(context, data, n):  
    price_hist = context.price_history_close[-n:]  
    print(price_hist)  
def before_trading_start(context, data):  
    for security in context.security:  
        context.price_history_close = data.history(security, ['price'], 90, '1d')  
        mvvol2 = getMovingVol(context, data, 2)  # 2 day mavg  
        mvvol3 = getMovingVol(context, data, 3)  # 3 day mavg  
        mvvol7 = getMovingVol(context, data, 7)  # 1 week mvg  
        mvvol30 = getMovingVol(context, data, 30)  # 1 month mvg  
        mvvol90 = getMovingVol(context, data, 90)  # 1 qtr mvg

Vladimir

Jun 21, 2018

This should be faster.

# ----------------------------------------------------------------------------  
stocks, ma_2, ma_3, ma_w, ma_m, ma_q = symbols('AAPL','NFLX'), 2, 3, 5, 21, 63  
# ----------------------------------------------------------------------------  
def initialize(context):  
    schedule_function(record_vol, date_rules.every_day(), time_rules.market_close())

def record_vol(context, data):  
    for stock in stocks:  
        if data.can_trade(stock):  
            minute_data = data.history(stock, 'price', ma_q*390,'1m')  
            vol_2 = minute_data[-ma_2*390:].std()  
            vol_3 = minute_data[-ma_3*390:].std()  
            vol_w = minute_data[-ma_w*390:].std()  
            vol_m = minute_data[-ma_m*390:].std()  
            vol_q = minute_data[-ma_q*390:].std()  

            record(vol_2 = vol_2, vol_3 = vol_3, vol_w = vol_w, vol_m = vol_m, vol_q = vol_q)

Jamie McCorriston

Jun 21, 2018

@Anil: Bryan's feedback of getting the data once and then slicing it is a great suggestion. In general, the slow step in this algorithm is actually retrieving the pricing data. If you get all the data you need in one call, and then just slice it to get shorter windows, you will save a lot of time. I attached a version that gets all of the data in one query and then slices into it as needed. It also uses Bryan's getMovingVol function which cleans the code up a bit (thanks Bryan!).

Disclaimer

You've successfully submitted a support ticket.

Our support team will be in touch soon.