Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Resource Usage Monitoring

Hi,

Is it possible to give resource usage of an algo. CPU total and per thread, memory, etc. over the the time of backtests and input data. Also live trading data is there when live trading.

This is to identify performance bottlenecks base on situation in data.

Suminda

12 responses

In addition any resource constraints on the algo.

At a later stage may be memory and CPU profiling of the code. Perhaps as a heat map against the line numbers.

We've invested some in performance improvements of Zipline (the backtester) but there is TONS of work left, we know. We'll probably invest in beefing up performance that everyone can use automatically before we invest in specific, complex performance tools.

That said, if you're having a performance problem, let us know. I'd love to help troubleshoot it with you and see if we can make the fix in the back end.

I saw Alex's thread about ARMA and I'm researching on that one.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi,

Some code snippets from what I am running. I have removed the proprietary parts of the algo.


import math  
import pandas  
import numpy as np  
import scipy as sp  
max_lag = 40  
def initialize(context):  
     set_commission(commission.PerShare(cost=0.08))  
     set_slippage(slippage.VolumeShareSlippage(volume_limit=0.25, price_impact=0.1))  
     context.env_var = {}  
     context.env_var[sid(8554)] = exec_env_var({'index' : [sid(8554)], 'sub' : [sid(19662), sid(19659), sid(19655), sid(19656), sid(19661),  
 sid(19657), sid(19654), sid(8340), sid(19660)]})

class exec_env_var:  
     def __init__(self, secs):  
        self.sec = secs  
     sec_price = {'index' : {}, 'sub' : {}}  
     sec_price_ln = {'index' : {}, 'sub' : {}}  
     sec_return = {'index' : {}, 'sub' : {}}  

def calc_model(hist, env):  
     calc_price(hist['price'], env.sec, env.sec_price)  
     calc_ln(env.sec['index'], env.sec_price['index'], env.sec_price_ln['index'])  
     calc_ln(env.sec['sub'], env.sec_price['sub'], env.sec_price_ln['sub'])  
     calc_sr_chng(env.sec['index'], env.sec_price_ln['index'], env.sec_return['index'])  
     calc_sr_chng(env.sec['sub'], env.sec_price_ln['sub'], env.sec_return['sub'])  
     calc_pair_rtn(env.sec, env.sec_hg_sig, env.sec_return, env.sec_sig_rtn)  


def calc_sig_rtn(secs, sig, rtn, dest):  
     if not dest:  
         for x in secs:  
             dest[x] = []  
             dest[x] += [sig[len(sig[x]) - 1] * rtn[len(rtn[x]) - 1]]  
     else:  
         for x in secs:  
             dest[x] += [sig[len(sig[x]) - 1] * rtn[len(rtn[x]) - 1]]  
             lst_trim(dest[x])  

def calc_sr_chng(secs, sig, dest):  
     if not dest:  
         for x in secs:  
             dest[x] = []  
             if len(sig[x]) == 1:  
                 dest[x] += [0]  
     else:  
         try:  
             for x in secs:  
                 tmp = sig[x][len(sig[x]) - 1] - sig[x][len(sig[x]) - 2]  
                 dest[x] += [tmp]  
                 lst_trim(dest[x])  
         except:  
             pass  

 def calc_ln(secs, src, dest):  
     if not dest:  
         for x in secs:  
             dest[x] = []  
             dest[x] += [math.log(src[x][len(src[x]) - 1])]  
     else:  
         for x in secs:  
             dest[x] += [math.log(src[x][len(src[x]) - 1])]  
             lst_trim(dest[x])  
 def calc_pair_rtn(secs, sig, rtn, dest):  
    i = secs['index'][0]  
    if not dest['sub']:  
        for x in secs['sub']:  
            dest['sub'][x] = []  
            sub_rtn = sig['sub'][x][len(sig['sub'][x]) - 1] * rtn['sub'][x][len(rtn['sub'][x]) - 1]  
            idx_rtn = sig['index'][i][len(sig['index'][i]) - 1] * rtn['index'][i][len(rtn['index'][i]) - 1]  
            dest['sub'][x] += [sub_rtn - idx_rtn]  
    else:  
        for x in secs['sub']:  
            sub_rtn = sig['sub'][x][len(sig['sub'][x]) - 1] * rtn['sub'][x][len(rtn['sub'][x]) - 1]  
            idx_rtn = sig['index'][i][len(sig['index'][i]) - 1] * rtn['index'][i][len(rtn['index'][i]) - 1]  
            dest['sub'][x] += [sub_rtn - idx_rtn]  
            lst_trim(dest['sub'][x])  
    if not dest['index']:  
        dest['index'][i] = []  
        idx_rtn = sig['index'][i][len(sig['index'][i]) - 1] * rtn['index'][i][len(rtn['index'][i]) - 1]  
        dest['index'][i] += [idx_rtn]  
    else:  
        idx_rtn = sig['index'][i][len(sig['index'][i]) - 1] * rtn['index'][i][len(rtn['index'][i]) - 1]  
        dest['index'][i] += [idx_rtn]  
        lst_trim(dest['index'][i]) 


 def calc_price(hist, secs, dest):  
     idx = secs['index']  
     sub = secs['sub']  
     for x in sub:  
         dest['sub'][x] = hist[x]  
     dest['index'][idx[0]] = hist[idx[0]]  
def lst_trim(lst):  
     while(len(lst) > max_lag):  
         lst.pop(0)  
@batch_transform(window_length=max_lag)  
def get_past(datapannel):  
     return datapannel  

There is something wrong with the mark down syntax.

Using similar function we create a series of transformation based on which final trades are decided.

There is something wrong with the mark down syntax.

Hi Suminda,

Could you be more specific?

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Look at the previous post in the thread. This is just a company and past and clicking on back quote. The output does not look right.

Suminda, it works better for displaying code if you click "codesample" rather than "backquote." I edited the post.

Thanks for the example! We'll look into it.

I edited it. Something went wrong.

It looks fine now.

Thanks for your support. I calculate about 20 more indicators not shown here. This is calculated using similar code. Let me know when you have some more insight into this.