Hi Cos,

batch_transforms use the event frame that's passed into handle_data(). fetch_csv just adds a new field to this event frame. The external data should thus show up in the datapanel that you access inside of the batch_transform -- does it not?

You are probably just filtering them out by your use of sids. If you do print datapanel.major_axis and print datapanel.minor_axis inside the batch_transform you can see the fields and sids that it contains.

Hello Cos,

You might also look into http://pandas.pydata.org/pandas-docs/dev/computation.html#expanding-window-moment-functions. I assume that when you say "... shift it backwards a couple of days" you are needing to compute a statistic over several different trailing window lengths. The pandas expanding window functions will work, so long as you flip the order of the data you feed them. Let me know if you need an example.

Alternatively, are you needing to know what the value of a statistic was several days ago, for a fixed trailing window length?

Grant

Hello Anony,

No time now to work on an example, but if you are talking about comparing daily overnight gaps (open compared to prior close), then the history function might do the trick (my understanding is that the batch transform will eventually be deprecated).

By the way, I wrote the "Sanskrit" above. It is actually pretty straightforward...you just have to dig into the numpy docs. I have some familiarity with MATLAB, so matrix operations seem natural (although I'm gaining an appreciation for all of the keying of data by security and loops over securities).

Grant

Anony,

Here's an example. Execution of the code in handle_data is only for the first minute of every day, so it runs fairly quickly (although not yet tested with a larger number of securities):

def initialize(context):  
    context.stocks = [sid(8554),sid(19920),sid(33652)]  
    context.prior_datetime = None  
def handle_data(context, data):  
    if context.prior_datetime == None or context.prior_datetime.day != get_datetime().day:  
        context.prior_datetime = get_datetime()  
    else:  
        context.prior_datetime = get_datetime()  
        return  
    opens = history(5,'1d','open_price').as_matrix(context.stocks)[0:-1,:]  
    closes = history(5,'1d','close_price').as_matrix(context.stocks)[1:,:]  
    print get_datetime()  
    print opens - closes  

Grant

Anony,

I don't know what kind of analysis/algo you have in mind, but it seems that some accounting for the wall-clock time lapse between close and the following open is in order here (e.g. early closes/holidays/weekends/etc.). The gap over a three-day holiday weekend should not be treated the same as for two consecutive normal trading days, for example.

Grant

Hello AM,

The attached is a bit of nonsense that goes long the securities with the largest gaps in terms of standard deviation. But it does use batch transform and daily mode. If it wasn't for Celgene...

P.

Peter, I haven't been reading this thread closely, but I did catch the reference to CELG_Z.

We had a bug that was fixed many months ago where we weren't handling the the sub-classes of stocks very well. CELG was one of the poster children for that bug. People were intending to trade CELG, and what they got by mistake was CELG_Z. I suspect that this list of stocks is a carryover from that time. Here is the same algo, but I took out CELG_Z and I replaced it with the "real" CELG which is what I expect you wanted anyway.

Hello Dan,

Thank you. I actually corrected the list at https://www.quantopian.com/posts/list-of-nasdaq-100-sids-to-use-in-your-algo in October last year. It seems Jess used the wrong list here https://www.quantopian.com/posts/intraday-strategy-gap-play-please-help yesterday and I copied it!

P.

Hello AM,

Sounds interesting. Please note that Grant has a much greater knowledge of NumPy (and mathematics in general) than me, so you may well find oddities in my code!

P.

Hi Anony,

Just a note that I botched up the code above. It should be:

    opens = history(5,'1d','open_price').as_matrix(context.stocks)[1:,:]  
    closes = history(5,'1d','close_price').as_matrix(context.stocks)[0:-1,:]  

This way, when gap = opens - closes is computed, the last row of the numpy ndarray should correspond to the current gap.

Grant

Hello Anony,

In Quantopian API trading with Interactive Brokers, I've gathered that at the start of day, data from Nanex will be accumulated into a minute bar (I'm not sure if Quantopian does the accumulating, or if the bar is fed to them). So, once the market has been open for 1 minute, all of the Quantopian algorithms will receive the first OHLCV minutely bar in handle_data. Your algorithm can then decide to submit an order based on the gap between the current open and the prior market close. If the algorithm submits an order, the guidance I've seen is that typically it will be filled by IB within seconds (but there is no guarantee); another call to handle_data is not required for order fulfillment. Your algorithm will not have access to the actual time the order was filled by IB (see Alisa response on https://www.quantopian.com/posts/position-entry-date), although you can after-the-fact look at IB records.

So my read is that for any kind of overnight gap algorithm to work, the new information will need to be reflected in the price rather slowly, otherwise the Quantopian/IB API won't be effective.

Grant

Hello AM,

I haven't looked seriously at amending your code to 'minutely' but the attached is 'minutely', uses 'history', and trades at 09:31 EST. A four month backtest takes 8 minutes to run.

I haven't looked at the operation of the algo at all i.e. it opens short and long positions simultaneously (GMT timestamps):

2013-10-01 14:32:00 ADSK    SELL    -246    $40.52  ($9,968.41)  
2013-10-01 14:32:00 ADSK    BUY      246    $40.50  $9,962.51  
2013-10-01 14:32:00 DELL    BUY      723    $13.82  $9,992.58  
2013-10-01 14:32:00 DELL    SELL    -723    $13.81  ($9,987.52)  
2013-10-02 14:32:00 ADI     SELL    -214    $46.62  ($9,976.89)  
2013-10-02 14:32:00 ADI     BUY      214    $46.60  $9,972.19  

The performance of minutely backtests is a major issue - in this example 388 of the 389 (389 of the 390?) calls to handle-data are superfluous. I think Dan mentioned the possibility of a 'history' which only delivers the 09:31 and 15:59 bars? Dan?

P.

For what it is worth, here's the min and max close-open gap, as a z-score for a 30-day trailing window, for Peter's data set above. It doesn't show any really wild swings. I suppose that the real question is does the gap contain any information predictive of the price? Not sure how to analyze that at this point. --Grant

Hello Anony and Peter,

Hmm...I may have gotten a "trade on the market-closed gap" algo to work (see attached). The code could be re-factored a bit and checked, but I think it is running properly. The returns are pretty choppy (high draw-down), but maybe to be expected since at any given time, the algo is short only one security and long another. Any idea if the algo stays within margin limits (or if more margin should be used)?

[EDIT] Please see https://www.quantopian.com/posts/market-closed-gap-trade for discussion and updates.

Grant

Hello Grant,

I think we definitely need a default margin model in addition to slippage and commission. I can see example IB margin calculations at https://www.interactivebrokers.com/en/index.php?f=margin&p=overview3 but I don't fully understand it yet.

P.

I started a new discussion, with the algo I posted above:

https://www.quantopian.com/posts/market-closed-gap-trade

Please comment/contribute there.

Grant