Attached, please find the algorithm that I could not attach to the post above (no "Add Backtest" button). --Grant

Hello all,

Any insights on this? I'd like to close it out.

Thanks,

Grant

Hi Grant,

So sorry for the delay - Dan forwarded this to me when you first submitted it and I didn't get back to look at it until now. I've been working on a really fun new feature, and I let this slip - sorry!

The issue you have uncovered is by design. We don't guarantee any order on the data parameter to handle_data or to the dataframe columns in the datapanel sent to batch transform. Instead, we decided to go the route of using keys/column labels.

Thanks for pointing this out.

thanks,
fawce

Thanks Fawce,

No problem...just didn't want it to slip by. I'm still not quite clear on what's going on. My goal is to have a sure-fire way of capturing a trailing window of security data in a numpy ndarrary. For the code above, the batch transform re-orders the columns when the sids are defined in a Python list. But if I define the sids with a Python set, the batch transform maintains the sid order.

Some questions:

1. I am not clear from your response that the behavior highlighted by my code above will be consistent. If I call the batch transform with the sids defined by a Python set, will I always get back an ndarray with the columns ordered according to the sid order in def initialize(context)?
   
2. When a call the batch transform with a Python list of sids, the ndarray columns are re-ordered. What criteria are used by the batch transform to order the columns? Or is it random?
   
3. What would be the best practice? From your response, it sounds like I should be using keys explicitly when extracting data from the dataframe columns. Is there a way to re-write the code above so that I obtain the same result, regardless of the Python data type of context.stocks (list or set)?
   
4. When using set_universe, are the sids in a Python list or set?

Grant

Hi Grant,

Thanks for detailing your thoughts. I'll jump right into your questions:
1. My read on the ordering is that the set is dumping the sids in order of their hashes, and that the datapanel['prices'].values is dumping the prices out in the order of the sid hashes. In other words, it is a coincidence of the implementation of the dataframe.values property and the implementation of set. Neither of those two objects should be considered ordered lists. The list, on the other hand, is ordered as you coded it. It should retain that sort until you modify the list.
2. It is deterministic, but based on the need to index into the columns, and so appears to be in order of the hash of the column key.
3. It really just depends on how you want to code your function. The key access is available on the dataframe, andif you want an ordered ndarray, pandas has a builtin function to produce a sorted ndarray. See the attached backtest for an example using the dataframe.as_matrix function.
4. The sids in the universe are available via the data parameter to handle_data - which is not sorted. You can iterate through the keys with:

for stock in data:  
   price = data[stock].price  

thanks,
fawce

Thanks Fawce,

This does the trick:

@batch_transform(refresh_period=R_P, window_length=W_L) # set globals R_P & W_L above  
def get_prices(datapanel,sids):  
    return datapanel['price'].as_matrix(sids)[0]  

Alternatively, for a window length greater than 1 (e.g. W_L = 5), I can use:

@batch_transform(refresh_period=R_P, window_length=W_L) # set globals R_P & W_L above  
def get_prices(datapanel,sids):  
    return datapanel['price'].as_matrix(sids)  

Time increases with increasing row number (with the last row corresponding to the current tic).

The code you provided would seem to maintain the ordering explicitly.

Grant