[Still raining here...]

From my understanding the context object, and everything you dynamically append to it is persisted throughout time until at what point an algorithm must be restarted (various conditions). The nature of this persistence, whether it's serialization and storage, is unknown. State stored on this object can be theoretically trusted to remain for subsequent processing events.

However, I have learned over the years that following the groundhog method of processing is the most robust. A groundhog wakes up, looks around the world, determines state from the conditions available to it, and acts on those conditions accordingly, and then goes back to sleep. Or, in other words, if you can form your algorithmic calculations and tests in a way that each time an event is triggered you rebuild from scratch all that you need to make decisions -- your algo will be much more robust. State maintenance can be problematic. But, as is, once again, my understanding, the context object will facilitate persistence.

Hello Mark,

Yes, context will work (a global might too?):

def initialize(context):  
    context.intermediate_result = None  

When your algo runs, you can store results in context.intermediate_result. Regarding "persistence" in live trading, my understanding is that Quantopian will work not to wipe out context data (e.g. when performing maintenance), but if the algo actually crashes, I'm not sure--it could be lost. Anybody know?

Grant

My need is to perform and store intermediate results from my own 'rolling forward' calculations including resampled candlestick bars and TA indicators based on resampled pricing action. For example, I want to know what resampled aggregated candlestick bars exist, or when CCI or MACD crossovers occurred let's say 7 bars ago, based on multi-minute data. For back testing optimization, I would like to store all of the above bars and TA indicator results for subsequent parameter iterations so I don't have to recalculate the same results on the same pricing data set. I can calculate the initial iteration results then simply read the calculated results on following iterations. I just need to know that Quantopian will allow me to store the intermediate data efficiently in 'fast' tables. By the way, are there any data accelerators employed? From my comments on a previous 1 min calculation time post, the calculation speed and data retrieval appear to be very slow based on simple algo back test times...

If you are talking about running a backtest, storing data, and then re-running the backtest using the stored data, then I don't think there is an easy/elegant way. There is no backtest-to-backtest storage, so you'd need to resort to copy and paste into the IDE, from either the debugger or log output (fetcher might work, too, but I've never used it). If you want to store data within a given backtest (e.g. on a rolling basis), then context will work.

If you need the copy-and-paste hack, then it used to be that in the debugger, with a setting change, a lot more data could be printed out than in the log window. If you need to do this, just let me know and I'll post an example (if it still works).

Grant

Grant, so you are saying there is no separate database instance within my Quantopian 'member space' that I can write data to outside of the context of a given algo build or backtest? That 'universal' database is not available to be accessed across algos, or available to synchronize algos if each algo is sharing the same IB acct? Does each algo have to be connected to a separate IB acct so multiple algos don't 'cross' their positions in the same account for the same security? Finally, does building or backtesting an algo 'wipe out' data connections assigned to that algo?

Maybe a different question is appropriate, can an algo write to a csv file format, then fetch_csv able to recover the csv data if the csv file exists?

Thanks!

Grant, I just read your post above about Context.intermediate_result. Is the .intermediate_result object a data object with data methods? Where do I find the API detail on Context?

...can an algo write to a csv file format, then fetch_csv able to recover the csv data if the csv file exists?

No. There is no means to write out data from a backtest to a file or memory location, and have it persist once the backtest is complete.

Where do I find the API detail on Context?

https://www.quantopian.com/help#ide-api

One thing to note is that the history API (https://www.quantopian.com/help#ide-history) provides a trailing set of data at the backtest start (sometimes called a backtest "warm-up"). So, you should consider if your calculations can be done on a rolling basis beginning at the start of the backtest; you shouldn't need to wait N days while accumulating data. In other words, if your indicators are derived from OHLCV bars, then you can re-compute the entire set of trailing indicators every minute (or less frequently, if you want), on a rolling basis. If your window is only 7 bars, then this should be efficient.