What's the rationale for locking together the granularity of instant backtest and manual backtests? If the point of the in-IDE test is to provide a quick snapshot of how your algorithm performs and/or handles events, what's the expected use-case of running minutely backtests in the IDE?

Scott! Good to see you! Shouldn't you be studying for finals? ;)

I'd argue that the point of the in-IDE test is to validate that the code is doing what you think it should - buying at the right time, selling at the right time, etc. You're refining an idea and you want to make sure that your code and your idea are in sync.

There are two problems.

1. Many algorithms depend on the by-minute granularity. If you are an intra-day trader, the daily backtest is meaningless in terms of performance. The daily backtest might still be valuable for judging if your code is faithful to your idea, but the returns are not meaningful.
   
2. Daily backtests are too slow to get a quick snapshot of how your intra-day algo will do over the long term.

The problem in #2 is something we can solve in the long run. We can make the backtester faster. We know we can, it just requires time and effort that we haven't applied yet.

The problem of #1 is not solvable. If you're doing an strategy that relies on minute data, there is no way to make daily data useful as a performance measure.

That's what took us to this proposal. The worst part of this proposal is the in-IDE test is slow for minute data. The workaround for that is to use a short timeframe. For everything else, this proposal is better (I think).

Make more sense? Something I'm missing?

No studying at this point, just a 12-15 page paper on Wittgenstein that needs to be produced by Sunday. One of the benefits of writing a thesis and doing mostly project-based courses is a lighter finals week than most :).

The point about daily data not being useful for certain kinds of algorithms makes sense. I think what initially confused me was that this change, if I understand it correctly, essentially makes the in-IDE backtester functionally equivalent to the "full" backtester, since you will run the exact same set of tests on both. While this is a departure from the current model, I think it's actually probably a good change, since if you don't care about watching all the fancy-dancy GUI stuff going on in the full backtester (I'm sure Jean will cringe if he reads that description...), you can just run a long test on the side of the IDE while you work.

My only other thought is that it might make sense to have separate functions for setting granularity, ie set_instant_granularity and set_full_test_granularity or somesuch, since I imagine a fairly common workflow will still be daily for the instant test and minutely for the full test, and it's a little awkward to have to go back up and change a line every time you want to run a longer test. That's a pretty minor concern in the scheme of things though, so if putting in a separate toggle required a lot of work my guess would be that your guys' development time would be better spent elsewhere.

Hi Dan,

I suggest considering, testing, and documenting in detail how the pre-defined (e.g. mavg) and batch transforms will perform under the various scenarios.

Also, a few more details on the daily bar data would be helpful. What are they? Are they the opening & closing price for the day, with the total volume for the day? What is the associated datetime stamp? Etc. I don't see the details on your help page.

Also, any reason you couldn't expose the daily bar data when a minutely backtest is run?

And for the daily bar data, seems you could also expose the datetime stamps corresponding to the opening price and closing price. This would be a handy way to capture when a given security started trading and stopped trading on a given day. Besides, the daily opening/closing price are time series data, so you are throwing away information by not also providing the datetime stamps.

Scott - better you than me writing that paper.

We definitely thought about having separate toggles for the in-IDE tester and full backtester. I'm curious if other people have the same reaction as you do - it would change our thinking. The reason we didn't put that in our proposal is that we just didn't think a split-testing mode would be that useful of a mode - the results are too different. I can be persuaded that it would be useful,still.

Grant - The built-in transforms and batch transforms will continue to behave as they do today. For built-in transforms like mavg(), in the daily mode, a mavg(3) will be the moving average of the last three days. In minute mode, mavg(3) is the average of the last three days in a rolling window, i.e. the mavg(3) of the price as determined at 1pm is the is the average of the prices since 1pm three days ago. Batch transforms will continue to update on the refresh period. The number of data points in a batch transform is different for daily and minutely, but the windows of data are identical. I do have a couple ideas how to expose this all better.

The daily bar is just like the minute bar - OHLCV. https://www.quantopian.com/help#ide-properties

We can't expose the daily bar in the minute bar because it would create look-ahead bias. Say it's 1pm; if you know what the closing price of today's stocks would be, you would make a mint! You have access to yesterday's daily bar using the batch transform.

So far I've got Scott interested in a split mode - definitely interested in hearing other thoughts over the next few days.

Thanks Dan,

Good point regarding the look-ahead bias. Sometimes I forget the real-time approach. There's some funkiness with mavg(days) and the batch transform in that under the minutely backtest, the windows don't shift on a minutely basis. I not sure your statement above is correct. Try logging mavg(days) in the minutely backtester. As I recall, the average will remain at a fixed value for an entire day. The same goes for a moving average calculated via the batch transform (since the minimum refresh period is one day...maybe...it'll also run with a zero value refresh period). When I get the chance, I'll post a sample algorithm to illustrate.

Hello Dan,

I've attached an illustrative backtest (nice new feature!). If I haven't botched something up, I am computing mavg(1) for two sids and I am also using the batch transform with refresh_period=1 & window_length=1 to compute a moving average. I log event time, sid prices, and the results of the mavg(1) and batch transform averages. Regarding the mavg(1) transform, it appears that it does not perform as you describe above. Have a look at the log output below, for the 2012-01-03 closing minute and the 2012-01-04 opening minute. The mavg(1) calculation resets to the opening price--it does not look back over the prior days minutes. In the second minute of 2012-01-04, mavg(1) provides the price average of the first two minutes on trading on 2012-01-04. It appears to be a within-day moving average with a varying window size.

I'm still getting up the learning curve on the batch transform/datapanel usage, but if I've done things correctly, under the minutely backtest, I would have expected the function I wrote to access a trailing window to compute the average. Instead, it just reports the opening price for the day (see log output for
2012-01-04 below). Interestingly, the outputs of mavg(1) and my function match on 2012-01-04 for refresh_period=0 & window_length=1 (although the minutely backtester slows way down).

Note that as I reported earlier, under the daily backtest, the log output is inconsistent for the algorithm I posted here. Sometimes there is output, and sometimes not. As you mentioned, a bug has been logged.

2012-01-03handle_data:25DEBUG 2012-01-03 20:59:00+00:00  
2012-01-03handle_data:26DEBUG price: [ 51.04 74.17]  
2012-01-03handle_data:27DEBUG mavg: [ 51.19194859 74.61089717]  
2012-01-03handle_data:28DEBUG bt mavg: None  
2012-01-03handle_data:25DEBUG 2012-01-03 21:00:00+00:00  
2012-01-03handle_data:26DEBUG price: [ 51.11 74.2 ]  
2012-01-03handle_data:27DEBUG mavg: [ 51.19173846 74.60984359]  
2012-01-03handle_data:28DEBUG bt mavg: None  
2012-01-04handle_data:25DEBUG 2012-01-04 14:31:00+00:00  
2012-01-04handle_data:26DEBUG price: [ 51.07 74.14]  
2012-01-04handle_data:27DEBUG mavg: [ 51.07 74.14]  
2012-01-04handle_data:28DEBUG bt mavg: [ 51.07 74.14]  
2012-01-04handle_data:25DEBUG 2012-01-04 14:32:00+00:00  
2012-01-04handle_data:26DEBUG price: [ 51.1 74.295]  
2012-01-04handle_data:27DEBUG mavg: [ 51.085 74.2175]  
2012-01-04handle_data:28DEBUG bt mavg: [ 51.07 74.14]  
2012-01-04handle_data:25DEBUG 2012-01-04 14:33:00+00:00  
2012-01-04handle_data:26DEBUG price: [ 51.05 74.42]  
2012-01-04handle_data:27DEBUG mavg: [ 51.07333333 74.285 ]  
2012-01-04handle_data:28DEBUG bt mavg: [ 51.07 74.14]  

Hi Dan,

The split mode suggested by Scott above sorta makes sense. Rather than two separate functions, consider:

initialize(context)  
    set_granularity(quick = 'daily', full = 'minutely') 

initialize(context)  
    set_granularity(quick = 'minutely', full = 'minutely') 

initialize(context)  
    set_granularity(quick = 'daily', full = 'daily')

 initialize(context)  
    set_granularity(quick = 'minutely', full = 'daily')  

The final one (set_granularity(quick = 'minutely', full = 'daily')) doesn't seem useful, so you could default to the first one (set_granularity(quick = 'daily', full = 'minutely')) and then just offer your original proposal, instead of the split mode.

Regarding the daily bars, I understand that you provide OHLCV data. For completeness, you should also provide the minutely datetime stamps corresponding to the daily OHLC prices. You discard this information, but it could be useful in an algorithm--the market is a time series. Did Quantopian derive the daily bar data from the minutely data? Or did you get both from your vendor?

The same could be said for the minutely bars, for which your vendor would need to provide the sub-minute datetime stamps for the HL prices (the datetime stamps for the OC are already derivable/accessible).

Regarding exposing the daily bar data under a minutely backtest, I'd encourage you to give this idea more thought. You have the data, and a look-ahead bias can be avoided if the daily bar data are only available in a trailing window. If a minutely backtest utilizes daily bar data, it'd be more efficient to use the data that you already have, versus writing a batch transform to compute the trailing daily bars on the fly. In the same fashion, you could provide access to the minutely backtest data under a daily backtest. This would allow convenient daily ordering based on minutely information, for example. The overall philosophy is that regardless of the granularity of the backtest, all Quantopian data should be available (both daily and minutely).

Hi Dan,

Another thought...I noticed that the datetime event stamp for a daily bar does not correspond to the minutely datetime stamp for the closing price of the day. The daily bar information becomes fully available at the close, so shouldn't the daily bar event datetime stamp be at the close (rather than at 00:00:00+00:00, as you use now)?

I've been able to track a handful of bugs related to the moving average window definition. Fixing them will take a bit, but it's very high on the list. It's the "next" project after the current milestone, so we should have a revision in January sometime.

Sounds good. No rush on my part. Anyway, I think I'll end up using the more flexible batch transform to calculate my own moving average, etc.

Hi Dan,

Few thought on Daily bars.
1. I understand that providing Daily bars could create a look ahead bias. On the other hand, by providing a "pre-rendered" Daily series on the minute backtest one can run some very common and useful tests with minimal overhead to the system. Some examples: Check if Closing price (EOD) is above the 200 day moving average before trading. Check if yesterday was an up day. Check the average daily volume for the last 22 days. These are easy to code using daily bars. Of course they can all be done with transforms but ti could frustrate non-programmers who are used to DailyClose < YesterdayDailyClose type of statements.
2. Eventually people will want to "rank" or "filter" stocks by going through the whole available universe. For example someone might want to trade the 30 best performing stocks of the last quarter. Or the stocks that fell more than X% in the last 3 days. This is easier with daily data, even for 100+ symbols (series).
Still, I am all for custom data frequencies.

Hello Dan,

I'm looking forward to this upgrade. The OLMAR algorithm that Thomas W. and I implemented seems to run pretty well under the quick/daily backtester. I'd like to run it on a daily basis in the full backtester, with access to all of the canned tracking and performance outputs. I'd started to modify the code to restrict trading to daily, but have held off since it sounds like this upgrade will go through.

What is the status?

Grant

Sanz, your points are very well taken. I think you're looking for data exploration as a feature. I (we) completely agree we need that kind of data exploration/stock screening/target identification functionality, and it's on the roadmap. That is a pretty big effort to do correctly. We're focused on getting to live trading before we do a big investment in data exploration, and that pushes data exploration out many months. That said, we are going to try to opportunistically do some quicker, smaller data exploration features over the next few months.

Grant, I'm hopeful that we can ship the daily/weekly granularity later this month. As a product manager, I know better than to make that a promise ;)

Hi,
Is it also possible to backtest on patterns in low Millisecs/Microsec range Time Series and Does it also include Nanex Feed/NxCore API ?
Thx
Anil

Hello Anil,

We aren't looking to support high frequency trading - the lowest we're going is minute level, at least for the forseeable future.

Thanks,

Dan