Hi Alan,

How orders work in daily and minutely mode is that, at the fastest, orders are executed at the next bar. So when you submit an order in daily mode on October 1st, the fastest it'll get executed is October 2nd at the closing. The same applies in minutely mode: 9:31 AM submit 9:32 execution (at the fastest).

However you can create your own custom slippage models to fit your needs. I believe Grant has created one on this thread (https://www.quantopian.com/posts/backtest-results-different-in-minute-and-daily-mode) to run on daily bars and execute on the same day.

Alan, does this version look more like what you were expecting? All I did was switch to minute mode and add a check for open orders.

Hi Seong,
I understand your argument. That is also what I guess.

For minute level test, this problem is not severe. But for day level, the back test result is totally off. And it can hardly be addressed by slippage model only.

A more reasonable solution is to update the definition of bar, no matter minute bar or day bar. A bar should have open/close/volume at the minimum. The order calculated based on close price of bar1 should be fulfilled by open price of bar2. If you want to make it more realistic, then take volume into consideration as well.

I point this out and urge for a solution cause the back test results here is unreliable. If my back test is good, I turn on the live trade. I may get totally different result.

Hi David,
Minute level would be a solution. I can run the calculation at 15:58 each day. So I can based on a price very close to market close to calculate and fulfill the order. But it has a couple issues:
1. Speed. This is quite obvious.
2. I need a trading calendar see if I should calculate at 12:58 for those half-day trading dates like the day after Thanks giving.
3. Most importantly, I cannot trust any day level back test result in this website.

To demonstrate this problem, I created two back tests:
The first one is a clone of what Florent contributed a week ago @ https://www.quantopian.com/posts/system-based-on-easy-volatility-investing-by-tony-cooper-at-double-digit-numerics
The second one, I changed her code slightly. I ran the test in minute mode. But only calculate once a day @ 15:58pm. This make sure I can use price close to last price to calculate and fulfill the orders within the same day rather than the end of next trade day.
You can see how big the difference could be:

Here is the second test I mentioned:

1250.53% vs -4.7%.
How should I understand the difference?

Hi Alan,

I'm looking for the 12:58 clause in your minutely backtest but can't seem to find it. Do you mind replicating it using schedule_function() and give that a shot? I believe right now it's executing every minute.

There is no logic for 12:58. As I said, I have to maintain a trading calendar if I want to do that. Without this, every year, I lost 3 trading days. But that won't change the performance upside down. Cause from the result, we didn't find any spike on those 3 days each year.
What I added on top of Florent's code is just:

    exchange_time = pandas.Timestamp(get_datetime()).tz_convert('US/Eastern')  
    if exchange_time.hour != 15 or exchange_time.minute != 58:  
        return

Even though handle_data will be called each minute, but given the code above, calculation will be fired only once a day shortly before market close. If anyone want to try Florent's idea, I think that is what he/she will do.
I am not familiar with the schedule_function(). So I don't want to bother. Even if it can allow be to trade 2 minutes before market close for those 3 days, it won't change the comparison above upside down. I would like to focus on the major difference.If you like, I can rerun the test for a period without half-trading day to demonstrate the difference again.

In backtesting, daily mode is used to get a sense of the strategy, to begin developing your idea. Once you have a concept, switch to minute mode to closely simulate the live trading environment. Using minute mode doesn't mean that you trade every minute - only that the algo receives minutely data. You can use schedule_function to schedule specific dates and times for trading. This eliminates the speed issues and need for maintaining the trading calendar - it's already built in.

When you're ready to deploy your algorithm to live trading, you are required to run a full backtest in minute mode to simulate the performance. This way you can directly see how the algorithm behaves. If you want to test your strategy before deploying with real money, you can paper trade it with Quantopian or connect your IB demo account.

Alan,

In this particular case, part of the performance discrepancy between the two backtests you've posted probably has to do with how fetch_csv handles time series data in daily vs minute mode. Basically, running that algorithm in minute mode allows the algorithm to examine the current day's closing VXMT value at the start of the day (which is impossible in real life, unless you're Marty McFly). The minute mode version of that algorithm would need to use post_func to shift the dates up by 1 day so the minute mode algorithm can't see into the future.

Matt,
I totally understand the look ahead issue. That is exactly why I let the calculation happens at 15:58. Even though there is still two minutes look ahead. It shouldn't be that big.
Correct, shift the data by one day avoid the look ahead bias. But that creates a bigger issue. Instead of 2 minutes look ahead bias, that uses one day old data. If you ask me to guess where the price is at 15:58pm today. I would say it is closer to close price today rather than the close price of yesterday.
Do you agree?

Alisa,
I agree with your opinion that minute level back test is way more accurate than the day level. What I want to say is, day level back test could have been much closer to reality if back tester execute the order in more widely accepted ways, either use the close price of current cycle or use open price of next cycle. When you see the last price of a stock, what price do you think your market order would be fulfilled? Without knowledge of BBO and book depth, your best guess is the last price you saw rather than the close price of next cycle, particularly when cycle is long like a day.
If we look at the discussions on this site, most back tests were ran at day level. Algorithms were discussed based on these day level back tests. Like the result I copied from Florent above, everyone is so excited. But it is so far away from the minute level test. That make the day level test result hard to compare.

Alan, we're in complete agreement. In the early iterations of the backtester, daily mode trades and fills orders at the end of day close price. An order submitted on Monday EOD is filled on Tuesday EOD. In hindsight, perhaps an improved simulation would be to submit orders at the end of the day and fill them at the next day's open. As we go forward, I would like to see the daily mode deprecated and use minute mode simulation. This would give more accurate pricing and fills, remove confusion, and align the behavior directly with live trading.

Why not make Daily just simply a subset of Minute that feeds data only at 15:59 (with fill at 16:00)? Thus maybe side-step the industry "Daily" OHLCV concept (rename it to something other than "Daily" if that seems appropriate), and just feed only one minute per day, the idea is to obtain a quick rough idea. Tons of issues evaporate (datetime, extreme slippage et al). Any down-side?

With schedule_function, Minute tests are still slow so that's not a solution, some might not realize this, to be clear, stay with me here, it's due to the fact that all 390 data frames are still being rounded up from the database, packaged, transmitted and the information for all securities fed in calls to handle_data(), 24 Gigabytes--with a G--per backtesting-year of frame content (with set_universe on max at 300 sids), even if only 1 minute of the day is specified in schedule_function. That 1 minute applies only to the new user-specified function that schedule_function was told to run, not handle_data() which still receives everything on all 390 minutes. Maybe the schedule_function func could be changed to be allowed to be handle_data().

Cool. Both Anony and Gary feed in very good point.
I think it really depends on how you want to trade finally.
If you want your calculation based on precisely the close price of day, then it is better to test it with Anony's suggestion, let your order get executed with the open price of next day.
If you want your order fulfilled within the same day to avoid the big price move over night due to earning announcement etc, then go Gary's suggestions, use 15:59 price to calculate and place the order. This way suffered from the look ahead bias mentioned by Matt if your calculation use EOD price of some user provided daily data. But hopefully the real 15:59 price is not far from the close price. Then such test is still a good approximation.
I would suggest Quantopian to make a small enhancement to support Gary's way. Currently using either schedule_function or my simply code snippet can achieve this. But under the hood, a lot of data were processed unnecessarily. That makes the test slow for user and expensive for Quantopian.
If the user choose to run the calculation at 15:55 for his daily test, should we fulfill the order use price at 15:55 or close price of day or price of 15:56? That is open for debate. Each can serve different purpose. It would be nice to provide it as an option. If no option provided, I think using open price of 15:56 minute bar would be the best approximation.

Just read through https://www.quantopian.com/posts/trade-at-the-open-slippage-model started by Anony.
It discussed about using customized SlippageModel to choose different prices to fulfilled the order. Today close, next open and next close etc. That should be able to address the issue I raised. I would strongly suggest Quantopian integrate this as a standard option for the daily test. That would save people a lot of time to duplicate the code. More importantly, avoid people running daily test and fulfill the order with next day close price. As @Andrew Blount mentioned, waiting until the next period's close is not how "real trading" works.

Thing with OHLC bars is that they aren't technically point-in-time. It's a range. You can conceptually break EOD data into multiple points like

* denotes a price event

*Day1.Close  
Post-close->Pre-open  
  Compute stats based on close. Orders can only execute on next day prices.  
*Day2.Open  
Post-open  
  Can do calcs on open prices. Can set limit orders, MOC.  
*Day2.High Day2.Low  
Pre-close  
  Since we don't know time ordering of high/low, you can only send MarkClose orders.  
  If an existing limit order had an attached stop and/or target, you'd always assume the most pessimistic outcome.  
  Example: Limit order was hit, then stop was hit.  
  In reality, this might not be true since Target might have been hit first.  
*Day.Close  
Post-close->Pre-open  
  Once again, we have close. Can compute orders for next open.  

Breaking up the Daily into discrete portions allows a bit more granularity while still only using daily data. With the above breakdown, it's possible to trade the close based on open price of the same day without lookahead bias. If you have minutely data, you can always add another column like price_at_noon and pre_close, which would create additional discrete portions/events.

The backtester could also breakup the day into two timeframes, market-open -> 5-minutes-before-close and 5-minutes-before-close -> close. You'd do this because it's possible you care about the daily range so far, which skipping to just the 15:55 minute bar wouldn't tell you. Precomputing new OHLC data is a lot faster than sending in every minute bar.

I'm not actually sure if the point of daily mode was to quasi-replicate minutely backtests with faster EOD or to test a different time frame.

You sound experienced. I'm a neophyte, do not understand why there could be a look-ahead problem in this case and do not know how Q works at the database level so I have a question.
I gather that along with daily Close prices, there are stored Open, High, Low and Volume numbers for the day and those are all used for daily mode.
So I'm wondering whether the following could make sense:

Use the daily OHL and V pre-stored information for daily tests, not Close though.  
For C, the close price, it is replaced with the minute-close 15:59 price.  
Only that assembled set is fed as 'data' in daily mode (only one frame per day).  
Fills can happen sort of at 16:00.  
The same daily mode fast execution.  
In Quantopian code, it is like minute mode (a lot of the same minute code is active) except it skips 389 out of 390 minutes -ish.

To put it another way, minute OHL and V for 15:59 are replaced by daily mode values.  

The 15:59 price is likely to be closer to the 16:00 price a minute later than a price 24 hours later, resulting in daily tests nudging closer to minute tests.

Any downside?

There's no look ahead there. My concern with the minute-at-15:59 was that it couldn't just be the minute bar. It'd have to be the session up until then, but you might have had that in mind and I misunderstood. I still think you'd need the actual close transmitted to you at some point, depending on how volatile the close is it might result in a new daily high. Some algos might need that info after the fact, even if it's not something you could trade on the day of.