To bump and slightly restate the question - when does daily mode trade in the day? Does it trade at the close, and if so, how does it factor slippage? Does the order roll over into after-hours or the next mornings open?

Hey Param,

Trades in backtesting mode are filled, at the fastest, the next bar. So in this case (daily mode) of creating an order on, let's say, May 6th, it'll get filled at the close price of May 7th. This might be why you're experiencing different results from daily versus minute mode since you're getting a difference of a full day in price in daily mode versus a difference of a few minutes in minutely mode. As for slippage, there's a default slippage model applied to your order which uses the VolumeShareSlippage model.

And when you use 'if (exchange_minute == 390)', the order gets carried over till the next morning because of the next-bar-execution behavior of the backtester so using '380' will get you something closer to what you're looking for. Alisa did an amazing job of addressing these issues on this topic (https://www.quantopian.com/posts/why-cancel-orders-at-end-of-day) so if you'd like to find out more, feel free to check out the link.

Let me know if you have any more questions.

Seong

Thanks so much for the reply, Seong.

I don't understand why the daily mode wouldn't trade the same day as the signal. It seems like this one-day delay would be immensely problematic for most algorithms. I find it almost impossible to experiment in minute mode as backtesting is painfully slow (several hours for a 10 year backtest), and is extremely redundant given that I only want to trade at once-daily frequency. Daily mode is typically around 3-5 minutes.

Are you guys working toward a way to execute trades the same day in daily mode? Is there a workaround in the meantime that would allow me to use the close price from the day of the signal? It just feels extremely inefficient to have to test an at-the-close strategy on every minute-bar of the day.

Hello Param,

Anony Mole posted a solution to this problem, https://www.quantopian.com/posts/trade-at-the-open-slippage-model. He takes advantage of the documented feature of applying a custom slippage model (https://www.quantopian.com/help#sample-custom-slippage). I've attached an example.

There might be a way to allow you "to use the close price from the day of the signal" when running an algo on daily bars. I'll see if I can sort it out. It would amount to shifting the 'trade_bar' in the process_order method back one bar (i.e. the day the order was submitted). One then has to be cautious of look-ahead bias, since when handle_data is called your algo will have access to the closing price which will be used as the execution price.

Grant

Hi Param,

Please disregard my post above, since when I look at the transaction details for the algo, it appears that Anony's custom slippage model is not working (or I have not applied it correctly).

Grant

Param,

Here's the fixed version. I had to add:

set_commission(commission.PerTrade(cost=0.0))  

Otherwise, there is a 0.03 commission difference that shows up in the Transaction Details report. For the attached backtest, you can see directly that Anony's model works.

Grant

Hi Param,

The attached code should do what you want. You should be able to run on daily bars, and buy at the closing price, thus simulating a MOC order. Note that following Anony's example of a simple linear model for the price variation over the day, you can buy close to the close (by setting fractionOfOpenCloseRange=0.1). Let me know if you find bugs, or if it doesn't meet your needs. Again, please be aware that there is the potential for look-ahead bias. Also, from the standpoint of good coding practice, my use of globals may pose a risk. Perhaps some Python gurus out there know how to do a better coding job.

Here's the log output, which you can compare to the Transaction Details tab (after running the algo yourself):

2014-10-01PRINTSPY prior open/close: 196.7/194.34  
2014-10-01PRINTBND prior open/close: 81.97/82.12  
2014-10-02process_order:52INFO Order:SPY open:196.7 close:194.34 exec:194.34 side:1.0  
2014-10-02process_order:52INFO Order:BND open:81.97 close:82.12 exec:82.12 side:1.0  
2014-10-02PRINTSPY prior open/close: 194.18/194.39  
2014-10-02PRINTBND prior open/close: 82.18/82.05  
2014-10-03process_order:52INFO Order:SPY open:194.18 close:194.39 exec:194.39 side:1.0  
2014-10-03process_order:52INFO Order:BND open:82.18 close:82.05 exec:82.05 side:1.0  
2014-10-03PRINTSPY prior open/close: 195.68/196.53  
2014-10-03PRINTBND prior open/close: 81.97/82.03  

You can see that the orders are executing as MOC with daily bars, and not using the next day's close for execution, which is the standard slippage.

Grant

Grant this is awesome

Thanks so much, Grant! This is an amazingly elegant solution and will certainly save me a lot of time!

You're welcome. If you have more questions or insights, just post here. --Grant

@Grant
Whew :O :D :)

@Param, @Fawce
Param said: "I don't understand why the daily mode wouldn't trade the same day as the signal"
Agreed. Long ago transactions were confirmed while still on the phone, today websites fill/confirm right away, not based on the price 24 hours later.
Daily mode fill next day is unreal.

Requesting Quantopian provide us with a switch to fill orders in Daily mode right away.
Couple of options maybe:
1. Reproduce what Grant has shown here with one line we invoke, or
2. The switch makes Daily price the 3:59 prices rather than close and fill at 4:00 same day.

Param said: "I find it almost impossible to experiment in minute mode as backtesting is painfully slow"
You can say that again.
My algo takes 45 seconds in Daily mode and about 3 hours in Minute mode (2009-01-01 to present).
Edit: Around 20 minutes with the exchange_minute code above.

Requesting Quantopian consider providing us with a switch to feed to handle_data() in Minute mode only the minutes a user specifies.
The option is for those who are just going to be discarding the other minutes' data.
Pulling them from the data provider, tasking the processors and transmitting them no more will save energy and time.
Roughly 1/390th the effort, around 1/4 of 1 percent.
Please make the consideration of that a priority, ask the appropriate person whether it is technically feasible.

Another twist here is that one needs to consider the definition of an OHLC daily bar. If I understand correctly, Quantopian (or their vendor) is doing this:

O: the price of the first trade of the day, regardless of when it occurs
H: the highest price of the day
L: the lowest price of the day
C: the last price of the day, regardless of when it occurs

So, for thinly traded securities, there is another caveat for the code I posted above, since the close for a given security may not occur at the market close. So, even if the algo uses opening prices, and you are assuming MOC orders placed shortly after the open, in reality, they may not get filled the same day. This mistake could be avoided if datetime stamps were provided for the individual OHLC values. The other approach would be to run a diagnostic algo on minute bars to check the trading frequencies and volumes of the relevant securities, to see that the MOC orders are realistic.

A further extension (/abuse) of the slippage model would be to figure out how to place multiple orders in the same day but at different effective times (e.g. close to the open & close to the close), using daily bars (straightforward, I think). This probably is not the best approach, but (painfully) calling handle_data 390 times a day may be overkill for the type of trading that is of interest to some folks here, where the capability to detect small minute-to-minute differences is irrelevant. The relevant timescales are days, weeks, and months for many, I suspect.

Grant

Hi,

I know it's a long time, but is the daily execution issue (order submitted today but actually executed next day) finally corrected yet in Quantopian and Zipline? or is there any plan at all to correct it?

Thanks,
ya