Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Trouble with Trades executing a minute after they should.

I'm working on an algorithm that trades every minute. Since the trades are so frequent, the idea of this algorithm is based on compound interest from small gains. However it has been unsuccessful due in part to my trades not going through until the minute following my buy/sell signals.

I'm using market orders with a feasible amount based on the security's volume to ensure the trades go through. Furthermore, I have done trades of this size for this particular security and can confirm that they happen immediately.

EDIT: I have implemented order_optimal_portfolio for both buy and sell but still have issues with trades happening 1 minute late.

Sample of Logs:

2018-04-30 06:31 PRINT ('49.77', '0.0190', '1.71', '1.95', '1568.29', 14)
2018-04-30 06:32 PRINT ('49.74', '0.0159', '-0.05', '1.88', '-103.18', 15)
2018-04-30 06:33 PRINT ('49.74', '0.0138', '0.00', '1.80', '-107.41', 16)
2018-04-30 06:34 PRINT ('49.48', '0.0074', '-0.52', '1.27', '-13099.48', 17)
2018-04-30 06:34 PRINT Buy Signal
2018-04-30 06:35 PRINT ('49.50', '0.0067', '0.04', '1.31', '-107.73', 18)
2018-04-30 06:36 PRINT ('49.54', '0.0066', '0.09', '1.34', '119.91', 19)
2018-04-30 06:37 PRINT ('49.75', '0.0093', '0.42', '1.88', '367.77', 20)
2018-04-30 06:37 PRINT Sell Signal
2018-04-30 06:38 PRINT ('49.68', '0.0068', '-0.14', '1.82', '-133.84', 21)
2018-04-30 06:38 PRINT Buy Signal
2018-04-30 06:39 PRINT ('49.80', '0.0080', '0.24', '2.02', '-271.67', 22)
2018-04-30 06:40 PRINT ('49.87', '0.0082', '0.14', '2.28', '-40.14', 23)
2018-04-30 06:41 PRINT ('49.88', '0.0072', '0.02', '2.11', '-88.90', 24)
2018-04-30 06:42 PRINT ('49.98', '0.0080', '0.20', '2.44', '1149.80', 25)
2018-04-30 06:43 PRINT ('50.14', '0.0097', '0.32', '2.58', '59.68', 26)
2018-04-30 06:43 PRINT Sell Signal
2018-04-30 06:44 PRINT ('50.07', '0.0073', '-0.13', '2.34', '-141.74', 27)
2018-04-30 06:44 PRINT Buy Signal
2018-04-30 06:45 PRINT ('49.98', '0.0047', '-0.19', '0.43', '38.99', 28)
2018-04-30 06:45 PRINT Buy Signal
2018-04-30 06:46 PRINT ('50.10', '0.0061', '0.24', '0.73', '-229.27', 29)
2018-04-30 06:47 PRINT ('50.20', '0.0070', '0.20', '0.92', '-16.87', 30)
2018-04-30 06:48 PRINT ('50.19', '0.0059', '-0.02', '1.43', '-109.98', 31)
2018-04-30 06:49 PRINT ('50.59', '0.0120', '0.79', '2.19', '-4050.79', 32)
2018-04-30 06:49 PRINT Sell Signal

On the full backtest page under transaction:

6:35 Buy
6:38 Sell
6:39 Buy
6:44 Sell
6:45 Buy
6:46 Buy
6:50 Sell

Please feel free to school me on anything I may be ignorant of

6 responses

order_optimal_portfolio trades what it can that very minute. Freeze is used when applying it to individual positions.

I need it to basically do what an order_value() function does. There are times where I get 2 or 3 buy signals before a sell signal.

How Can I get the Buy Signal to act more similarly to a order_value?

edit: Got it to work with:
if (context.portfolio.positions[context.security].amount) > 0:
context.buy_weight += 0.2
etc...

I got the Sell Signal to work with the order_optimal_portfolio():
context.security = symbol('CRSP')

Create a pandas series or dict with securities and weights to order
weights ={context.security:0}

Create a TargetWeights object to set the order weights
This will become our ordering objective
weight_objective = opt.TargetWeights(weights)

Execute the order_optimal_portfolio method
order_optimal_portfolio(objective = weight_objective, constraints = [])

Still has MOST trades happening a minute late...

The Quantopian backtest engine (ie Zipline) simulates orders as being filled in the ephemeral time just after the close of the current minute and at the very start of the following minute. The order fill logic is run before the handle_data and any scheduled methods. The order fill logic uses data from the previous minute to determine fill price and volume. It's at the very beginning of the following minute and orders are timestamped as of that minute.

Therefore, new positions won't appear in the same minute that orders were placed. However, since orders are filled at the very start of the following minute, new positions are visible to any code run the following minute.

Attached is a backtest which shows this sequence. Look at the logs. They show orders as filled the minute after they are placed. They are filled at the close price of the previous minute. But, since they are filled before any scheduled methods, the positions can be traded again that same minute. Hope this makes sense.

I've looked at your clone and I am trying to understand your explanation. This is one of my first backtest algorithms on this site, so I thank you for your help and patience.

"simulates orders as being filled in the ephemeral time just after the close of the current minute and at the very start of the following minute"

This appears to be the case with my algorithm as the price executed is between the interval of [n+1,n+2], where 'n' is the price at the minute I wish the trade to be executed and 'n+1' and 'n+2' are the prices at the following minutes.

The timestamps are showing me that I am missing the prices by definite intervals every time, causing the algorithm to miss its buy/sell signals. I have thought about attempting to identify those marks the minute before, but that has a much lower probability of success and I believe that the current signals are possible to act on in actual practice.

Is there any order function to file an order at that exact time; or at least much before the close of the current minute?

But, since they are filled before any scheduled methods, the positions can be traded again that same minute. Hope this makes sense.

I see this. The following signals still are firing correctly despite the order missing the previous signals. I am still able to manually record prices to check, within a margin of error, how the algorithm would perform. However, I would love to utilize all of the nifty tools the full backtest provides.

Maybe it would help to have a conceptual overview of the backtester.

Generally the program is called Zipline and it's open sourced at https://github.com/quantopian/zipline . The code one writes in an algorithm is executed in the following order:

  • initialize / this user defined method is executed exactly once when the algo is started
  • before_trading_start / this user defined method is executed every trading day before the markets are open no orders can be entered here.

The following steps are repeated in this order every minute of the trading day beginning at minute 9:31 ET.

  • the orders in the open orders list are executed based upon the slippage model / the first minute there won't be any.
  • handle_data / this user defined method is executed every minute beginning at 9:31
  • scheduled functions / any user defined scheduled functions are executed if they meet the schedule rules

The only two areas which orders can be placed are 'handle_data' and any 'scheduled functions' (or methods called by those). When an order is placed (ie one of the order methods is executed) it really just appends a new order to the open orders list. It doesn't get executed at that time. It just gets created and added to the list. If you check open orders right after an order is placed one can see the new order but the 'filled' field will always be 0.

It isn't until the next time through the loop (ie the next minute) that the open order list is checked and the orders are acted on. The fill price and fill amount is calculated based upon whatever slippage model one specifies in the 'initialize' method (or the default if none is specified). The fill price and volume are always based upon the previous minutes price and volume. So, if it helps, one can think of the order as being filled during the previous minute (the same minute that the order was placed). The fill price and volume are based on that data, and the resulting new positions are available to any code the following minute. However the timestamp in the trades list is the following minute because of the timing as shown above.

One comment you made "where 'n' is the price at the minute I wish the trade to be executed" . Remember that any prices one gets are ALWAYS from the previous minute. The 'current' price is actually the close of the last minute. To avoid any look ahead bias one can't know what the close price (for example) of the current minute is (it hasn't happened yet). One can only see prices which have already happened (ie from the last minute). This may be part of the confusion?

Hope this helps.

Ahhhh I see. That makes sense and is what I was afraid of.

Thank you Dan. I'm back to the drawing board.