Every minute your portfolio value changes consequently 1% of your portfolio also changes. Portfolio value is probably holding cash + closed positions using the bid/ask. If you want a function that performs daily on 1min back-test you can either use the date or schedule a function

I dont think that is correct. From the documentation:

" Portfolio value is calculated as the sum of the positions value and ending cash balance. Orders are always truncated to whole shares, and percentage must be expressed as a decimal (0.50 means 50%)."

So the above example is trying to make SPY 100% of the portfolio. So I would assume this means there would only ever be a small amount of cash (less than the value of 1 SPY) in the portfolio. Now I can see that perhaps not 100% of the order is executed on each call to handle data but why does that lead to many transactions? In the real world, if I put in a BUY order it can be filled using several partial fills and would still only count as one transaction. I am also mystified as to why there are any sell orders since the code seems to ask for 100% in SPY, what scenario leads to it selling anything at all?

I too am confused by this behavior.

The order_target_percent calculates the percentage based on the current portfolio value. You can find the Zipline code here: https://github.com/quantopian/zipline/blob/98ee8efe3db6a5cf3edb6b44b675bdb3276a750b/zipline/algorithm.py#L931

In general, the order_target methods are robust in your algorithm and will scale as your portfolio shrinks/grows. It will seek to the target value by checking the portfolio value each minute.

I added some debug's see what was happening. It seems that cash accumulates in the account at certain times (do we get dividends?) and then extra equity is purchased. To simplify the numbers, I just used $10000 and ran on daily data.

From the log we see cash changing:

2007-10-24handle_data:11DEBUGportfolio value 10760.76 : spy value 10849.95 : spy amount 71.00 : spy price 151.56 : cash 89.19 : leverage 0.99
2007-10-25handle_data:11DEBUGportfolio value 10743.01 : spy value 10832.20 : spy amount 71.00 : spy price 151.31 : cash 89.19 : leverage 0.99
2007-10-26handle_data:11DEBUGportfolio value 10894.95 : spy value 10984.14 : spy amount 71.00 : spy price 153.45 : cash 89.19 : leverage 0.99
2007-10-29handle_data:11DEBUGportfolio value 10934.00 : spy value 11023.19 : spy amount 71.00 : spy price 154.00 : cash 89.19 : leverage 0.99
2007-10-30handle_data:11DEBUGportfolio value 10867.26 : spy value 10956.45 : spy amount 71.00 : spy price 153.06 : cash 89.19 : leverage 0.99
2007-10-31handle_data:11DEBUGportfolio value 10990.80 : spy value 11131.02 : spy amount 71.00 : spy price 154.80 : cash 140.22 : leverage 0.99
2007-11-01handle_data:11DEBUGportfolio value 10705.38 : spy value 10845.60 : spy amount 71.00 : spy price 150.78 : cash 140.22 : leverage 0.99
2007-11-02handle_data:11DEBUGportfolio value 10729.52 : spy value 10869.74 : spy amount 71.00 : spy price 151.12 : cash 140.22 : leverage 0.99
2007-11-05handle_data:11DEBUGportfolio value 10663.49 : spy value 10803.71 : spy amount 71.00 : spy price 150.19 : cash 140.22 : leverage 0.99
2007-11-06handle_data:11DEBUGportfolio value 10798.39 : spy value 10938.61 : spy amount 71.00 : spy price 152.09 : cash 140.22 : leverage 0.99

And then eventually an extra SPY is bought

2008-01-03handle_data:11DEBUGportfolio value 10256.66 : spy value 10396.88 : spy amount 71.00 : spy price 144.46 : cash 140.22 : leverage 0.99
2008-01-04handle_data:11DEBUGportfolio value 10006.03 : spy value 10146.25 : spy amount 71.00 : spy price 140.93 : cash 140.22 : leverage 0.99
2008-01-07handle_data:11DEBUGportfolio value 10043.66 : spy value 10183.88 : spy amount 71.00 : spy price 141.46 : cash 140.22 : leverage 0.99
2008-01-08handle_data:11DEBUGportfolio value 9855.51 : spy value 9995.73 : spy amount 71.00 : spy price 138.81 : cash 140.22 : leverage 0.99
2008-01-09handle_data:11DEBUGportfolio value 10136.16 : spy value 10135.57 : spy amount 72.00 : spy price 140.78 : cash -0.59 : leverage 1.00
2008-01-10handle_data:11DEBUGportfolio value 10209.60 : spy value 10209.01 : spy amount 72.00 : spy price 141.80 : cash -0.59 : leverage 1.00
2008-01-11handle_data:11DEBUGportfolio value 10080.72 : spy value 10080.13 : spy amount 72.00 : spy price 140.01 : cash -0.59 : leverage 1.00

It appears that price < cash triggers a BUY but it is not done at the price listed as it somehow got to negative cash.

In the backtest, dividends are added as cash to your portfolio. Since your code is saying "use all of my cash to buy SPY" it takes the extra cash and reinvests it to buy more SPY shares. You can read more about the handling of dividend and M&A events in the help doc: https://www.quantopian.com/help#ide-dividends

One thing to keep in mind is order_target() methods don't consider the status of open orders when they make their target calculations. They only look at the number of filled orders and then calculate the buy/sells needed to reach the target value. To prevent overbuying/overselling swings, you can add a check to the code to exit if orders are still being filled:

handle_data(context,data):  
  if get_open_orders():  
    return  

Today order_target_percent behave much better than yesterday only 7 transaction instead 244 on first day.
But there left overshot on first day of trading. It ordering 40% more than available cash and sell those positions on next two bars. Why?

2 2007-05-02 - Buys $14,962,812.86, Sells $4,962,919.52 (7 transactions)
6:32 AM
SPY
BUY
34092
$149.80 $5,106,981.60 6:33 AM
SPY
BUY
33116
$149.38 $4,946,934.31 6:33 AM
SPY
BUY
11099
$149.79 $1,662,519.21 6:34 AM
SPY
BUY
21775
$149.06 $3,245,781.50 6:34 AM
SPY
SELL
-11532
$148.57 ($1,713,263.11) 6:35 AM
SPY
SELL
-21819
$148.94 ($3,249,656.40) 6:36 AM
SPY
BUY
4
$149.06 $596.24

The rest orders are dividends reinvestment, but why it happens six week later on last day of the month in Jan, Apr, Jul, Oct?
SPY pays dividends on quarterly expiration Friday in Mar, Jun, Sep, Dec.

Why Q buying 40% more than available cash and sell those positions on next two bars?

Vlad,

1. Your algorithm is using 10,000,000 dollars and trying to buy all that in 1 minute. If an order is too big, Quantopian assumes the order would fill over 2 or 3 minutes.
   
2. order_target_percent() does not check for open orders which have not fully filled yet. So the over-ordering is because of this.

I have added lots of PRINT statements to your example to show the state of open orders:

2007-05-02 PRINT 13:31 : SPY=148.79, volume=395908, SPYshares=0, open position value=0.00  
2007-05-02 PRINT 13:31 : NO OPEN ORDERS  
2007-05-02 PRINT 13:31 : target_value=10000000.00, req_shares=67208  
2007-05-02 PRINT 13:31 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=0,left=67208]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:32 : SPY=148.84, volume=136368, SPYshares=34092, open position value=5074253.28  
2007-05-02 PRINT 13:32 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=34092,left=33116]  
2007-05-02 PRINT 13:32 : target_value=9967263.16, req_shares=32874  
2007-05-02 PRINT 13:32 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=34092,left=33116]  
2007-05-02 PRINT 13:32 : OPEN ORDER 2007-05-02 13:32:00+00:00 [amt=32874,filled=0,left=32874]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:33 : SPY=148.83, volume=176860, SPYshares=78307, open position value=11654430.81  
2007-05-02 PRINT 13:33 : OPEN ORDER 2007-05-02 13:32:00+00:00 [amt=32874,filled=11099,left=21775]  
2007-05-02 PRINT 13:33 : target_value=9937991.59, req_shares=-11532  
2007-05-02 PRINT 13:33 : OPEN ORDER 2007-05-02 13:32:00+00:00 [amt=32874,filled=11099,left=21775]  
2007-05-02 PRINT 13:33 : OPEN ORDER 2007-05-02 13:33:00+00:00 [amt=-11532,filled=0,left=-11532]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:34 : SPY=148.90, volume=233100, SPYshares=88550, open position value=13185095.00  
2007-05-02 PRINT 13:34 : NO OPEN ORDERS  
2007-05-02 PRINT 13:34 : target_value=9936138.75, req_shares=-21819  
2007-05-02 PRINT 13:34 : OPEN ORDER 2007-05-02 13:34:00+00:00 [amt=-21819,filled=0,left=-21819]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:35 : SPY=148.97, volume=1484000, SPYshares=66731, open position value=9940917.07  
2007-05-02 PRINT 13:35 : NO OPEN ORDERS  
2007-05-02 PRINT 13:35 : target_value=9941612.41, req_shares=4  
2007-05-02 PRINT 13:35 : OPEN ORDER 2007-05-02 13:35:00+00:00 [amt=4,filled=0,left=4]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:36 : SPY=149.03, volume=490577, SPYshares=66735, open position value=9945517.05  
2007-05-02 PRINT 13:36 : NO OPEN ORDERS  
2007-05-02 PRINT 13:36 : target_value=9945616.15, req_shares=0  
2007-05-02 PRINT 13:36 : NO OPEN ORDERS  
2007-05-02 PRINT --  

Which account for all 7 transactions.
Hope that is now clear!

You can get round this by using

    # check there are no current open orders  
    if not sid in get_open_orders().keys():  
        order_target_percent(sid, target)  

and you get the following output

2007-05-02 PRINT 13:31 : SPY=148.79, volume=395908, SPYshares=0, open position value=0.00  
2007-05-02 PRINT 13:31 : NO OPEN ORDERS  
2007-05-02 PRINT 13:31 : target_value=10000000.00, req_shares=67208  
2007-05-02 PRINT 13:31 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=0,left=67208]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:32 : SPY=148.84, volume=136368, SPYshares=34092, open position value=5074253.28  
2007-05-02 PRINT 13:32 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=34092,left=33116]  
2007-05-02 PRINT 13:32 : target_value=9967263.16, req_shares=32874  
2007-05-02 PRINT 13:32 : OPEN ORDER 2007-05-02 13:31:00+00:00 [amt=67208,filled=34092,left=33116]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:33 : SPY=148.83, volume=176860, SPYshares=67208, open position value=10002566.64  
2007-05-02 PRINT 13:33 : NO OPEN ORDERS  
2007-05-02 PRINT 13:33 : target_value=9948648.71, req_shares=-362  
2007-05-02 PRINT 13:33 : OPEN ORDER 2007-05-02 13:33:00+00:00 [amt=-362,filled=0,left=-362]  
2007-05-02 PRINT --  
2007-05-02 PRINT 13:34 : SPY=148.90, volume=233100, SPYshares=66846, open position value=9953369.40  
2007-05-02 PRINT 13:34 : NO OPEN ORDERS  
2007-05-02 PRINT 13:34 : target_value=9953342.40, req_shares=0  
2007-05-02 PRINT 13:34 : NO OPEN ORDERS  
2007-05-02 PRINT --  

The elephant in the room; why does that ordering function not take into account open orders? Seems like a no-brainer, and the first step to relieving everyone of having to put the same tedious boilerplate in every algorithm in python.

James,

If order_target_percent() does not check for open orders why not to incorporate

if not sid in get_open_orders().keys(): into the function?

To be correct, they'd probably want to add code that modified outstanding orders to the new target. Of course if they do that, they maybe need to handle the possibility that an old order gets filled before the cancel-and-replace or modify is committed, and so then they need to resubmit a new delta order.

Perhaps they hope to leave all this complexity to algo writers.

To be correct, they'd probably want to add code that modified outstanding orders to the new target. Of course if they do that, they maybe need to handle the possibility that an old order gets filled before the cancel-and-replace or modify is committed, and so then they need to resubmit a new delta order.

Perhaps they hope to leave all this complexity to algo writers.

As Simon points out, its probably not that easy.There is no certainty that the order will ever get executed completely.

It would be nice to see a function which calculates the percentage taking into account what is still left open though.

Well, I would add that it's easier for them, in event-driven C++ or Java, than it is for us, in once-per-minute python! :)

The thing is, anyone ordering 10,000,000 dollars of SPY in a single order is, of course, clinically insane. The way they do this is by building it up in small amounts, based on time (e.g. every 10 minutes plus some random component) and volume (e.g. 3% of overall volume).

Yup, in real life, one would be using an IB-provided smart execution algo, VWAP or otherwise.

Dear Quantopian Support Team,

It looks like Q using x% of overall volume limitation.
But how big is x?

25% of the bar's volume I believe.

Its in the help doc:

Slippage must be defined in the initialize method. It has no effect if defined in handle_data(). If you do not specify a slippage method, slippage defaults to VolumeShareSlippage(volume_limit=0.25, price_impact=0.1).
To set slippage, use the set_slippage method and pass in FixedSlippage, VolumeShareSlippage, or a custom slippage model that you define.

someone more familiar with zipline may explain it in words?

So I can't decipher everything in that code but essentially its:

• if order amount > 25% of this bar's volume, you get a fill for 25% of the bar's volume and the rest is left for next minute.
  
• volume_share = the minimum of [0.25, volume of all your transactions for this stock processed in this minute / this bar's volume]
  
• simulated_impact = volume_share^2 * the 'price_impact' as 0.1 by default * last traded price (this is negative for shorts)
  
• the price you get for this transaction is = last traded price + simulated_impact

so for a huge order (>25% of volume) for a stock trading at $110 with a minutely volume of 2000, then you will get it at...
volume = 0.25*2000 = 500 for first transaction
simulated_impact = 0.25^2 * 0.1 * 110 = 0.6875
so you got 500 volume for price of 110+0.6875 = $110.6875 each

Or at least something like that...

Regarding the problem of open orders & order_target_percent(), one brute-force approach is just to cancel all open orders prior to applying order_target_percent(), although I'm wondering if this could be done confidently in a single call to handle_data. It seems that the right way to do it would be to cancel the open orders, confirm that they are canceled, and then apply order_target_percent() with the new allocation (a minimum of two calls to handle_data). Otherwise, if the order cancellation and order_target_percent() are both called within the same bar, it could be such that more ordering is done than desired (a question for the Quantopian staff)? --Grant

Hello everybody,
This has to be the most common wall people run into. I've explored a few ways to get consistent functionality regardless of pending orders, but everything seems to have unintended consequences, mostly because limit/stop orders throw a wrench in everything. In my mind, the biggest reason for not including open orders is because it's not right for an execution engine to 'assume' it knows how the developer wants to handle pending orders. All the ugliness around partial fills, open orders, rejected orders, etc. are decisions the trader should consider. I am coming at this with live trading in mind, but I'd argue it's good practice to run into these issues while backtesting too.

The attached algo is a re-write of the "order_target" style functions in zipline, I added a couple extra keyword arguments to allow you to either include or cancel any open orders. Including the open orders seems intuitively more correct to me if you only use market orders (faster in backtests too). Canceling is probably the better choice for limits because you most likely missed the market and have a stale limit on the book. For stop orders, it depends on why that stop is there, you may or may not want to include it in your position.

This issue is pretty annoying, but because there's so many possible scenarios and different order types (hopefully more coming), the best solution is to make no assumptions and leave you to deal with the nitty gritty.

Best,
David

Fine idea there, makes for a tidy way to keep things organized. Thanks for sharing that, worth checking out.

Quick side note and pardon me as I don't want to detract from the important subject being discussed, I wish this part could be tiny 6pt font barely readable: 700% percent returns are acceptable, I guess, for a couple of months, although there is an important thing for folks to keep in mind, the code above borrowed $15 million to buy SVXY 1,410 times and never sold, only appears that impressive b/c Quantopian's chart and metrics like Sharpe are based on initial capital and currently (3-2015) ignore margin/borrowing. more info
An easy way to watch out for that is to use this occasionally:

record(cash = context.portfolio.cash)  

Anyway, alright, back to the topic, the examples, extending functions and what they do, and orders. I've been using something a little bit similar, trade() where all buys/sells take place.

Incidentally at some point I'm hoping to see a variation on the typical order_target_percent() algos that could soften the occasional selling at a loss that occurs, somehow.

Today I was writing some code for the tracking of orders that might relate to this and I find it clarifies more than I had expected:
It uses context.orders (as a dictionary with order id's as keys, and values simply as 1 or the number of shares etc, or could be a list instead), storing only the id any time a new order is made, everything else in the order object is thrown away for now. Then in each bar, iterate thru context.orders and use get_order( the id ) to retrieve the order object. This dipsy-doodle is because once filled, the order is otherwise gone and won't show up with get_open_orders(). So, knowing when the order is filled (and to what degree) can be useful in a lot of ways, plus the commissions are there too. (Just wish it had the official fill price for certainty). Once the filled info is collected, that key (id) in context.orders is deleted. That's the general idea I hope can help someone at some point in some way.

I'll just reiterate that I think it's a mistake to casually delegate all this algo writers, who have the distinct handicap that they are working in python, and only get one shot per minute to deal with it, not to mention that tedious event-driven order handling code is not what they came here to do. The promise of a function like order_target_percent is that, upon reading it, the trader thinks, "Thank god this method exists, because I have an asset allocation algorithm I've been working on, and I just want the OMS to work the orders necessary to get me to my target allocation. Now, where are the arguments that specify the aggressiveness and timeframe?". Oh wait, there aren't any, because this method isn't order_target_percent, it's order_immediately_at_market_the_delta_between_the_current_believed_portfolio_and_target_percent.

Honestly, and I feel like I have been repeating myself on this point, providing and owning the smart OMS and risk management components is THE killer feature, in my mind. Disowning this responsibility is a huge turnoff.

Frankly, if people are going to be compelled to each write a real OMS in every Quantopian algorithm, they are sure going to hate doing it in python, once a minute, from a partially reconstructed history of what happened sometime in the last minute. My 2c.

Also, perhaps you should do a poll of how many people actually use order_target_* methods in combination with stops, or even limits. I would bet that 99% don't, stops and limits are for jockeying a single asset, order_target_percent is to track a moving target. In fact, it might be worthwhile to make it clear that this is an OMS-level function, perhaps something like (pardon my terrible python):

oms.order_target_percent(weightedSids=[12345,0.5,345345.0.5], legExecutionAlgo={'Name':'VWAP','Aggressiveness':0.1,'Timeframe','2h'}, keepLegsBalancedToWithin=0.05)

If such an OMS algorithm was running, I'd personally be happy to have the whole thing cancelled the moment my algo made any manual orders, to avoid conflict.

At first I was charmed with the simplicity of the function and when I saw the behaviour in live trading I quickly stopped the algorithm and rewrote the order function myself. One flaw is that it looks at cash.... The order function should not read the cash balance but the settled cash. Even settled cash needs to have a cushion as it does not reflect orders already in play so at the moment I use the following to rebalance: the settled cash minus value of current orders*1.01 minus the estimated transaction cost. Then I execute every minute one leg if the previous leg was executed where I readjust the limit price in case it moved. This is at the moment in experiment mode and I'll post it when it works properly...

I need to build the function such that it can handle cash accounts... First execute the sell parts of the rebalance and then during the next 3 days iteratively execute the buy parts as it takes up to 3 days to get the sale reflected in Settled Cash. Big disadvantage is off course that the price will move in 3 days and that the rebalancing will be imperfect and could even be damn dangerous when you are hedging legs against each other and the hedge starts moving. Because of this delay I'll adjust the # of shares to the price movement so the value of the legs are defined.

A python guru can make a class structure out of it as I'm just a simple algo builder.

Did anyone make something like this already?

There's a potential problem with this check:

handle_data(context,data):  
  if get_open_orders():  
    return  

Presently, under live real-money trading (and paper trading at IB, as well, I think), Q automatically cancels all open orders at the market close. However, if this practice is ever discontinued, then if an order get botched up and hangs around at IB, you'll never be able to order the security again. So, it would seem to be a good practice to also do a periodic sweep of open orders, cancelling any potential stragglers.

Grant, you should definitely periodically sweep through an cancel open orders, you should really do it at the end of each day since that's what happens in live trading, schedule_function makes that pretty easy to do. It really shouldn't make a difference if Q stops canceling orders at EOD, IB's default "time-in-force" is a DAY order, so they would still get cancelled by IB even if Q didn't explicitly do it.

An issue I see is almost the opposite scenario where backtests are inaccurate because limit orders can fill several days later in backtests when they would have been cancelled at EOD in live trading.

Peter,

did you end up getting anywhere with that function? Just wondering... would be good to look at.