Hi Will,

There are a number of newly added order methods that you might want to take a look at here. To close out an existing long position you can call either of the following:

order_target_percent(context.aapl, 0)

order_target_value(context.aapl,0)

Either of which will close out your position in context.appl and will save you the steps of checking how many shares you currently hold in order to place a direct share count sell order. These methods are really convenient for rebalancing portfolios as well

Best,
Jess

Hello Jess,

If there are pending/open orders, how will order_target_percent and order_target_value work? I'm wondering if the best practice would be to always check for open orders, prior to submitting additional orders to change the position based on the current position. Or do order_target_percent and order_target_value look for open/pending orders and compensate accordingly?

The code looks something like this:

for stock in context.stocks:  
    if not bool(get_open_orders(stock)):  
        # stock order processing here  

Grant

Hello Jess,

I'm wondering how the Quantopian/Interactive Brokers interface handles calls to order_target_percent and order_target_value. For example, for order_target_percent(stock, 0), once the target percent is achieved, will Quantopian continue to send orders for zero shares to IB, if it is called in the algorithm? If IB receives such zero share orders, is there any consequence, such as commission charges? Thousands of zero share lines on a trading report?

Grant

Hi Grant,

The live algo I'm running for QA in a Quantopian real money account is actually exactly the one Dan posted in the open beta announcement. It uses order_target_percent to rebalance exposures in each of 9 sector ETFs. The way it works in practice is that it only sends an order to IB when the number of shares is non-zero. You can actually see this in this screen shot of my live algo dashboard. This particular algo went live on 1/24, when it allocated $27k and change across those 9 ETFs, subsequently every trading day it will trade whole numbers of shares to keep the exposure to each stock at the target level. Notice that there are only transactions for a subset of the ETFs each day, and there were no non-zero transactions made on 1/29 or 1/30. Then again yesterday on 1/31 we bought one share of XLP and sold one share of XLU.

As a side note, this algo is not something I'd advise anyone to deploy to real money without modification - there is no reason (other than burn in for us) to be rebalancing this type of strategy every day. This is a long only strategy that will track the SPY but cost you as much as $9/day for the privilege.

Some easy modifications would be to adjust the sector weights based on a fundamental market view (e.g. I want to be overweight tech and underweight consumer discretionary) or to add logic to take advantage of sector momentum (e.g. overweight sectors with strong trailing 30, 60 or 90 day returns). But even as is - I think its a great algo for testing in Paper mode (on Q or IB) for folks that want a working minute-mode algo to learn from - so I'm attaching it here again for that purpose.

Thanks...Grant

Thanks Anony,

You are not the first to remind those of us with zero active trading experience that there is a whole set of practical complications not captured in backtesting. One concern I have is that a significant Python coding effort is required to have a robust system. It is pretty easy to have a backtest "go off the rails" or crash due to a data problem or bug, so adding real-market interactions is bound to be error-prone. But then, I don't have any experience...

Regarding your comment about the trade costs, and whatever Quantopian will charge per month per algorithm, to me they seem way too high based on the fact that we are talking about fully automated systems both at Quantopian and Interactive Brokers. They haven't lowered the barrier to start out with ~$1000...much more is required, which is beyond the reach of the masses.

Grant

Hello Grant,

It's been mentioned before but the lowest cost to entry for retail algorithmic traders will be Forex which is, of course, a potential minefield of bucket-shop brokers hunting for stops and actively trading against the retail trader. I haven't gone that route yet but there is a Swiss Forex broker that I am considering who seem to be about as legitimate as they can be who are an ECN broker i.e. they do not have their own dealing desk working against you. An account can be opened for around $100, I believe, and they have an automated trading platform with no monthly charges that I am aware of so far. By going through the right Introducing Broker it is possible to get 40% of trade commissions reimbursed as well. What puts me off is that strategies are coded in Java.

P.

Well, the hope is, for me, that Quantopian will evolve into a platform that really does take care of all the tedious nightmare code revolving around robust OMS, fail-fast algorithms which can easily get back to their prior state on restart, and the risk management and portfolio optimization of a bundle of strategies operating on different universes simultaneously.

I am not really sure where it's all going though. The target market seems to be people with over $50k+ set aside for active automated trading, who also have the significant statistics and programming background to implement strategies in code, but who don't want to build their own cloud-based platform with which to trade. They seem to be positioning somewhere between Wealthfront/Betterment and Deltix etc.

Question from anony:

Does the Q maintain its own position/portfolio records during live trading or have you built your IB adapter such that you query IB's account callbacks in order to mark to market account balances? Either M2M at the daily rollover or on a continuous basis if running minutely? Said another way, in order to arrive at a value of a portfolio based on a backtest (like the above image) yet shift into actual monies when true trading has superseded the simulator, do you pull in IB numbers and abandon Q's non-system of record position totals, or do you do the opposite; merely ship orders into IB without regard to IBs system or record status keeping Q's representation of a position as the expected position value?

When you launch your algorithm using your IB account, IB becomes the system of record for your cash and portfolio (this is true for both paper and real money - they are identical in the eyes of Quantopian systems). On the back end, Quantopian is regularly getting position and cash updates from IB. If for some reason Quantopian's positions diverge from IB's, then IB's answer is used. When that happens, we push a system event to the algorithm which you can see in the "logs" tab of the live algorithm dashboard.

As a practical matter, we see this type of thing in our current live trading:

• You start your algorithm with no positions, but IB has open positions. On algo start, Quantopian picks up the true IB position.
  
• The cash balance is queried on start.
  
• IB commissions are sometimes calculated asynchronously, and you'll see a cash update in Quantopian when the commissions come in late.
  
• You make a deposit or withdrawal to your IB account, and the cash position unexpectedly updates

In general, our goal has been to make sure the algorithm has the best information it can about the situation - price, value, positions, order status, etc. The algorithm needs to make use of that information. If the algo blindly orders 100 shares over and over without evaluating the position and open orders, you're in trouble. If the algo makes assumptions about the amount of cash you have, and doesn't check the portfolio, you're in trouble.

I do think that a combination of historical backtests and walk-forward paper trading will enable you to build an algorithm that handles all of these situations. We are trying to make it easy to write a good algorithm, but we aren't (yet? ever?) able to abstract away all of the challenges.

Thank you for that reassuring clarification, Dan. Just so it's crystal clear from the coding perspective, can we expect all the data in the Portfolio object and its Position objects to be in sync with the IB data on every invocation of handle_data?

...can we expect all the data in the Portfolio object and its Position objects to be in sync with the IB data on every invocation of handle_data?

Short answer: Yes.

Longer answer: IB sends updates to us asynchronously as the contents of your portfolio and your cash on hand change, and we process those updates immediately when we receive them. Therefore, each time handle_data is called, the Portfolio and Position objects will be as current as the data that IB has sent us.

Hello Jonathan,

My understanding is that although you receive the IB updates asynchronously, the state of affairs at IB is only updated effectively just prior to the call to handle_data, right? In other words, real-time intra-minute info. from IB is unavailable to the algo.

Grant

Anony's description is essentially accurate.

Thanks Anony and Jonathan,

Another consideration is algorithm time-out. My understanding is that a call to handle_data can last up to 50 seconds (although I haven't been able to replicate this in the backtester--see https://www.quantopian.com/posts/algo-time-out), after which the algo will crash. So, there is no need to be frantic--computationally intensive analysis can be performed (assuming that a memory limit is not exceeded, which is another constraint). Jonanthan, is this correct?

A diagram (swim lanes?) would definitely help.

Gotta run, but the topic of the Nanex-supplied last_sale_price is new to me. Hmm?

Grant

Anony - thanks for the helpful description a couple replies above. The only thing there that I think isn't quite right is that the last_sales_price is not continuously updated - it is essentially the same thing as the close price, but on portfolio rather than data.

Anony - The minute is being driven by Nanex's NxCore product, our datastream. Their data product is built such that they can guarantee a continuous stream of data with data that is timestamped by their servers and delivered to us in a predictable, consistent order. Basically, they are handling a lot of the data problems for us, and we just have to make sure we're getting a clean stream from them before we package it into bars.

As for the behavior when a new minute bar becomes available; We don't have a specific heuristic that decides which algo processes first. All of hundreds of live algos are distributed across dozens of servers. On a given server, each algo has its own thread. The OS of the server decides which thread to run, and I couldn't even tell you how it chooses. We watch the servers and server load and make sure nothing is CPU starved. I'm sure our implementation of this detail will change as we get larger in scale.

As for the order handling: your estimation of order handling is correct. How the messaging gets handled is reasonably complex in execution, but simply described in outcome: your algo always has the best possible information about the current state of affairs. For instance, if you place an order, it is filled in part, and then you cancel that order, your algo knows that the partial fill has happened, and it knows that the order is still opened. That order will be listed as "open" until either a) it is complete filled or b) the cancel is confirmed by IB. As you know, orders fail for many other reasons too - a short sale may fail because the broker has none to loan; insufficient funds; insufficient margins; etc. In all of these cases the order is reported back to the algorithm as "canceled."

So, how do you handle all that uncertainty? You have to be smart about your algorithm coding. Imagine this truly naive algorithm:

1. Minute 1: Submit order for 100 shares of Apple
   
2. Minute 2: Cancel order from minute 1
   
3. Minute 3: Submit order for 100 shares of Apple
   
4. Be astonished to find out you actually own 200 shares of Apple because order 1 was filled before cancel

This is what you should write instead:

1. Minute 1: do an order_target() for 100 shares
   
2. Minute 2: Cancel order from minute 1
   
3. Minute 3: do an order_target() for 100 shares
   
4. Note that you have 100 shares of Apple, exactly as you hoped/expected you would.

By doing that type of coding, where you check your state before attempting to change state rather than assuming you know the existing state, you can let Quantopian's back end deal with all of the gnarliness of order rejection, partial fills, etc.

Grant: you are correct that if your algo takes too long (50 seconds) it will get shut down. I agree there is no need to be frantic. It's just that if your algo spends 20 seconds computing, then it will be 20 seconds before your orders are issued.

I have been thinking about a way to diagram the whole thing - it's on my to-do list somewhere.

Thanks Dan,

Regarding the 50 second time-out, is there any way to check for the time remaining before the algorithm will be shut down (e.g. time_remaining())? Also, I tried to replicate the 50 second limit in the backtester, but arrived at at time out of ~ 4 seconds, I recall. Is the time-out different for the backtester?

Similarly, is there a memory limit, and a way to check how much memory has been used?

I don't have use cases for the time-out and memory limits--just curious at this point, but maybe someone will figure out an advantage to using as much of the time and memory as are available.

Grant

Grant,

Regarding the 50 second time-out, is there any way to check for the time remaining before the algorithm will be shut down (e.g. time_remaining())?

You can store the return value of time.time() at the start of your algorithm and subtract the current value of time.time() from that stored value at any point in your algorithm to find out how much time has elapsed.

Also, I tried to replicate the 50 second limit in the backtester, but arrived at at time out of ~ 4 seconds, I recall. Is the time-out different for the backtester?

We validate your algorithm code before we actually run a backtest, and there is a 5-second handle_data timeout, rather than 50 seconds, during validation. We only call handle_data once during validation, so if you don't sleep in your algorithm during the first invocation of handle_data, you can get past the 5-second validation timeout and make it to actually running the backtest. The attached backtest demonstrates this. The last time it logged before raising an exception for spending too much time in handle_data was "INFO 49.8 seconds elapsed".

Thanks Jonathan...Grant

Hello Jonathan,

I cloned your algorithm immediately above, but upon attempting to run it, got the error:

The module/attribute 'time.sleep' is not available.
File test_algorithm_sycheck.py:18, in handle_data
File safety.py:165, in getattribute
File safety.py:129, in check_attribute_access

Is time.sleep no longer available?

Grant

time.sleep is no longer available because it disrupts the timing of 1 minute bars and won't pass in validation. We don't allow you to speed up or slow down the data bars in backtesting or live trading. The algo data frequency is 1 minute or 1 day for backtests.

Hello Alisa,

Here's a work-around. Got it up to ~48 secs. per call to handle_data.

Grant

Just for yucks, here's another approach:

import time

def initialize(context):  
    context.sid = sid(24)  
    context.first = True  
def handle_data(context, data):  
    start = time.time()  
    if context.first:  
        context.first = False  
        return  
    dt = time.time() - start  
    while dt < 30:  
        dt = time.time() - start  
        pass  
    dt = time.time() - start  
    log.info('%.1f seconds elapsed' % dt)