I would like to use Quantopian to backtest "real world" portfolio gains. In particular, I'd like to deal with sources of loss that Quantopian currently ignores. The biggest is taxes. Another is margin loan interest.

I'm writing this note to list some of the difficulties I've found so far, and to share the techniques I've used to get around some of them, while asking for advice on some of the other issues (or improvements on some of the hacky things I've done in my "solutions").

To properly track these costs, it seems like you need three things.

1. The ability to "see" taxable events (distributions, capital gains, margin loan size).
   
2. The ability to compute tax effects. (Tax rates, understanding of short and long term gains cancel out and carry over from year to year, etc.)
   
3. The ability to enact those effects on the portfolio.

I don't think Quantopian should deal with #2 at all. It would be cool to see a library of code that helps out there (maybe it knows tax brackets, cutoffs between them, state tax rules, etc). Maybe someone knows of one? But that is almost certainly not the job of the backtesting system itself.

I think #3 ought to be easy. Quantopian should have an API call like

context.adjust_portfolio(amount)  

that simply adds or removes cash from a portfolio. I'm mostly concerned with costs, so for me 'amount' would be negative here, but it would be positive for people who want to look at portfolios with periodic contributions.

Since that API doesn't exist, it seems that we have two options.

1. Use the commission/slippage model to pull money out of the portfolio as needed.
   
2. Keep track of taxes paid over time, and force your algo to trade as if the money didn't exist.

The commission/slippage technique has the problem of ruining basis calculations since the money we try to "hide" this way gets added to a buy's basis as a trading cost, and that screws up the capital gains calculation later. The tracking technique is annoying because it ruins all sorts of calculations that Quantopian is trying to do for you. You can no longer use calls like order_target_percentage(), or even ask for your leverage, because Quantopian thinks you have more cash than you really do.

With that said, tracking tax paid and keeping it out of trades is currently the best I can think of. A hybrid hack that might work: Remove the cash with certain buys of a particular security, and have your commission model ONLY withdraw the money when you buy that security. That would remove the money, but leave your basis in "real" transactions alone. You could buy one share of the special asset, have the commission be your current tax liability, and then immediately sell it, with zero commission. Anybody try that? I'll followup with an attempt if anyone shows interest.

I'd really prefer context.adjust_portfolio().

Now, onto the #1 need: The ability to "see" taxable events. First, there's finding dividends. Ideally, I think Quantopian should support
a callback that is invoked once for each dividend payout. It could be rolled into schedule_function, like:

schedule_function(track_dividend,  
                  event_rules.every_dividend())  

track_dividend() would get called with the asset, amount paid, and context. I suppose it would get called at close of the payout date? Or at open of the next day?

Currently, I'm using the following hack, which does not figure out which asset caused the dividend, but it does seem to track dividends properly. It just notices the sudden jump in cash from market close one day, to market open the following day.

def initialize(context):  
    schedule_function(begin_day,  
                      date_rules.every_day(),  
                      time_rules.market_open())  
    schedule_function(end_day,  
                      date_rules.every_day(),  
                      time_rules.market_close())  
    # call end_day once, to initialize  
    end_day(context, None)  
def begin_day(context, data):  
    dividend = context.portfolio.cash - context.last_cash  
    if dividend < -0.005:  
        log.warn("Negative dividend!? "+str(dividend))  
    if dividend > 0.005:  
        log.info("Dividend of "+str(dividend))  
        context.tax_paid += dividend * context.dividend_tax_rate  
    # TODO, we should be able to know which asset had the dividend by  
    # noticing the change in reported price of yesterday's open.

def end_day(context, data):  
    context.last_cash = context.portfolio.cash  

It could be enhanced to store closing prices of every holding. Then in the morning, check yesterday's price to see if it's being reported the same. If not, that asset caused the dividend. (This should work because I've read that Quantopian "lies" about historical prices in order to "trick" algos into doing the right thing: not treating a dividend price change as volatility.) It should even be possible to attribute dividends to the right asset when two assets have a dividend the same night, by checking the size of the "lie" about yesterday's price.

Again, I'd sure prefer an explicit callback. Quantopian is already doing the heavy lifting of computing dividends and rejiggering prices. Just tell my algo, please.

That's dividends, now we need to track gains. That should be easy: just compare the selling price to the purchase basis. That works (though it wouldn't work if we use the simplistic commision based tax payment trick, above). But position objects don't report a date, so it's not possible to track long vs short term gains. In a related problem, a position object combines multiple buys in one position. (Maybe that's why it doesn't have a date: what's the date if you bought IBM on two different days?) Anyway, I would love to see an expansion of the context.portfolio.positions idea, to context.portfolio.lots. It would also be indexed by asset, but would contain a list of Lot objects, rather than a single Position. Each Lot object would be similar to a position, but it would have a date. Multiple lots would also be used when an order is filled in multiple pieces. The commission would be split across the lots.

Adding a Lot abstraction could complicate ordering. You might want to specify which Lots are sold. Alternatively, to keep things simple, we might start with a FIFO or LIFO scheme. Eventually, maybe the order() can could take Lot IDs?

My current alternative is to try to build that lot table myself, by processing each filled order. I think this is something that belongs in the backtester though. In the meantime, I'll get started on that, and share the code if anyone is interested. Even to do this, a new hook would be nice:

schedule_function(track_order, event_rules.every_fill())  

I'm not exactly sure how to see every part of an order fill without it.

Finally, we come to margin interest. I haven't thought much about that, but I think it's easy enough, as long as none of the hacks we use for other purposes confuse the leverage calculation. I guess we can just calculate the interest owed daily based on the portfolio's leverage, and "pay it" by removing the cash from the portfolio in the same way we come up with for "paying" taxes.

I'd welcome any ideas for better short-term solutions for these problems, or better API proposals for "real" fixes.