Hi Dennis,

I took a quick look at Zipline (https://github.com/quantopian/zipline/blob/master/zipline/finance/performance.py) and found:

 def __core_dict(self):  
        rval = {  
            'ending_value': self.ending_value,  
            # this field is renamed to capital_used for backward  
            # compatibility.  
            'capital_used': self.period_cash_flow,  
            'starting_value': self.starting_value,  
            'starting_cash': self.starting_cash,  
            'ending_cash': self.ending_cash,  
            'portfolio_value': self.ending_cash + self.ending_value,  
            'cumulative_capital_used': self.cumulative_capital_used,  
            'max_capital_used': self.max_capital_used,  
            'max_leverage': self.max_leverage,  
            'pnl': self.pnl,  
            'returns': self.returns,  
            'period_open': self.period_open,  
            'period_close': self.period_close  
        }  

Grant

Thanks Grant. I had tried to find 'capital_used' with a Google search but it didn't locate this file.

I think the portfolio object is set in its own function. I wasn't able to use the properties from core_dict in my algorithm (e.g. max_leverage).

def as_portfolio(self):  
        """  
        The purpose of this method is to provide a portfolio  
        object to algorithms running inside the same trading  
        client. The data needed is captured raw in a  
        PerformancePeriod, and in this method we rename some  
        fields for usability and remove extraneous fields.  
        """  
        # Recycles containing objects' Portfolio object  
        # which is used for returning values.  
        # as_portfolio is called in an inner loop,  
        # so repeated object creation becomes too expensive  
        portfolio = self._portfolio_store  
        # maintaining the old name for the portfolio field for  
        # backward compatibility  
        portfolio.capital_used = self.period_cash_flow  
        portfolio.starting_cash = self.starting_cash  
        portfolio.portfolio_value = self.ending_cash + self.ending_value  
        portfolio.pnl = self.pnl  
        portfolio.returns = self.returns  
        portfolio.cash = self.ending_cash  
        portfolio.start_date = self.period_open  
        portfolio.positions = self.get_positions()  
        portfolio.positions_value = self.ending_value  
        return portfolio  

The code also reveals the calculations for capital_used (period_cash_flow) and cash (ending_cash).

I found the code for context.portfolio.capital_used (aka period_cash_flow). It is adjusted by subtracting the net sum of transaction dollars. So that would cause short orders (negative dollars) to mask long orders. And it's also why it is overall negative instead of positive in long positions. This matches my hypothetical q_cap_used equation in my backtest.

    self.period_cash_flow = 0.0               # initialized to zero  
    self.period_cash_flow += cash_payments    # dividends  
    # subtract transaction spot price x number of shares  
    self.period_cash_flow += -1 * txn.price * txn.amount  

I also found the code for context.portfolio.cash (aka ending_cash). As expected it is calculated from period_cash_flow:

    self.ending_cash = starting_cash   # initialized to starting_cash  
    # sum of starting_cash and 'capital_used' (aka period_cash_flow)  
    self.ending_cash = self.starting_cash + self.period_cash_flow  

There doesn't seem to be a zipline calculation for absolute capital used and remaining cash. So I guess I will just have to keep using my home-brew solution:

    # calculate capital_used using absolute amount: abs(amount)*cost_basis  
    pos = context.portfolio.positions  
    abs_capital_used = sum(abs(pos[s].amount) * pos[s].cost_basis for s in pos)  
    record(abs_cap_used = abs_capital_used)  
    # calculate free cash: starting_cash + pnl - capital_used  
    port = context.portfolio  
    abs_cash = port.starting_cash + port.pnl - abs_capital_used  
    record(abs_cash = abs_cash)  

@Quantopian: Can you address this in zipline?

I've been bitten by this on essentially every algorithm I have designed or considered, it should be something that I can set as a startup parameter and then not have to worry about. Something similar to the max_notional and min_notional that you recommend in your algorithms, but enforced by the backtester itself versus reinventing the wheel every. single. time.

Ryan, do you think Grant's homebrew solution is the right one? Or is there a different, similar calculation?

Once we have the useful calculations identified, figuring out how to do limits can be next.

Hi Dan,

I think it was Dennis C who posted a "homebrew solution," not me.

Grant

It looks correct, but the other part of the problem is that I don't know what it's supposed to look like. I'm a n00b at all of this, I can understand the idea of notionals and limits, but I can't seem to make them work when it comes to application.

I've done the equivalent of these calculations before, but the numbers never added up. If I update "cash on hand" like this, the values don't match the zipline cash-used values due to slippage / commission / other:

self.cash_on_hand = self.cash_on_hand - (data[stock].price * order_size)  

This is the idealized equation if I'm pursuing a long-only strategy (it scares me to bet money I don't have), but it leaves out the other bits. I can't even get the numbers to balance between cash_on_hand and some summation of portfolio.* or a combination thereof. If I want to add shorting into the mix, it gets even worse. Multiplying by cost_basis seems like a good idea, but again, I don't know if that's correct.

My ideal situation would be:
- Set the initial cash-on-hand to a fixed value, defaulting to portfolio size.
- Make that number available in the init function so other vars can be set relative to it (i.e. short_limit = start_value * 0.5).
- Set the short limit to a fixed value, defaulting to zero.
- Have zipline process transactions as it does now, but tracking both long positions and shorts, separately.
- Reject transactions that would violate either limit (OVER_MAX_LONG, OVER_MAX_SHORT).

Without having these kinds of limits enforced in the engine, I don't see any point to specifying an initial investment amount. That only changes where you put the zero-mark, it doesn't change the behavior of the backtester in any way. With these limits in place, things like credit, dividends, commissions, slippage, etc. could all just affect the cash_on_hand value, and the engine would respond appropriately.

Want to short something? Check your credit (MAX_SHORT), and if you have "cash" put through the transaction.
Get a big dividend? cash_on_hand++
Long on 10 stocks and short on 6? The numbers still add up (and the limits still work!) because you're under MAX_LONG and over MAX_SHORT.

Making this happen I leave as an exercise to the reader - I certainly don't know how to do it, or I would've gotten at least one of my algorithms to reconcile. =)

@Dan and Ryan. My understanding is also limited as I'm new to this as well.

I posted a rather detailed explanation of how I believe margin works in another thread. That is based on Reg-T Margin which is going to be the case 99% of the time. See the margin description at IB.

The other kind of margin is Portfolio Margin which has a minimum account requirement of $110k and very different rules (certain baskets of securities are considered risk balanced and available margin goes way up).

Note that IB has additional margin available for same-day positions (details vary depending whether it is long or short and stock price). But the end-of-day requirement is always the same: 50% margin for Reg-T accounts. So I just use 50% margin as a simple, easy to understand default. (There are some exceptions: leveraged ETFs have less margin available).

One nice thing about having extra margin for same-day positions is that we don't have to worry as much about the order of buys and sells. Meaning that your algorithm can issue a sell order and a buy order at the same time and not care if the buy order executes first (unless the orders are huge). That is to say if you are aiming to maintain a 50% margin level it's okay if you accidentally rely on another 15% of margin before it is returned via the accompanying sell order. This greatly simplifies the logic in portfolio rebalancing strategies for instance.

Because of all of this I think any backtester imposed limits should be configurable (just like we do with commission and slippage model). But I think the default model should be an end-of-day 50% margin limit.

Even without enforced limits we need accurate variables for capital used and available cash. From that we can calculate buying power and margin levels.

My home-brew calculation determines the capital_used variable first and then the available cash. This is only accurate at the start of handle_data and assumes no open orders (which tie up available cash and reduce new buying power). Typically I take that 'cash' variable and add or subtract any buys and sells for orders sent by my logic. That way I know how much unencumbered cash I have at any particular moment. Since I recalculate available cash every event I don't worry too much about commission fees.

The bottom line is the existing portfolio variables 'capital_used' and 'cash' are not useful for Reg-T margin calculations. And I seriously doubt they are even useful in a Portfolio Margin account. Making the change in calculation suggested by my home-brew formula would not impact the use of these variables in a non-margin account (e.g. long only position). And the change would make the variables suddenly useful for Reg-T long/short positions.

Hi Dennis,

What's the status of this effort? It sounds like you are on the right track for capturing the regulations and the Interactive Brokers rules.

I agree that Quantopian should consider some default settings that would conform with limits on a typical account. Perhaps some sort of warning could be provided (e.g. on the returns chart, highlight time periods over which the rules were violated).

Grant

@Grant, from Dan's comment above it seems like they are willing to listen. But as users we have to identify our needs clearly. I think I've gone a long way in that direction. However I'm not an expert and I may have overlooked some things.

It would be useful if someone here with more of a financial / broker background could weigh in. I'm coming at things from the perspective of a programmer so while it may seem logical to me that doesn't make it necessarily so.

Has there been any update to this topic recently? I asked a similar question in relation to live trading, however I haven't been able to find a solution. My issue is that there doesn't appear to be a way to limit your starting_cash in live trading that I can see - this makes it a little tricky to run an algorithm if you don't want it to be your only one!

In addition, your capital_used parameter doesn't seem to increase as the algorithm accumulates profits, so I'm doubtful that this issue has been fixed yet?

Any updates on this? I'm working on an algo that often trades somewhat illiquid equities and constantly having problems with a negative cash balance as open orders do not affect the available cash.

I've figured you could write a function to tally up the cost of all orders but that brings forth another problem. I don't think there's a way to find the limit price on a given order, only the sid and amount of shares. You could keep a list of orders and their prices as you place them but that's very cumbersome.

Anyways; thought I'd check in and see if this has been worked out yet, thanks!