@Peter, thank you so much for scrutinizing our model so closely! Sharing your critique gives the community a chance to improve the system - this is exactly why we decided to opensource our backtester, Zipline. I wrote a bit about the behavior of the simulation here and you can see the code here. We will also be presenting Zipline at NYC PyData in October, we'll be sharing the slides after the talk and in the meantime you can see the abstract of Thomas Wiecki's Zipline talk.

To your questions:

• In Zipline, but we haven't exposed that to Quantopian's code editor yet. The backtester code is available here. You can see that it allows a "no slippage" simulation, which disables any price history modification.
  
• The simulation is not currently parameterized. This is something we need to do, and has been requested frequently.
  
• I am very grateful that you pointed out your findings about the bid/ask spread and that you took the time to share them with the community. We want everyone to know and chime in.
  
• You are right that we only provide market orders, and I agree that limit orders are a critical need.

At the moment, we use a default simulation style on all backtests, which is called PARTIAL_VOLUME. The theory for this model is to approximate what your trades would have done to the buy/sell balance at the point in history. To do so, the amount of the order is compared to the volume of booked trades in the current bar. The forumla we use is: simulated_impact = (volume_share)**2 * .1 * direction * event.price. In addition, we limit the volume_share, which is the order's fill amount as a percentage of the bar total volume. The limit is 25%.

In the case of FXA, the volume is very thin. The security appears to trade 100k-200k per week, or an average of 100 shares per minute. Given that we are trying to fill all your orders in the subsequent minute bar, that means your orders will very often be hitting the 25% volume limit. Our slippage model would then calculate your order's price impact as (estimating $100 price): (.25)^2 *.1 * $100 = $0.625. We apply this by adding it when you place a buy, and substracting it when you sell, so that the total spread is $1.25. This is a perfect storm for our simulator - low volume, high price, maximum slippage on virtually every order.

I can think of two ways to look at this example:
1. The price impact model we have breaks down for high price, low volume securities.
2. Our order model, rather than our slippage model, is too naive.

It took me most of today to catch up with your thinking in the last sentence of your post, but I think it is dead on. The root cause of this behavior is that the order model is too aggressive and over-simplified, and it is the order model that breaks down in thinly traded stocks.

As you mentioned, we only handle orders as market orders. Furthermore, we assume that we should fill your order as quickly as possible. In a thinly traded issue, that tends to maximize the price impact. We chose to do it this way initially because it was simplest, and because we felt users could override the behavior by adjusting order amounts. In other words, we deferred the order management to the algo writer. My take, based on your feedback here, is that we should focus on enhancing the order module to be a bit more sophisticated so that your algorithm can place limit orders, provide a fill instruction (e.g. fill over the course of a time frame, buy at the vwap, etc).

In those more sophisticated order types, I would still apply the slippage model. One goal of opensourcing the backtester is to draw more people to work on the simulator. I really like the idea of parameterizing the model because it allows backtests to be quoted with the parameters, so that the intended behavior of the simulator is clear to those interpreting the backtest results.

Now, I write all this having spent a significant amount of time and effort on the existing system, which I know could be clouding my judgment. So, I really mean it when I say: please tell me if I am wrong. All that matters is that we end up with a world class backtest and trading system. I think that is far more likely with community members like you holding Quantopian to the highest standard.

Thank you again for your careful review, and thank you for sharing it. This is exactly the kind of feedback we hoped would come from the community. Please keep it coming, and think of yourself as part of the team!

Thanks for the detailed response. I did indeed see from the code that there was a no slippage option. Could you please post a code snip-it or example on how to place the simulation in this mode. I'm still trying to get my head around Python.

I was playing with the "Epic fail for stat-arb" algorithm and was getting the same results that you posted when I noticed the pricing spread problem causing that shaped profit curve. FXA is a share proxy for the Aussie Dollar. Shares are issued in 50K blocks. There is quite a long list of these share trusts. The book is very deep and even thou the volume can be light you really can not move the price. One further question, does the simulator have access to the bid ask minute values. If so perhaps this could be a selection option with a parameterized slippage amount, or alternatively just price could be used. With these options knowledge of the usual trading spread would be needed by the user to set the spread amount to a proper value, e.g. 1c or say 10c.

@Peter, we need to do a little work to make this possible from your algorithm. We think a natural way to enable this is to let your algorithm specify the parameters for slippage in the init method, by setting a value on context. We're thinking you would set the style like this:

# Use the fixed slippage model, which will just add/subtract a specified spread  
# spread/2 will be added on buys and subtracted on sells per share  
# commission will be charged per share  
context.slippage = FixedSlippage(spread=0.05, commission=0.01)  

We could also parameterize the volume share model like this:

# allow algo to buy up to half a single bar, scaling price impact with 10%  
context.slippage = VolumeShareSlippage(volume_limit=.5, price_impact=0.1)  

We would allow your algo to set this property in the init method, and the system would just ignore any changes during the backtest.

@Peter, sorry I didn't answer one of your questions - we do not currently have bid/ask data, just the trades.

@Fawce, perhaps you could reposed the arbitrage model showing how to use the new fixed slippage "price spread" model when the code is available. It would be interesting to see how, if any, the performance changes.

@Peter, that is a great idea, will do.

@Peter, we've added api methods to set slippage, and to set commission. We also added a really slick autocomplete UI. You can read about the api calls and the autocomplete here: https://www.quantopian.com/help#ide-slippage

Quick example. To set a fixed slippage of $0.02 and a per trade commission cost of $5.00:

def initialize(context):  
    set_slippage(slippage.FixedSlippage(spread=0.02))  
    set_commission(commission.PerTrade(cost=5.00))  

To use a per share commission of $0.01 instead:

def initialize(context):  
    set_slippage(slippage.FixedSlippage(spread=0.02))  
    set_commission(commission.PerShare(cost=0.01))  

Are there plans to incorporate minute data for the ask and bid?

In my humble opinion those are as essential as the last sale price.

Also, are there plans to allow slippage to be set outside of handle_data? I would prefer to set it dynamically to respond to a stock's current state.