A couple of questions, Dan, if you would be so kind to answer. What makes you think this was a flash crash? The only market-wide flash crash resulted in exchanges voiding trades and a trader sent to jail for market manipulation. There was no such action today, as far as I know. No trades were voided, no market dislocation was observed. The volatility was high, but it has been higher. The market has been signalling for weeks and arguably oil, Chinese equities and other pockets of the market have seen recently much larger drawdowns. Today was just a normal, very quick market correction, the kind that makes the allure of being an investment professional less appealing. How many phone calls from nervous investors did you field today?

I've been around the track a few laps. Today was not a market dislocation, it was panic selling. It happens every four or five years and is far too infrequent to be modeled easily with statistics. Trying to "create your own reality" when what really happened, Simon was "scalped" by an ill-conceived stop loss policy, is worse than just taking your lumps. Many market participants saw this day coming from months away, many others tried to get out this morning. No flash crash, just good ole market fluctuations.

Certainly, you can't just inject capital into Simon's portfolio and then advertise the subsequent results as "performance". Doing so is a huge disservice to any future investor or client who uses Quantopian. You need to report the actual, GIP compliant performance otherwise you are just running a scam.

My two cents - which are worth less today, but I'm calling it $2.00 anyways.

This just occurred to me. Simon's performance was +5% close of business Friday. To have triggered the 10% stop loss (actually a drawdown to $90,000) his algo must have seen a 15% drop or more. on Monday Please, don't bury bad returns by using a "do over". It is just not right.

Since the performance was not being updated on the winners page, and the other three winners had remarkably stable performance, varying by less than 1/2 of 1% throughout the day, were they given a "do over" too? This "do over" really calls into question the entire performance reporting. My guess is that at least two other contest winners got stopped out today.

Dan,

I would tend to agree with Sally. Everyone should take their lumps. You could re-start Simon's algo, but if he makes any money, it should be distributed over all of your 45,000 users (each one getting a nickel) or given to charity.

Grant

Giving my opinion here, I don't think Simon should be held responsible for the liquidation. Like Sally said, "Simon was 'scalped' by an ill-conceived stop loss policy." It was a flawed stoploss system, but that had nothing to do with Simon, the 'internal' stop trigger should have caught this with a few simple sanity checks and not liquidated the account. The stoploss will be patched for going forward for all contestants, so it doesn't seem fair to make Simon bear the weight of our growing pains.

Was I the only one who saw insane paper trading results border lining on unbelievable? I know it was abnormal but I have an algo return 18% for a period of about an hour this morning then one hour later go down to 6% (a little higher than it was the day before). Am I the only one suffering from what almost looks to be a data malfunction? I know it was a weird market, but this seemed to be something different.

Here is some related information about abnormalities in today's trading (especially ETFs):

It was definitely weird, but I don't think it was a data issue, I had a reversion strategy get lucky and make 5-10x more than usual on a few trades. I think a lot of money just changed hands really fast because it happened to $ algos too.

I tend to support the opinion that we don't do anything with any of the algos in current competition. I would say there is no absolute "fairness". Or in some sense, it is actually fair that these algos are running on the same "flawed stoploss system" with the same liquidation rules. I suggest that we decrease the "weight/importance" of this competition as a whole and at the same time improve the framework to better execute/evaluate future competitions.

Hi Andrew, this doesn't affect competition algorithms, just past winners running Q's capital in IB.

I don't think Simon should be held responsible for the liquidation

Well, I'm not sure I completely agree. We don't have the details of the algorithm (he could publish it, if he wanted), but Simon seems pretty knowledgeable about the market, and is familiar with Quantopian's system. He's not responsible for the liquidation, since he didn't click the buttons to stop the algo and sell. But he may have been well aware that his algo could respond as it did, and end up below the $90K mark (I certainly knew that my winning algo could end up in the rubbish heap).

Also, the idea that Quantopian was applying a flawed stop-loss system is simply not valid. They did a thumbsuck that $10K was the absolute max. they'd be willing to lose. What if they had said that $5K was the warning level, with a guard band of $5K? Perhaps now they effectively have a $10K warning level, with a $10K guard band?

Although perhaps not germane to Simon's recent debacle (since no trades were executed), one thing that Quantopian could do is actually publish the full architecture of their system, end-to-end, and open-source everything, hardware and software (as Dan says above, "we are committed to transparency."). I have to wonder if this so-called "flash crash" event highlights that users don't actually fully understand how Quantopian works, vis-a-vis the actual real-time, high-frequency market. Nanex is capturing high-frequency trades (how?), they end up in the Quantopian feed and are used by the algo, with trades send asynchronously. IB picks up the trades and executes, but at what time with respect to the actual market events captured by Nanex? And where do the history() data come from--Nanex or the backtest data set or some combination of the two?

market is misbehaving

Ironic. Isn't Quantopian hoping to build a business on a market that doesn't behave? If all securities were perfectly priced at all times, then how would the whole hedge fund idea work out? I've gathered that the market is a global robot war, so the idea that there should be some sort of well-behaved, "fair" prices may be inaccurate.

Dan,

Would you be willing to elaborate on the mechanics of your stop-loss applied in this case? Was it simply someone sitting at a terminal at Q headquarters? Or was it automated, either at Q or at IB?

Also, I've gathered that you'll move away from IB for the hedge fund (at least eventually), so perhaps you won't be developing any sophisticated real-time risk management until that happens (and the prime broker can do it for you)? And on a similar vein, what about the Nanex feed, minute bars, etc.? It would be interesting if you could describe your plans in this area. I've gathered from comments on the forum that your present system is basically unworkable for a real hedge fund; maybe this "flash crash" event supports this thesis?

Grant

Sally is 200% right.

Yesterday was NOT a "flash crash", whatever that word even means. Given the right setup, yesterday morning is a dream come true to a certain type of trading/market making strategies.

There is simply no do over in the real world. There are always lessons learn every day in the market, but lots of these errors is simply where the rubber ("algos", produced and priced at minute bar frequency) meets the road (Individual stocks rotation, market panic, stop loss triggered, bid/ask changing faster than your cancel/replace order.)

Most pros have seen this before and understand how to take advantage, or step away entirely.

However, reading between the lines, Dan already revealed what happens behind the scenes. But the reality is that people need to be aware that there is no rule that says price can't fluctuate more than X% intraday, and for all 4,000+ stocks at the same time. By some calculation, Dow moved 4500 points yesterday. down up down up....

It's another day, good luck to all.

By the way, algorithms, models, strategies are all overloaded now in meaning... and my understanding of the terms will differ.

one thing that Quantopian could do is actually publish the full architecture of their system, end-to-end, and open-source everything, hardware and software

Asked and answered, numerous times, most recently here.

I looked at Jonathan's link and realized that perhaps a different question was asked, and that question was explicitly not answered to a certain extent for obvious reasons.

I think people are not talking about about zipline, mongoDB, python objects or inline browser syntax highlights. I am pretty sure people are asking about "RISK Management" policies and practice, in particular yesterday morning from 9am to 12 noon. And in particular, operation risk factors pertains especially to yesterday when it comes to portfolio liquidation policy. data latency and brokerage communications.

When Dan mentioned Net Liquidation Value being used (from IBKR), that is a very well known metric from their API. How much cross validation was there? Maybe the threshold is too narrow? Maybe the best thing is doing nothing? What was the slippage between NLV observed and NLV realized? (Positively or Negatively.)

These are relevant questions given the extreme intra-minute volatility that happens maybe once every few years... (for now. But they do happen...)

Love to hear some insight, instead of people guessing.

I have to wonder if this so-called "flash crash" event highlights that users don't actually fully understand how Quantopian works, vis-a-vis the actual real-time, high-frequency market.

I've yet to see any evidence that this has anything to do with people's understanding of how Quantopian works. Perhaps you could explain in more detail why you think that.

Nanex is capturing high-frequency trades (how?), they end up in the Quantopian feed and are used by the algo, with trades send asynchronously. IB picks up the trades and executes, but at what time with respect to the actual market events captured by Nanex? And where do the history() data come from--Nanex or the backtest data set or some combination of the two?

There aren't any big surprises in the answers to these questions, and none of them is terribly relevant to the events under discussion.

Nanex gets trades from real-time tapes provided by the exchanges, just like anyone else.

We've discussed before the timing of how trade data makes it way into algorithms and trades makes their way out, e.g., here.

history() data comes from our historical data, not from Nanex data, even in live trading. We've considered changing that to allow history() data for live algorithms to come from Nanex data, or even allowing history() data for backtests to come from Nanex data, or even allowing the algorithm to select the history() data source, since each of them has its advantages and disadvantages and which one is appropriate may depend on the algorithm. We'll probably implement one or more of these at some point, though they aren't currently high on our priority list. The source of history() data has no relevance to what happened yesterday to Simon's algorithm.

the market as a whole wasn't working appropriately. In a more rational and orderly market,

Dangerous verbiage when juxtaposed with one of this field's most famous adages: "Prudent speculators never argue with the tape. Markets are never wrong, but opinions often are."

I think the restart can be acceptable as a way of being fair to Simon for the forced liquidation due to a Q policy that didn't appropriately reflect all market conditions. Selling into a low-liquidity panic due to odd marks happens all the time, both voluntarily and through broker liquidations due to margin and I guess this is how you learned that in times of low liquidity and panic, stop orders get destroyed. The people upthread were mentioning GIP compliance and I do hope that this is strictly a change to the contest as far as paying Simon and the loss will actually be reported in any results to potential investors -- anything else is clearly wrong.

For Simon's algo, why was it restarted at Friday's valuation and not at the value that it would have had at the end of Monday? If the intent was to fix forced liquidation, shouldn't the solution to be to put him back into positions he had before the liquidation and not ignore any return of those positions on Mon?

Let me suggest a slightly better liquidation policy to handle such and other situations; I'm guessing this is more important for the fund (where liquidating into such market regimes will be an absolute disaster due to market impact) than for the contest itself. You should add the notion of a 'restricted mode' in which an algo can only flatten existing positions and not enter new ones; it's a good thing to have in any case. Then, if the current mark drops below your cutoff point (say 10%) you switch the algo to the restricted mode. You need this to protect yourself against berserk algos which may make rapid trades and lose everything over the course of day. Additionally, you should have a second rule under which you forcefully liquidate the positions and shut down the algo. That has to be less suspect to false positives but doesn't need to respond as quickly. Sensible candidates for that rule are 'algo has been in the restricted mode for more than one day,' etc.

You can do over a backrest: just rerun it.

You cannot do over live trading. What happened, happened.

However ill-conceived Quantopian's stop-loss policy may have been, it had been published, and Simon could have taken steps in his code to avoid having his portfolio liquidated. He chose not to, or (less likely) didn't do it correctly. He should bear the consequences of his decisions. Restarting his algorithm with a different "reality" than what actually happened would be like a teacher rewriting a student's exam paper after he failed. Is this the message Quantopian wants to present potential investors? Other competitors?

Or let's do over Vietnam, and keep doing it over until we win.

Hi Everyone,
Thanks to everyone participating in this thread for their advice and analysis. Openness is our superpower because our community always responds creatively. I'm really grateful for the constructive attitude here.
I'd like to clarify what happened yesterday, and I thought a simple event log would be best:

• 8:00am - Simon's account value is $105,030.21 as of the close Friday
  
• 9:32am account value drops to $96,983.81
  
• 9:35am - our automatic net liquidation alert triggers. This alert monitors the IB net liquidation value of the account. Account value is $88,548.71 -- a significant drawdown compared to expectations for this algo.
  
• 9:35am - We manually confirm the IB NLV number against our own dashboard, which is driven by NxCore pricing data. Both show account value of $88,500
  
• 9:36am - Simon's algo is stopped, and a liquidation algo is connected to the account. Our protocol for the contest calls for immediate execution, and our team executes exactly as planned. The process of deploying the liquidation algorithm takes 2-3 minutes.
  
• 9:38am - We realized that we were in the middle of a severe "V" price movement for the account, most likely caused by very limited liquidity for positions in Simon's portfolio. I call an audible to NOT liquidate the portfolio, but it is too late.
  
• 9:38am - The liquidation algo forcibly moves the account to all cash. Account value is $100,644.63.

We spent time reviewing the event to ensure we using both IB and NxCore to confirm the drawdown. Since we used both, and having reviewed the tick level data today, we are confident that the market did (very briefly) price the portfolio at $88,500.
While we did exactly what we said we would do in the situation, it was clearly a huge mistake to liquidate the portfolio. I view that as a bug in our protocol: inadequate coverage of scenarios in our process. Such V shaped moves are well-known possibilities and automated liquidation is not the right reaction. The change we will make to our protocol is to add a 10 minute pause between algorithm halt and position liquidation. During that pause, our team can switch from automated handling to manual review to cope with events like Monday. In the coming days, we will be conferring with our advisors in order to further improve the protocol.
With this understanding of the events, I have two remaining concerns:

• being fair to Simon
  
• maintaining total transparency for anyone participating in (or following) the contest.

In talking to Simon, he specifically said that he was "ok taking his lumps" like everyone else, since the stop out and liquidation was valid by our rules. Nonetheless, I feel responsible for the bug in our protocol, and I insist on adding cash into the account to return to Friday's closing net liquidation value. I chose Friday because it is a firm quantity we can identify without estimation. My goal is to be fair to Simon, and I think using Friday's close is fair to him.

Do I want to be judged for giving this "do-over"? Yes. I made a mistake and I am taking the opportunity to fix it. I want anyone considering submissions to Quantopian to know this is how we operate: open to all, transparent, and fair.

For everyone following the winners page, I want to be clear about two things:

• The contest is not a fund, nor an advertisement of any kind to investors. It is an incentive to rally our community around building great algorithms, without requiring participants to incur any costs or risks. It was born of intense and constructive community feedback.
  
• The winners page must meet our standard for transparency and openness. Therefore we will attempt to tell the whole story: Simon's headline will report the final value of his portfolio after liquidation on Monday -- $100,644.63. In addition, we have a note that links to this thread and shows the current portfolio value of his restarted algo after the cash injection. That meets Quantopian standards for transparency.

Thank you again for spending your time here and participating in the community. I started Quantopian because I wanted to make products and provide services to Quants - brilliant people who always seem willing to teach. Everyday we are getting a little bit better together.

thanks,
fawce

+1. Thank you for the openness, Fawce. In my opinion, accurate un-interrupted reporting of trading performance is of uttermost importance - whether it is aimed at attracting investors or not - and this solution seems to address this.

I believe everyone running real-world algorithms has made this exact experience of price spikes at least once - be it an massive intraday drop, or a sudden short-squeeze. It's almost like a coming of age ritual. Happy belated birthday, Quantopian. :-)

You may want to check your math on the winners' page, Simon's portfolio is quoted as "$100,644.63 0.06%". I believe $100,644.63 is 0.64% increase over $100,000.

I also will have to respectfully disagree with the V-shaped market argument. Looking at a two-day minute bars, there was a gap down on market open on Monday and then once price discovery had matured a few minutes later, the market bounced like a ball, approached its low for the day around 3:30PM and is darn close to the market low of Monday at today's close.

My opinion is that this was a normal event following an 8.5% drop in the Shanghai market. I'd follow that with a caution about trading at market open.

That was great insight. Thanks fawce.

As someone who have seen this movie many times, it is not easy.

Good luck to every one.

Fawce:

It seems like your protocol is modified as follows:
1. if hit < 90K, stop algo
3. if holdings are not above 90K 10 minutes later, liquidate positions
else if holdings are above 90k restart algo (which has to deal properly with restarts)

With this protocol you're opening yourself up to more risk if this happens in the last 10 minutes of the market.

@Simon: Were you trading BKCC? If so see:
http://www.streetinsider.com/General+News/Blackrock+Capital+Investment+(BKCC)+Trades+Canceled+Below+$5.86Share/10838630.html

Robert

@ Jonathan,

Sorry, got a little carried away with irrelevant information. Thanks for your feedback.

The contest is not a fund, nor an advertisement of any kind to investors. It is an incentive to rally our community around building great algorithms, without requiring participants to incur any costs or risks. It was born of intense and constructive community feedback.

Fawce,

I have to think that you have finite capital to throw at this effort, otherwise you'd have a lot more winners. I also have to wonder if you like Simon's algo as a candidate for the fund, and you need to see how it plays out with real money. Unless you have other algos funded outside of the contest, the winners will be all you can hang you hat on (you can run simulations until you are blue in the face, but you'll need real-money trading results to attract capital). I fully appreciate the promotional value here, but might there be more to the story?

Grant

What if the market hadn't recovered? Just because it was a V move doesn't make the stop wrong. It was stopped rightly so!
If the risk is set to -10% and you stop at -10% the market go to -11% then all the way back to +1% then you try to undo the stop?

If Quantopian wants to be realistic if money was lost the algorithm is out. I don't mind if you restart it and keep it private and give
the money to Simon. but not in the competition, it would also be nice to have the result of the stopped algorithms shown in the winners page.

Being transparent (Scientific) also means showing bad results and the mistakes.

I cannot find a Broker that will return my money in a bad stop-limit. The fact that the stop was not automatic doesn't justify it as a mistake we assume the stop is built in and automatic.
We all want to see how the algorithms and Quantopian behaves under bad circumstances so we can gain some trust but as soon as Q start to erase and manipulate the results then where should the trust come from?

If I had an account and I had stopped it could I have claimed my money back?

My algorithm profits from permanent price impact due to inefficiencies in the creation process of several ETPs. This manifests most often during times of market distress.

It holds a hedged basket of ETPs, rebalanced daily. As such, it is usually almost (but not quite) perfectly hedged. There's a trade-off there. Nonetheless, if you peek through my holdings of the ETPs and examined my net holdings of their underlying assets, you'd find that my portfolio was substantially flat, and leveraged, notwithstanding discrete hedge rebalancing.

The morning in question, the account equity started off at 1.05. In the dislocation, there were two effects in play. One was a legitimate drawdown of around 5% to due to the imperfectly balanced hedges. The other was a ~12% transient effect caused by these ETPs trading far from their Net Asset Values, or Indicative Values. I'm still working on getting higher frequency data to dig in more deeply.

The first effect mean-reverted throughout the day, and the second effect mean-reverted in about 5 minutes, certainly within 15 minutes from open, I'm not 100% certain. Regardless, my algo would have been flat and then some by closing. Whether the second effect was worth triggering a stop for is the point under debate, I suppose. From my perspective, I "knew" that the underlying assets were still hedging one another, and that it wouldn't be long before arbitrageurs turned on their machines again. But, the funds did trade at those prices, and so for a few minutes, my mark-to-market was definitely below $90k.

I was perfectly willing to let the stop stand, if only to avoid ugly threads on the message boards, but it wasn't my call. It would have been a little unfair if I was the only contest winner ever to have been automatically liquidated, but I'd have accepted that.

For what it's worth, my algo would have jumped over 12% that night. With the do-over, I've caught the 5-6% mean-reversion of that overnight equity gap, but not the overnight spike. This catches us up to now, where my algo I think is trading around 0.99, from the 1.05 it was restarted with yesterday at lunch time. Those who relish schadenfreude, take heart that despite the top-up, I am underwater rather than up 10-15%.

I'd have made some MS Paint pictures to go with this, but my laptop is installing updates, sorry.

EDIT: I did take a quick snap of the drawdown vs paper trading: http://imgur.com/rONadsX

To be clear, I wasn't involved in the decision to liquidate or not, I "knew" the portfolio was hedged in restrospect/research.

EDIT: Also, I stated the above all rather categorically, but these are just my conclusions based upon a couple of evenings looking at the data I have. I plan to improve the hedging methodology to avoid getting so imbalanced, and perhaps even to step in and take advantage of these price-to-NAV dislocations as much as one can without access to the intraday NAV data.

I appreciate all the contributions that people have made to the community here.
However, this just seems like another academic mistake that wouldn't be excused in the real world.

I have to agree with Carter Pelot. In the "real" world, Simon wont' share his algorithm, even in pseudo code. Fawce wont' publish the minute-by-minute on events and subsequent decisions in such details. And I won't point this out to everybody.

Good luck to all.

Thanks Simon,

If I'm reading you correctly, it would be helpful to have a feed of the intraday NAV data. Does such a thing exist? Would it help at the minutely rate of Quantopian?

Another way, what would be your ideal platform to trade your winning algo (assuming you could re-write it to leverage the platform)?

Grant

That data would be helpful for a whole class of algorithms, those that attempt to trade ETFs which are at a discount/premium to NAV. I hear that ETF sponsors publish this data to the tape every 15 seconds, and on Yahoo Finance sometimes you can find it (ETF ABC would have it's IV published under ^ABC-IV) but it doesn't seem consistent.

Additional data is always helpful, whether it's NAV data or index data or futures data; if you can find or calculate the 'fair value' of something, that goes a long way in determining whether you want to buy or sell it. If all you have is the price an ETF just traded at, it's hard to know whether it's suddenly worth only that, or whether someone else just got suckered and the limits to arbitrage are preventing other people from correcting it. That helps to decide whether you want to swim against the tide and try a mean-reversion trade (or just hold on), or whether you become the next sucker, to extend the metaphor.

I'd also just briefly like to draw some inspiration from this; a lot of the very best trading opportunities are when people are buying or selling against their own self-interest, because they are not adequately informed or because they are compelled to trade. When you wonder who was selling ETFs at a 10% discount to their NAV when the chaos was over in minutes, well, it was us, and if you'd been there with a bid and a hedge, you'd have pocketed that 10% for yourself. Something to think about.

I personally don't understand the outrage behind the restart. It's Quantopain's money they should be able to do whatever they want with it. If they set Simon's algo back up then it obviously has some good further out backtesting results or for some reason they want to see a larger sample size with real money. No matter what their motives are it is perfectly within their right and I personally have no problem with the algo being set back up. It was clear to me as a neutral bystander that Simon algo does at least seem to be on to something as shown by the 5% or 6% returns before the incident. They could have set it back up and not let us seen it and only Simon would have known about it. They choose to be transparent and in a rare bit of community hostility seemed to get crucified for it. Just my opinion which is worth about as little as the next guy.

Spencer I totally see how you could think that and I agree with you - Q can do whatever they want with their money. I was just throwing my two cents in on the matter. I think everything Simon (and the rest of the community) is doing is amazing, and I am very interested in all the research. Events like these are truly what help us all learn.

@Spencer
If course Q can do what they want but for people who want to use them to invest their own money it is not nice they are taking this path. Erasing bad results.
It is about being scientific you cannot restart an algorithm in real-life.
The fact is the algorithm could have done something when it was -9%.
The best thing to do would be Quantopian pays Simon 5% or 10%. keep the stopped result in the contest page and Simon submits his algorithm for September.
But to remove from the contest page or even replacereplace the bad result with what would have happened is just unrealistic. It is just what banks do. They have replaced all their ETF that had bad results from 2008 with new ones lol

@ Spencer,

The Quantopian team is counting on us to work for them for free (I can just picture the slide to their VC's, "And look...our R&D labor costs will be zero! The lowest in the hedge fund industry!"). We are the crowd in "crowd-sourced hedge fund." They might argue, but in my opinion, Quantopian is not transparent enough, and might gain in the long run by giving users more power (and even some form of ownership). It is their money, but they'll just be burning it if they don't connect with the crowd and maintain a trusting relationship. It states clearly on https://www.quantopian.com/open, "If the balance drops below $90,000, the algorithm will be stopped and no profit will be awarded! It's important that your algo avoids large drawdowns." So, if it seems like they are weaseling around their own rule, it is not exactly a relationship-building move. In the end, they probably made the right business decision in re-starting the algo, and in revealing their actions to the crowd, allowing us all to vent our spleens (which they likely anticipated). We can all just move on, Simon hopefully will make a bundle of money, Quantopian will get to fund his algo (or some variant) in their hedge fund, with real-money trading results as part of the due diligence package, and press releases can announce the success of all parties involved (with the nitty-gritty technical details of the liquidation left out). One has to wonder, if Fawce's "audible" had not been too late, would any of this have been revealed? I have to think not (which probably would have been the right business decision, so I'm not criticizing in a practical sense...they have a business to run).

@ Simon,

So, say I take something like SPY, that tracks the S&P 500. At any instant of time, shouldn't it be fairly straightforward to derive what should be the NAV, based on the prices of the underlying securities? Or is it not so simple? Would one actually need a feed from the ETF itself? And why would ETFs publish their holdings, anyway? Is it a regulatory requirement?

I'd just like to point out that the competition is essentially one long, expensive, learning process for Quantopian. At the moment it appears to have cost them ~$38k, although that could change. One may extrapolate that and compare against the salary of an experienced full-time developer. In any case, those hard-learned lessons are worth every penny, and they are the sort of lessons that can only be learnt when playing with real money. Be prepared for whatever the 2nd lesson is ;)

My opinion.

The handling of the contest reveals a couple flaws in Q's investment model. First, the idea that your algorithm is yours to own and Q's staff will not look at the code or understand the strategy is what I call a 'black box' approach. As Simon's experience shows, this is not optimal and may be fatal in the long run. Simon gave a perfectly reasonable and acceptable qualitative analysis of what went wrong on Monday for his strategy and if he will be forgiving I'll condense it down to a two-word phrase; stale prices. One leg of his strategy was being priced and the other leg was not (or was not priced appropriately; i.e. at fair value). Whoever sits at the trading desk needs to know this, have the tools to assess this rather common experience and take appropriate, discretionary action until Q figures out how to handle these regularly occurring gaps algorithmically in their trading operation.

For example, let's say I had a two security ETF and my strategy was to short the ETF and go long the two underlying equities. At market open on Monday, my two equities dropped 10% and the ETF did not trade; bam automatic stop loss liquidates my portfolio even though in all likelihood the selling of the ETF set a price that brings my simple portfolio back to zero. I'm not saying that is what happened to Simon, I'm saying the stop-loss rule and black box limitations on traders makes this possible, even likely. Imagine if instead Simon's portfolio was $50M instead of $100K. A good trader would have 'pinged' the market, trying to sell small amounts of the ETF and buy small amounts of the underlying, acting as an arb until others recognized the opportunity and fairly priced the ETF. My guess would be that would have cost maybe 5 shares of the ETF (<$1000) to accomplish this.

Q has focused on algorithmic portfolio construction, but where is the sophisticated trader development?

On Monday, I sat in front of my bloomberg, watched ETFs trade at ridiculously low prices, far off fair value. I stayed my hand at these ephemeral opportunities since my clients have different investment objectives than trying to compete with the HFTs who were ruling that moment in the market. You might have been able to buy some ETFs far from fair value but not in size and not with simple market or limit orders and certainly not as a multi-billion dollar "algorithmic portfolio construction and shouted-out trading instructions" firm that is currently Q.

Second issue that I'll opine on is the subsequent handling of this event; it displays a lack of confidence in the current process. Instead of picking apart the actions of Q's staff or the statements of the peanut gallery, I'll offer an analogy from my life experience. Someone in a different thread mentioned Long Term Capital Management. During the late 1990s, I worked for a very successful hedge fund in a small group that ran a fund of hedge funds. LTCM refused to accept an investment from our fund, worried we were going to whore their brilliance. In fact, one of our prop trading desks already was deeply involved in similar convergence trades exploiting the calendar spread inefficiencies in the treasury market where leverage can be astronomical since the hair-cut on treasury collateral was minuscule. Two days before LTCM went belly up, we received a call, they would accept a $50M investment. Only the train delay between Manhattan and Greenwich saved our investors from being at the end of a long line. Of course, our prop desk blew up and not only caused an almost end of life event for our parent, but reverberated in the executive suite in the largest bank in the US. Although 90% of the staff was let go the quant/trader/PM of the failed strategy was retained because management understood the risks of the strategy and had confidence in it from that understanding.

Without confidence you will not be successful in the world of investing. You can not have rational confidence in that which you do not understand.

Q's response to the minor issue of a $10,000 drawdown showed a lack of confidence. Changing the rules, even of a game, displays a lack of confidence in your ideas and hints at a lack of integrity in your management of money.

My opinion, feel free to flame.

@Lucas - please look at my post above. We are reporting the final value of Simon's liquidated portfolio on the winners' page, with a note pointing here and also providing the current value of the restarted account. I believe we've already did what you are asking.

@Grant - I do think it is awesome that we have the largest research department in the world. I also think it is a fantastic deal for you and everyone else using the platform. I'm proud that we came up with an incentive structure that is attractive to quants and scalable for us.

@Sally - Thanks for the candid thoughts and interesting anecdotes. On Monday, we had all the information necessary to trade properly, but we didn't. I know we will learn from this and get better.

We continue to analyze the events of Monday, and ...

Learning Continues

Today we realized that in addition to the net liquidation value dropping below $90k, the leverage of the algorithm also shot above the contest limit of 3x. We have an automated trading guard in place for the leverage, but we had a configuration bug that disabled it for this algo.

When we restarted the algo yesterday, it tripped our max order size trading guards. The algo was moving from a $105k cash to fully invested, and the guard was configured based on starting from $100k cash at the contest start.

To get the algo to its target positions, we decided to place trades manually to match the orders that were canceled. We decided to use the exact share counts, without accounting for price changes. That was a mistake, because the manual orders then put us (ever so slightly) over the leverage limit.

At this point, you may be wondering why we soldier on with this algo. My reason is that we're still learning from this experience. To me, the biggest area of improvement is to our automated guards, and designing a better trading protocol to cope with events that trip the guards. I don't see a generalized solution to that problem - I think we will need to plan around the specifics of each algo, working with each author.

These are difficult problems. Some of them are faced by the current industry and we can learn from and copy their practices. Some of them are unique to Quantopian due to our set up. We haven't gotten this far with the company by shying away from hard problems. We'll continue to tackle the hard problems and learn. With gusto, humility, and openness.

As I understand, the algo owners are not required to maintain the "black boxedness" of their algos. In fact, there is incentive to reveal strategy details and code to the Q, when competing for capital. All else being equal, wouldn't Q pick an algo for its fund that it fully understands over a "black box" with any number of hidden risks. Yes, there's the risk that the core idea will somehow be stolen (although Q can see what goes in and what comes out, so they'll get a pretty good idea), but if the algo doesn't get funded, then where are you?

Post related to web response time moved to https://www.quantopian.com/posts/web-response-time, at Quantopian's request to keep thread on-topic. Please respond there.