Great! By this do you mean that the regular quantopian.com is now running the new platform code?

I think so. I was just going through the help documentation and it already reflects the changes.

Yes! Quantopian backtests, live trading, documentation, everything is now running the new code.

Awesome!

vwap was removed from data. I found that useful.

Is there a temporary way to toggle between the q1 and q2 environments?

Please give us a way to switch between Q1 and Q2... One need to be able to compare simulation results to be sure that the code has been migrated correctly... Without that, there will be quite a bit of frustration going around...

Please give us a way to switch between Q1 and Q2...

+1! Too late now for this transition of course. The best you can do I guess is try to replicate old backtests which you've already run. The time to do live comparisons between Q2 and Q1 ended yesterday.

I would be happy with some sort of smoother, opt-in transition. Maybe testdrive (beta), quantopian.com (production), tirefire (deprecated). If people do nothing, their algos migrate from quantopian to tirefire, where they will continue to run and trade as normal. But, if they fail to migrate to production (or click an auto-migrate button if nothing needs changing?) before the next major release, THEN they are screwed.

I've been working on porting my live-trading algos using other people's laptops while on holiday, in order to make sure I don't lose a bunch of money (unnecessarily) tonight/tomorrow, so I am not a terribly happy camper about this process. It makes me wonder again whether I should write my own rig.

Minh - you can migrate your old vwap to the new method. There's specific information in the migration guide

Keith and Shi - we don't have a Q1 environment available. In the majority of cases, Q1 code runs just fine in Q2.

If you're looking for a comparison, you can look at old backtests. The old backtest results are not updated - they are sill the Q1 results. If you're having problems tracking down changes, please contact support or open a new thread in the forums.

Well, I only realized the existence of Q2 yesterday, spent two hours migrating the codes to an executable state. But results between Q1 and Q2 were not matching. I know the mis-match comes from the part how I filter "bad symbols" (such as non-existent data[stock].price, which i am assuming is handled by either data.is_stale() or data.can_trade()). anyways, i should have realized that there is a Q2 earlier.. i feel sorry for those who suffered the same fate as mine but are trading with live accounts. i hope you guys have notified the licensed-algo authors in advance for this change. i guess such is life.

@Simon - yes, pretty rude transition.
No actual committed notice of the change.
just "try this out, we'll be transitioning in the near future".
Glad I don't have any real $ committed to this platform.
Certainly blew up all our contest entries...but hey!...there wasn't any entry fee!

I re-read my replies above and I think I could have done better. I was too short.

First, the good news: the vast majority of algorithms are unaffected by this change. We built "shims" between Q1 and Q2 so that most Q1 algorithms continue to work, no change required. Most of our community isn't going to have any (negative) impact from this change. The contest keeps running with very little change to the leaderboard, the paper trading records extend, the real money trades. Most people won't have any difficulties with the transition.

For our real-money traders, we had a higher-touch communication program. I don't think we had to stop a single real-money algorithm in this launch, and almost everyone had a chance to test before today. I wonder that we might have been too aggressive in our messaging - the changes in Q2 are generally minor for real money traders, and we may have caused unnecessary consternation.

The bad news: Some people had algorithms that were affected by this change. We identified them and contacted them tonight, to the best of our ability. The people who are most aware of the change are the ones who are feeling the pain of the change. I understand the distress and frustration.

We are very sorry that some people are having a bad experience with the Q2 launch. We knew that some people would. We decided to go ahead with the change because the benefits are so big - algorithms are so much faster, the trading limitations are far fewer, the API is more intuitive, and the pricing is so much more accurate. We view this as an exceptional situation. We almost never make a breaking change to our API, certainly not of this scope. We don't anticipate that we will need to do it again in the foreseeable future.

If you are having any difficulties with the transition from Q1 to Q2, please contact us. We're prepared and excited to help.

I definitely had/would have had to restart my real money algo. Never mind.

On another note, the performance gain on the production site doesn't seem nearly as substantial. Are there other infrastructural differences between testdrive and prod?

I would also add that in my humble opinion, making an irreversible non-backwards-compatible change which compels people to make like non-backwards-compatible changes in their code in a single night "Big Bang" is a catastrophe waiting to happen. It might go without incident this time, but to me, it seems like the risks are unappreciated, either because they are not seen (which I doubt) or because the risks are on the authors' balance sheets, which is dismaying.

@Dan, the upgrade is great, the changes are minor, and going forward, all is good!

My only issue is with the lack of notice as to when the change was going to occur.
A simple "The Beta for Quant 2 is good, so we're going to make the change in 48 hrs" would have sufficed.

Ok, thats it from me...actually glad you have this forum for public issues...most companies don't, so thanks!

@Simon, yea man, i was shocked when i logged on an hour ago and my golden goose just crapped and died. LOL. ah well. i am just happy that at lease it seams that the live/licensed algos are handled seamlessly. A heads up would be nice though.

Regarding the performances of simulation, i do observe a increase in speed for simulations with shorter time span, but not too much. i ran simulations over 13-year period, in minute mode, which used to take hours, not sure how many hours it will take now because it's still running... haha. but by the look of it, it will take at least 2 - 3 hours to finish one run. almost no observable difference..

@Dan Dunn, i have a question about Q2. When i run simulation in Q1, with certain variations of the algo, i ran out of memory even in the backtesting environment (research environment was literally not useful for my purpose because of its memory constraint). Does Q2 has the same amount of memory per user? or has that been changed?

Quantopian 2 has unfortunately not produced much of a performance improvement for me. Most of the time is still being spent in doing queries for fundamental factors. Good thing my production algorithm is also fundamentals oriented and trades very infrequently, it gives me time to put an update in place.

I too feel the transition could have been better handled. There was an announcement of an upcoming change, and then the change just happened without a firm date first being announced. Lots of web sites put banners up on every page to inform users of an impending major event. I wish that practice had been implemented in this case too.

Sunil

I had implemented a custom commission model to simulate tax computation. That now produces a catastrophic failure. This was of course going well outside the beaten path in using the quantopian API... Has anyone gotten custom commission models working with quantopian 2?

This is the entirety of the error message I get:

 Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.  

Sunil

Congrats on Q2!
Read thru Q2 documents and it looks good to me.

The Q2 changeover has been handled poorly. The communication has been unclear, AFAIK there was no extensive period to let people live test their algos and I found out that my algo would be switched to Q2 from Q1 with very little notice.

When I tested my algo on Q2, it had one error which caused it to stop (thankfully). Once that error was fixed, the algorithm 'ran' but with vastly different performance than in Q1; a change in the behavior of close_price (it seemed to sometimes return NaN whereas .price didn't) led to some pathological behavior in my algo. Luckily, I found it in testing but if the automatic porting 'succeeded' and didn't fail due to the detected error, my algo would've had the potential to lose a bunch of money for me for tomorrow's trading.

Even still, after fixing that bug, backtesting in Q1 and Q2 gave drastically different P&L numbers, even after setting the slippage and commission models to be the same. I played some whack-a-mole but couldn't resolve all the differences so I put it off -- and now I check my email and see that Q1 is fully gone. I still have no idea where the difference comes from -- I imagine some of it is from changed data but have no proof.

I understand that you guys are primarily trying to be in the hedge fund business and I'm not a paying customer so your main focus is probably elsewhere, but here's how I think the migration should've happened.

1. Give a clear schedule of when the Q2 will become available for testing, for live trading and for when Q2 will be disabled. At least a week or two of lead time between announcement and the mandatory migration would be nice.
   
2. Provide some tools to reconcile between Q1 and Q2. AFAIK, the changes are the default slippage and fee models, source of data and some code changes. You should have the ability to run a Q2 algo using Q1 data and slippage models, to verify that the performance is exactly the same; if it's not then there is some code error in Q2 or algo which needs fixing.
   
3. Allow people to opt-in for live trading on Q2 before it's mandatory. In particular, keep old algos running on Q1 and have new starts use Q2.
   
4. Do not migrate existing live algos to Q2 automatically. When you want to deprecate Q1 (which should be a while after Q2 is available for live trading), you should do something like log the currently running Q1 algos out and force the users to acknowledge changes when logging back in.
   

Have these changes benefited the research environment? For example, does get_pricing now return dividend adjusted prices in the same way that the history function does in backtesting?

I have a bunch of posts on testdrive that are still open issues, in my opinion. It would be great if they could be addressed there. Alternatively, I can migrate them over to what is now Q2.

A specific test case was to confirm:

Your algorithm can price and trade all securities, not just a few hundred at a time.

On https://testdrive.quantopian.com/posts/memory-error, I tried ordering all stocks, and ran out of memory. I don't think the Q2 platform actually supports 8000+ securities.

would have been nice if we had a more simplistic way of transitioning our code..this has been troublesome. Not happy with the time it has taken.

Well done!

Thanks for all your feedback. We apologize for the disruption this transition has caused for some of our community members. We hope to not have to do launches with this impact frequently (or ever) but should we have to do it again we will take all of your suggestions into consideration.

Regarding the performance improvements. Quantopian 2 has the potential to be much faster than Quantopian 1, but it does require algorithms to be written in specific ways. We’ve explained those best practices here. We didn’t touch the speed of ingesting fundamental data. Fundamental data is still slow. It’s something we know we need to improve.

Shi - Q2 has the same amount of memory allocated per algo as Q1. On Test Drive we found some memory bugs. We fixed them, but it is possible there are more. Please contact support with any issues you see.

Sunil - There were some changes to the custom slippage API. The help docs have been updated, and can be seen here. The TLDR is that the parameters required in process_order have changed as well as what you return.

Additionally, here is an example of custom slippage in action.

Shawn - The research environment has not yet be updated to reflect the Quantopian 2 data or zipline version. We will be working on both in the coming days/weeks.

Grant - We might have a disagreement over what it means to "support 8000+ securities." There's no real-life use case for putting 8000 securities into your portfolio. What Quantopian 2 does enable is huge baskets of stocks that are selected from the full 8000, selected at any time. We've run successful tests with 1000 long names and 1000 short names. That's similar to the largest portfolios in quant funds today. Supporting that use case, and other related ones, is what we're interested in. We're not aiming for an arbitrary goal of holding one share of everything. I'm quite comfortable saying that we support 8000+ securities without supporting the buy-one-share-of-everything use case - it's not a useful, meaningful test.

The practical, technical limitation on how many stocks you can actually hold at any one time is related to the liquidity of the securities. The expensive computation in this area is doing the price lookup. Looking up a price for "now" is very fast. Looking up historical prices is more expensive. Illiquid securities trade infrequently, and the farther back in time we have to look for the most recent price, the more the computational cost increases. Baskets of 1000 long and 1000 short are possible on highly liquid stocks. If you trade in illiquid securities, the baskets will need to be smaller. Of course, constructing a basket of illiquid stocks is probably a bad idea - again, it's a use case that we're not aiming to support.

Alex, you are spot on.

What I see here is the equivalent of when taking off a bandage and you are encouraged to just "rip it off, because it will be over quicker."

BTW why on earth would you change the order of parameters when writing a method to replace an existing method? Specifically the "history => data.history" change. I understand that you needed to add a parameter. Adding it at the front or back is fine, but changing the order of the parameters that did not change, Why? It just makes conversion that much more tedious.

@Karen, the problem I encountered is not with custom slippage but with custom commission.

Sunil

Paul - we changed history for consistency and a better, easier to understand API moving forward. Sorry for the inconvenience.

Sunil - we didn't make any changes to the custom commission model. If you are willing to share your code with [email protected] we can help debug it.

@Karen Just wanted to make sure we can still run history in 'daily'frequency, and based on the mitigation guide, it looks like we can. For some reason, I thought only minute data was available on Q2? Perhaps I heard wrong?

Thanks so much.

Karen, not to bash you guys, but, which is worth more, "consistency and a better, easier to understand API" or reducing the workload on your user community? I only have a few smallish algorithms to change, but I can imagine others have a lot more that will need to be changed.

Having said that I am enjoying learning a new language and testing my concepts.

Shoaib - daily data is still available through the pipeline API and in history. The backtests are only running in minute mode (which you can read about here).

Great to see the product continues to evolve and get better. (-: These changes make things more intuitive for me. Thanks for all your hard work making it happen!

Hello Karen,

I posted my "order everything" example here:

https://www.quantopian.com/posts/error-when-ordering-all-stocks

Indeed, it will handle orders for ~ 2000 securities. When the number is unlimited, I no longer get a specific memory error, but a generic one. Shouldn't the error messages be the same between testdrive and quantopian?

No need to burden this thread with another response, but it would be nice for someone to have a look at my post and explain why the code crashes. It is not intuitive that ~8000 orders should be a problem. How much memory could each order consume?

just a heads up. I just realized that if I stop using fetcher data, my simulation time drops from 3-4 hours to a couple of minutes. Great job Q.

Hi Quantopian,
I just wanted to give some feedback on 2.0:
As I'm quite new to algorithmic trading and Quantopian I had a really hard time to get going in the beginning, so I wasn't really happy to see a new version with major changes being made. Therefore I read quite alot about it to actually understand the changes and finally I really like the new version ( especially the fact that there is no universe anymore). It further encouraged me to keep learning and hopefully sharing my first algo with the community in the near future.
love the website and all the free content that is being provided

Is this change going to reflect into the github zipline?

Karen / Dan - first off, very big congrats on v.2!! Lots of great improvements including my favorite so far of way faster algo backtesting.

Question - Will you guys be posting a custom volume slippage in the help section - is it possible to override in v.2? Getting massive disparities on re-running the exact same code and pretty sure it is due to volume share slippage since dealing with sec's with lower than ideal daily trading volumes.

I tried

slippage.VolumeShareSlippage(volume_limit=0.25, price_impact=0.1)  

in initialize (help still says to run initialize so not bothering with the class method), but when set breakpoints, and pull the simulation parameters, it is still using v.2 slippage model

instancemethod: <bound method AlgorithmProxy.assert_keywords_and_call of AlgorithmProxy(  
    capital_base=100000.0  
    sim_params=  
SimulationParameters(  
    period_start=2016-01-17 00:00:00+00:00,  
    period_end=2016-04-20 23:59:00+00:00,  
    capital_base=100000.0,  
    data_frequency=minute,  
    emission_rate=daily,  
    first_open=2016-01-19 14:31:00+00:00,  
    last_close=2016-04-20 20:00:00+00:00),  
    initialized=True,  
    slippage=VolumeShareSlippage(  
    volume_limit=0.025,  
    price_impact=0.1),  

Hi Umar,

Is there any chance you're forgetting to use set_slippage()? If not, could you share a bit more of your code for some more context?

yes I did ... realized it short after - sorry, I thought I deleted my 'jump the gun' question shortly after posting it... thanks Jamie

Jamie McCorriston 6 minutes ago
Hi Umar,

Is there any chance you're forgetting to use set_slippage()? If not, could you share a bit more of your code for some more context?Blockquote

I have a lot of problems with the interface

1) I can only view the logs of the backtest
2) I can only see backtests that are finished, not running ones
3) it seems that the backtester is sometimes very slow and then fast....

Anyone experiencing problems or are you all in coding heaven?

I can confirm that there seem to be a problem with algorithms using custom commissions that Sunil reported above.
I used a variation of approach suggested by Grant in this thread https://www.quantopian.com/posts/removing-cash-from-portfolio-to-pay-taxes to better model IB commisions ($10 minimum monthly fee) and interest charges on margin loan.
It worked fine before update, but gives following error when running on Quantopian2:

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.  

I tried debugging it, but the problem seem to happen outside of the algo.
Here's a sample code where the problem is reproduced:

# From https://www.quantopian.com/posts/removing-cash-from-portfolio-to-pay-taxes  
def initialize(context):  
    set_commission(CashOut())  

def handle_data(context, data):  
    cash = context.portfolio.cash  
    record(cash = cash)  
    order(sid(38986), 1) # VGSH  

class CashOut(commission.PerTrade):  
    """  
Calculates a commission for a transaction based on a per  
trade cost.  
"""

    def __init__(self, cost=5.0):  
        """  
Cost parameter is the cost of a trade, regardless of  
share count. $5.00 per trade is fairly typical of  
discount brokers.  
"""
        # Cost needs to be floating point so that calculation using division  
        # logic does not floor to an integer.  
        self.cost = float(cost)

    def calculate(self, transaction):  
        """  
returns a tuple of:  
(per share commission, total transaction commission)
"""
        if transaction.amount == 0:  
            return 0, 0  
        if transaction.amount == 1:  
            return 1000, 1000 # $1000 for testing

        return abs(self.cost / transaction.amount), self.cost  

Has anyone run into import issues?? Can no longer use import json.

From what I've played with so far, the backtester is now way faster in minute mode. Good job! I had to do some recommended optimization like calling history less often and not executing code minute by minute unless necessary. Using fundamentals is unfortunately slower but tolerable.

Every time I look through past code for instruction, I am finding that it doesn"t work. Maybe there isn't any up to date instruction now. I'm such a noob. Glad 2 is a better system, just a harder learning curve for me now.

Every time I look through past code for instruction, I am finding that it doesn"t work. Maybe there isn't any up to date instruction now. I'm such a noob. Glad 2 is a better system, just a harder learning curve for me now. Wish I could say I was no longer irritable .......lol.