I am unable to sign in on the new version. Are you having technical issues?

Hi Shawn ,
The test drive environment is a complete new (and temporary) environment and requires that you create a new account. You can do so here.

I am able to put in my e-mail and a password, however I cannot click the "I accept the Terms of Use and Privacy Policy" check box (it cutely jiggles when I click "Get started" but it won't let me put a check in the box).

Also, your little "Q" in a rounded-corner box symbol is directly over "Terms of Use and Privacy Policy" -- it needs to be shifted down toward the bottom of the page.

I'm using Ubuntu 14.04 LTS & Firefox 45.0.

Never mind...it finally worked!

Aside from Zipline changes, is the python version different from Q1 to Q2?

Why can't I see daily/minute mode when running backtest? Is it always minute mode by default?

fantasmic!

any idea when Q2 will be fully deployed?

Pravin,

Check the updates. No more daily mode. Only minute mode but just as fast!

I imagine this will be fully deployed with a couple of months. I am sure the more people use it and report any bugs the quicker it gets live.

Beautiful. I presume the content after migration will use the new comissions/slippage default values too? Removing the not-so-fun task of writing order execution logic to avoid slippage is a huge help.

Secondly, my cloned algo fails after 5 months of backtesting (Too much memory used). That 5 months of backtest ran incredibly fast in Q2 though.

The same algo can run a full length backtests on Q1, even though it takes hours.

more fast,less function

Hi folks,
I'm thrilled to see the enthusiasm and thanks for the testing you have done so far. To answer your questions,

Kaveh - the version of Python has not changed with this update.

Pravin - daily mode has been removed, you can read about why here.

Miles and Shoaib - The hope is to launch in the next week or two, but it does depend on how the debugging goes and what feedback we get.

Kayden - We haven't nailed down all the details of the contest with Q2, we'll be working on that this week and will share more information as we make decisions. I'll be in touch to talk about the algo that is failing with memory issues, it will be helpful to know more.

James - if you find functionality missing, please let us know what.

All the feedback is very helpful, so thanks!

Will Q1 and Q2 coexist for some time, or should i assume until my algo are Q2 compatible i will not be able to live trade ?

Hi Florent,
Most Q1 algorithms will run on Q2 for some period of time. Algorithms that are currently live trading should continue without a problem. However all new algorithms must use the Q2 API in order to be launched into live trading. Details can be found here.

This is great. Thanks for the quick reply, and most importantly thanks for addressing so much of your community's feedback in this new release of Quantopian.

This part of the summary seems like an error seeing as it's identical to "Default Slippage Model"

https://testdrive.quantopian.com/quantopian2/summary#default-commission-model

In the absence of a specified list of stocks can you take the list of stocks from the stored universe data structure which can be used for the basis of tracking what is been traded in the algo.

Also support a DF orders which you can specify parameters for multiple securities at the same time

I took a test drive with Q2 but unfortunately I've experienced the same memory problem, I described in this post:
https://www.quantopian.com/posts/error-in-fundamental-data#56f5da99f44e1db34b00064e

What about the feature request, I proposed here?
https://www.quantopian.com/posts/pipeline-api-feature-request-retrive-single-data-points-instead-of-a-full-array-of-data-window-length

It's a small API change but very important for algorithms that are based on data that do not change daily.

Constantino, refactoring and improving the performance of the fundamentals database is definitely on our backlog. We certainly want to improve how we store and deliver the data for use through applications like pipeline.

We appreciate the feedback.

Thanks
Josh

Good to known! I added you as a collaborator to my Piotrosky Score implementation as Pipeline.
If you like, you can use it as benchmark, when improving the performance and delivery of fundamentals data.

If possible, I'd also like to contribute to zipline as a programmer.

Hi ,

I'm just testing one of my algos . So an order for TLT failed to fill at EOD. Will the algo liquidate it the next day ?

 WARN Your order for -14079 shares of TLT failed to fill by the end of day and was canceled.  

Neeraj - thanks for catching that. We will fix.

Suminda - Quantopian 2 removes the concept of the tradeable universe and makes pricing and volume data available for all securities at all times. The data object is no longer an iterable containing your securities; it is now an object containing several API functions including current, history, can_trade, and is_stale.

Since you can no longer iterate over the implicitly defined tradeable universe, you should now be keeping an explicit reference to any security that you want to trade. The most common ways to do this will be to maintain lists of securities that you might want to trade in the context object (e.g. context.longs, context.pipeline_assets).

You can read more about this here.

As for order objects accepting as dataframe or list, we agree and will work on it in the future.

Lionel,
Your algorithm will not liquidate if an order is not successfully filled. It will continue running, but that order has been canceled. Orders fail to fill if there is no volume. You might consider using the pipeline dollar volume universe to get more liquid stocks.

Hi Karen

TLT is one of most liquid ETFs on the market @ 9,436,080 avg volume . Q2 is still in beta , so I will do more test later. I never got that error when testing in Q1.

Thanks

The end of day cancel is new with Q2, so it wouldn't have happened in Q1. However, if this is a very liquid asset, it likely shouldn't happen. I'll check into it.

The bolded line in the code fragment below is generating this error msg in the Q2 test drive simulator:

"InvalidKeywordArgumentToAPIFunction: 0060 Invalid keyword argument to API function."

for long_stock, long_weight in context.long_weights.iterkv():
if long_stock in data:
if get_open_orders(sid=long_stock):
continue
if long_stock in context.dont_buys:
continue
order_target_percent(long_stock, context.long_leverage * long_weight)

This code fragment is take from:
https://www.quantopian.com/posts/quantopian-lecture-series-long-short-equity-algorithm

I looked through the new API docs and the code looks correct to me.

Thanks...

I hope this isn't a spacebar heating inquiry :) In my Q1 code, I do something like

    data.history_cache = data.history(assets, 'price', 100, '1m')

and then use data.history_cache instead of data.history for improved performance. Currently however, I get a

AttributeError: 'zipline._protocol.BarData' object has no attribute 'history_cache'

error (presumably the BarData class doesn't allow the addition of new fields). Is this a permanent change or will it be fixed? I'm concerned that once you upgrade everything to Q2, my algorithm won't run without me making changes and putting the cache in a different place would require changing some data flow in my strategy, which is possible but will take some effort.

So I ported my main algorithms to Q2. Two issues:

I believe I'm having the same problem as Lionel above. It is taking longer to fill some orders than I would expect. If I read correctly somewhere that the fill rate is supposed to be 2.5% of average dollar volume, I think you may have a decimal point in the wrong place somewhere.

Also, I ported one algorithm that makes heavy usage of fundamental data in a pipeline (a pipelined version of Magic Formula vs Acquirer's Multiple). It is taking crazy long to run. Running in Q1 in daily mode was not speedy, but is considerably faster than this. I think until you can either improve the speed of fundamentals data, Q2 isn't ready to go out.

Hi Karen

It looks like schedule_function does not work nether in Monthly mode nether in Weekly mode.
I agree with Lionel and Tar Xvzf that 2.5% of 1 minute volume limit is unrealistically tight especially for open end instruments like most of ETF .
Where from that number came? I was trading TLT manually and do not remember that market order for 1000 shares took more than 3 seconds to fill.
I agree with Lionel that Q2 should accept and fill market_close() orders, just simply start running schedule_function 1 minute before.
Otherwise what it for:

schedule_function(trade, date_rules.every_day(), time_rules.market_close())  

I also strongly support Grant's request for Good Till Date or Cancel order option.

@John Friedrich. Remove sid= from your function call. So you call should be - if get_open_orders(long_stock): That should solve the error :-)

Well, I'm confused. Does get_open_orders(sid=long_stock) work or not? Per the attached, it does work. However, recent testing showed that it does not (see my report on https://www.quantopian.com/posts/quantopian-lecture-series-long-short-equity-algorithm). Did something get changed, creating a bug, and then get changed back?

John, Shrikrishna, Grant: for get_open_order, just pass a security without naming the parameter (e.g. get_open_order(sid(24))). We've renamed the parameter from sid to asset to be more consistent with other parts of our system, but you can just pass the security and it'll work. We've updated our help docs accordingly and will ship that out soon.

Alex S: adding custom properties to data is something that we supported by accident in Q1. If you use context.history_cache = ..., everything should work. Unfortunately adding custom properties to data is not something for which we've built a compatibility shim, so it won't work in Q2.

Tar Xvzf: Q2's default slippage model lets you take 2.5% of the volume for a minute bar, instead of Q1's 25%. We've found that to be a much more realistic value, as the previous model was creating some unrealistic price impacts. This is also the value that our research team uses to evaluate algorithms for allocations. You can always customize your slippage model, and Q2 now gives you the full data object so you can write much more complex slippage models that incorporate data from other securities if needed.

As for fundamentals in pipeline, we know that is currently a very slow part of our system and we are actively working on improving it, separately from Q2. Your algorithm shouldn't be noticeably slower in Q2 than in Q1's daily mode if it makes good use of schedule_function. We're happy to help look at your code if you'd like.

Vladimir: We had some bugs in schedule_function that have just been fixed on testdrive. Can you try again and let us know if you have any problems?

Thanks for all the feedback!

Hi Jean

I got schedule_function working in Monthly mode and in Weekly mode. Thank you.

Slippage is where our backtester calculates the realistic impact of your orders on the execution price you receive.

Setup I tested:

set_slippage(slippage.VolumeShareSlippage(volume_limit=0.025, price_impact=0.1))-default
Initial capital $100000 not a million.
schedule_function(trade, date_rules.every_day(), time_rules.market_close(minutes =60))
(I gave my algo 60 minutes to fill the order)

And every day I getting something like this:

2007-01-05 WARN Your order for -1121 shares of TLT has been partially filled. 933 shares were successfully sold. 188 shares were not filled by the end of day and were canceled.  

Is this impact realistic?

FYI: Average trading volume for TLT is 643 000 per hour- 600 more then order

https://drive.google.com/open?id=0B10uueaFeSYvN3FSRWVOT3lfbWs

Vladimir:

Our data source shows that TLT has a total trading volume of 847,900 on 2007-01-05 (your number was for today, not in 2007). In the last hour of 2007-01-05, TLT traded 37,800 shares. 2.5% of that is 945 shares, which is pretty close to what you saw.

Again, feel free to write a customized slippage model if you prefer.

Hope that helps.

Jean:

I am not talking about a bug in Q2 calculating volume limitation but about the reality of 2.5% limitation in default slippage model.
Where from that number came?
Why in reality the order may be executed in 3 sec but backtester needs more than hour for that on one of most tradeable instruments?
What also I see now in Q2 that algo is dramatically loosing control of assets allocation even on initial capital of 100000.
Slippage model is the ruler.
We need to find more balanced solution and that solution should be in default slippage model.
```

when do you expect Q2 to be in production? alternatively, i'd like to submit algorithms for Q hedge fund, when will that be available?

@Jean: Re [We had some bugs in schedule_function that have just been fixed on testdrive. Can you try again and let us know if you have any problems?]

I can confirm that from my end it looks that the problem is fixed from the only strategy I've ported to Q2, and that as of today, results seem aligned with Q1. I will report in couple days after further testing and validation. (quite stressful for live traders to see such issues and the big push to move to Q2)

It took a couple of hours of work to port my algos to Q2, but it all seems to be working now. It's very much faster, and I am looking forward to trying out some new strategies using the new lack of universe restriction (opening gap trades, anyone??).

Re: the slippage model, someone really should write a proper square root or 3/5ths impact model, to drop in instead of their quadratic one. It could even have a sideline for ETFs where even square root would overstate one's impact. Doesn't help contest folks, but it would be a big plus for people who are simply trying to backtest microcap strategies.

Opening gap trades are kinda sorta supported. Q2 trading doesn't start until 9:31 am. And pipeline only runs in before_trading_start (correct?), so one would need to use history to identify the gapped stocks. I'm not sure yet how many stocks/data Q2 will support. See:

https://testdrive.quantopian.com/posts/memory-error-1

For ~8000 securities and a 20 day minutely window, there isn't enough memory. But if the gap analysis only requires a few data points per security, then maybe it'll work. There's no way to tell how much memory has been used and the total available, which would be a nice feature to avoid crashes. Maybe it is a matter of pre-allocating the data array in context?

Well, for starters, I would probably only want to look at the top 2500 most liquid stocks anyway. Identifying a gap at 9:31 would only require two or three data points each. Then, on those that gapped (maybe 100-200?), you probably have time to request more substantial history to fit some models to see if the gap is tradeable.

Yeah, but it'll be easy to get burned without better access to what's going on under the hood. It would seem straightforward to provide guidance on the amount of memory available and some tools to manage it. With Q2, it appears that one could access history in before_trading_start so the temptation will be to spend 5 minutes crunching all 8000+ securities using minutely data over a substantial window (390 X N, where N is 5-90 days).

Is TestDrive shut down now ? I am having problems starting my backtests.

Sorry that should have read "shut down", I've corrected it.

It's still working here...

Test Drive isn't shut down. We plan to keep it up beyond the launch of Quantopian 2 so people can move algos back.

We did see that you had some issues earlier today. We are looking into what happened, but things seem to be working again. Sorry for the issue!

Hi Karen, congratulations on the new API, I think it's great.

Here is a very small but annoying issue I have: when I hover over an API keyword in the code editor, the tooltip window shows up at the wrong place and it's impossible to access the contextual help directly. The window disappears by the time I try to get the cursor on it.

Google Chrome on Win10.

Cheers

Hi Karen, I'm very satisfied with the Q2 API! Removing the universe limit and allowing historical data everywhere is a very useful change! I'm migrating all my algorithms.

I've a question: would be possible in the future to use the function data.history() also with fundamental data? I mean something like this to get the last 4 quarters:

hist = data.history(assets, 'morningstar.income_statement.total_revenue', 4, '1q')  

or for example like this for the last 5 years:

hist = data.history(assets, 'morningstar.income_statement.total_revenue', 5, '1y')  

Thanks and keep the good work going!

Hi Constantino,
I'm glad you like the new API.

The best way to get historical fundamentals data is via the pipeline API. We do not have plans to add it to data.history at this time. With pipeline it can be challenging to get just the last 4 quarterly numbers, which is on our radar to make easier, but you can get the data out.

Hi Karen,

thanks for your prompt reply!
I've already tried with the Pipeline API but unfortunately a memory problem occurs. I already talked about that with Josh Payne and described the problem in this post (making another feature request):
https://www.quantopian.com/posts/pipeline-api-feature-request-retrive-single-data-points-instead-of-a-full-array-of-data-window-length

In this moment I added also you as a collaborator to my Piotrosky Score Pipeline implementation. Maybe - as I already said above to Josh - you can use it as benchmark, when improving the performance and delivery of fundamentals data.

Thanks!
Costantino