Here's a cloneable copy of the Optimize API tutorial notebook.

@Karl: I think it might have been a transient issue as our servers rolled over. Are you still seeing the errors this morning? Existing algos using the deprecated names should continue to run fine, aside from the deprecation warnings.

@Karl just to make sure it's clear, the expected argument to TargetWeights is a dictionary mapping individual asset objects to their target weights. So, for examples, a usage might look something like:

# Allocate 25% long to AAPL/MSFT and 25% short to TSLA/IBM  
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})  

There's also an alternative constructor, TargetWeights.equal_weight for the common case where you have a list of assets that you want to allocate to with equal weight:

# Allocate 25% long to all four assets.  
my_objective = TargetWeights.equal_weight([AAPL, MSFT, TLSA, IBM])  

In writing up this example, I noticed that equal_weight is missing from the docs for TargetWeights. I'll make sure that gets updated.

One thing we don't currently have a convenience method for is the common case of wanting an equal-weight long-short portfolio. That would probably be a nice addition to the API.

@Karl your example wouldn't currently work as written, but you could write a small utility function do something like it easily enough:

def equal_weight_long_short(longs, long_weight, shorts, short_weight):  
    weights = {asset: long_weight for asset in longs}  
    weights.update({asset: short_weight for asset in shorts})  
    return opt.TargetWeights(weights)  

Hi Scott,

Are you still planning to put the optimize API on a public github repository? Will it be part of zipline?

Thanks,

Grant

Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders.

How is this implemented in the Optimize API?

What happens to equities that are in current portfolio but absent in the TargetWeights specification.

Say: My current portfolio positions are {GOOG: 0.5, X: -0.5}

If I specify my targetWeigths as follows:
Case A)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})

Will GOOG, X be automatically closed since they are part of the universe(based on Abhijeet's definition). Or do I need to add GOOG, X to the TargetWeights explicitly to close them as in Case B below.

Case B)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25, GOOG: 0, X: 0})

Will Case A and Case B result in the same resultant portfolio (i.e addition of GOOG:0, X:0 has no effect and is implicitly assumed)?

Any feedback on my question above regarding the details of how the Optimize API "accounts for open market orders when determining target positions"? I'm interested in all use cases, including backtesting, Q paper trading, and real money (& paper?) prime broker trading.

@Abhijeet Regarding the name changes "quantopian.experimental.optimize as opt should now do import quantopian.optimize as opt.
The TargetPortfolioWeights objective has been renamed to TargetWeights..."

Not sure what is the purpose of throwing deprecation warnings and make the community make a new revision of their existing algorithms when the corresponding changes are purely cosmetic. There should be an easier workaround on the server side by redirecting the old namespace and functions to the new nomenclature when we expect identical results.

@Leo - the old names continue to work (for example, you could still import from quantopian.experimental.optimize instead of quantopian.optimize at the moment, and they will both do the same thing). However, we recommend using the new namespaces, since Optimize is no longer in its experimental release. The main purpose of the deprecation warnings is to help authors upgrade to the new names, and also to indicate that the old names may not be available in the future.

@Abhijeet, I will definitely upgrade to the new namespace as I wont want my algos to stop working in the future.

My concern is that we lose some months of out of sample performance if the old namespace are done away with at some point in future. To clarify, if I wrote an algorithm 3 months ago with order_optimal_portfolio and now I no longer have the guarantee that that algo will run in the future, then I essentially lose 3 additional months of out of sample and get reset to T=0 (with my transformed algorithm with the new namespace) to start accumulating out of sample performance.

Seems a waste of a few months if releases are not backward compatible and we sunset the old namespace at some point.

It would be nice if namespace transitions are handled seemlessly by Quantopian, automatically transitioning to the new namespace and continuing to function as before. I have seen that in some old threads with a new tab suggesting "Migrated to new ...". is that not possible to setup for order optimal portfolio?

Leo, the good news is that I don't think you're really going to lose any out-of-sample performance. If you update only that code, and the performance of the updated algo matches the previous version, then the effective out-of-sample date is when you finished the first version. If you're getting good performance 6 months after the first finish date - let's talk!

The other thing to keep in mind here is that you are using code that we had categorized as "experimental." Experimental APIs are exactly that, and don't come with the same level of commitment as our regular API. Now that optimize is generally available, you can expect a higher level of stability from the API.

@Dan, I didn't realize revisions start date can be that of original algorithm if performance of the algorithm doesn't change. Since you maintain a copy of the code when each backtest is run, I assumed that the backtest essentially crystallized the start date (and code). Thanks for clarifying. I will go ahead and make the code change to the existing algorithm, instead of creating a new algorithm which I was initially planning to do (and it appears that mistake would have made me lose the out of sample period).

You are correct that the backtest date is the date that crystallizes the start date of the out-of-sample period. But let's imagine how this could unfold:

You ran a backtest with the experimental API in June. In December, six months having elapsed, it will get automatically evaluated. If it meets our criteria, we'll contact you. You'll mention that it used the experimental API, but you have an updated version in September that is otherwise identical. At that point we can all discuss the merits of using the updated code, and whether we should use it or wait another 3 months for it to season. If the performance of the two algorithms is identical we would take the new code and use it instead.

When you think about it, we're always making judgement calls like that. Every time we update anything, Quantopian APIs, pandas, numpy, sklearn, etc., we're potentially disrupting the effectiveness of the out-of-sample period. There are some parts of this process that require good judgement.

@Dan, my intent was to state that it would be nice if the experimental api is retained till at least end of Mar 2018, so that atleast one of the algorithms either the existing experimental version or the version transformed to the new namespace (and a backtest kicked off now) is always in the running for an allocation at any point in the future (after they both have reached their 6 month out of sample periods respectively). A few months (of delayed algorithm consideration) is not a big deal but giving the existing experimental versions some grace period before the newer namespace versions can be 6 months out of sample will mean no lost out of sample time for the algorithm developer with regard to consideration by your automated discovery process. I am relatively new to Quantopian so not sure how much time there is between deprecation warnings and actual sunset (i.e apis stop working), so it is possible I am more concerned here than the situation warrants.

Oh, yeah, that will be fine. We don't have any plan to remove the old namespace, and it will be around for many months.

@Dan / @Quantopian,

Is the Optimize API available for use in Zipline / or available on Github? Thanks,

How do I change the data fed to the optimise API so that instead of using a z score (which I fully don't understand), it uses something tangible like pe_ratio as the input to the algorithm?
Also, the optimise API seems to be erratic in performance -meaning it does seem to give good results, but can also crash big time and correlation with market swings is pretty high despite opting for a market neutral optimiseer (1 of the options to the algorithm).

thanks
-kamal

Hi, just another friendly enquiry about whether the Optimize API will be open sourced. I understood from this comment on the CVXPY project's github that you wanted to do this. Now that the license issue has been resolved for some time, is open sourcing this library still on the roadmap?
Regards, Mark

Ping? Is the Optimize API going to be open sourced anytime soon? Not too fond of using a black box for something this important.