Maybe the lecture on Kalman Filters?

Nothing that advanced. But will take a look.

I'll rephrase my original question: In Quantopian, for a moving average cross over example strategy, I'd like to investigate the optimal number of bars for calculation of the moving averages in combination with various stop, targets and volatility metrics.

I don't want to do this in situ/in realtime. I want this to be done separately and outside of realtime trading. I simply want to run lots of backtests and output the peformance data from each run (or print to file). Ideally I would like to specify rules as to how next inputs are selected (e.g. genetic algo based). I may also want to run this alongside rounds of walk forwards. From this, I will then choose parameters to be applied to any realtime algos I have.

This isn't anything new or complicated. I just want to know if parameter optimization akin to those engines in many other platforms (Tradestation,Multicharts, Ninjatrader, MT4) is possible in Quantopian?

How would I run an optimization involving 1000s of iterations where I need to find the best fitted inputs for 2x moving averages, a stop and target input, a volatility input against an objective function of my choosing in Quantopian?

Hi ww3361,

i read with interest your comment that you are an " ...experienced professional developer of automated trading systems" and that " ...parameter optimization is an essential strep in [your] R&D process". With that in mind it is just my guess that there is probably a significant difference between the type of systems you develop and the ones that Q is looking for, especially in terms of the time-frames and other constraints involved.

Many of us started out playing the "optimization" game with commercial platforms like Metastock, Tradestation, MultiCharts, AmiBroker, etc,and found:
1) These kinds of automated optimizations are great for running quick scans to see the ranges of parameter values that DON'T work.
2) Optimization is indeed useful for building Short-term systems that will require frequent re-tuning & re-optimization, where this can feasibly be done.
3) For Longer-term systems development, "optimization" is something of a dirty word equivalent to curve fitting and usually results in great In-Sample results and associated very poor OOS results.

The problem for "optimized" systems is that as soon as the market regime changes significantly compared to what has been seen in the past, the system will then inevitably UNDER-perform, irrespective of how much care you might have may have taken with Walk-Forward, etc. (See for example published advice by experienced developers like Perry Kaufman).

The situation with Q is that the algos, once implemented, have to be "left alone". (To anyone from Q, if this situation has changed recently then please feel welcome to correct/advise me if necessary). With this in mind, systems designed to maximize ROBUSTNESS under a wide range of un-known market conditions will inevitably work better in the Q environment than "conventionally optimized" ones.

I wonder if Q's lack of provision of a easy-to-use 3-D display optimizer of the kind found in the standard commercial packages might well have been a completely deliberate choice on their part.

Cheers, all the best, TonyM.

Thanks Tony.

I was really hoping to avoid a conversation on the merits and pitfalls of optimization. That topic is huge. Oh well.

I categorically disagree with your statement.
"These kinds of automated optimizations are great for running quick scans to see the ranges of parameter values that DON'T work." I’m looking to have a play with Quantopian’s futures datasets and get experience with python for the medium frequency trading I do. Purely for my own education.
Totally agree that one can end up lost in meaningless in sample simulations if not careful. But, for me, it remains a vital step on the long and painful R&D road. I'd be happy to discuss further over a coffee at my office if you like. Just PM me. Characterising a complex optimisation space literally guides the route we take when we evolve a raw idea. Even if an algorithm has an element of in situ machine learning, I still want to have a play with optimising inputs at discreet intervals to investigate the various permutations, in hindsight, as to how the algo could choose to learn in real time.

My findings from doing this for almost 10 years is that the following steps have worked well:

-Datasets.... As granular and as clean as possible.
-Simulation..... as close to real life as possible.
-Initial idea mock ups....early signs of raw performance when applied to a basket. Otherwise, unless dramatic findings are made during optimization, drop.
-Optimization.... identify variables that drive performance and those that are redundant. A well designed optimization will provide useful information when it succeeds or fails. It can also point the user in the direction of rule tweaks that yield better efficacy. I frequently end up with signals that are completely different (even inverse) after months of optimization.
-OOS testing and cross validation...... The signals need to perform on unseen data and with frozen inputs applied to a breadth of related instruments and timeframes.
-Live trading: diversify your parameters as much as possible (this is made so much easier when you've spent so long in the optimization space!) and deploy on tiny size
-Scaled up to match rest of portfolio of systems (adjusted for correlation) after about 12 months.
-Periodically run discreet reoptimizations (for me that's twice a year).

Anyway, it's complete opinion. I'm only in this for profit.

Regarding the question: I'm guessing the answer might be a no?

Hi ww 3361, Tony,
@Tony, glad to see you are still around!

@ww3361, the answer your question is yes and no. The answer is such because the Q framework environment has some limitations:
1. There are limited number of whitelisted opensource library for optimization, i.e , sklearn optimize, cvxopt
2. There are compute time limits imposed and thus optimization of 1000 stocks with maximum iteration of 100 each to optimize a couple of parameters will almost certainly trigger a compute timeout error.

To make it work means overcoming these two limitations most specially (2).

Thanks!

Hi @James, thanks for your nice "welcome back". Yes, i'm very much still around, but have been out of the loop here at Q since the end of last year. Looking forward to getting back into it again ASAP. Cheers!

Hi @ww3361,
;-)) well even if you categorically disagree with my bold-faced statement that "automated optimizations are great for running quick scans to see the ranges of parameter values that DON'T work", actually I think you may find that we are both very much on the same page!

I deliberately chose to express my comment in a way that may seem somewhat provocative, and I will explain why. The optimizers in the commercial packages are great tools for sorting through a lot of ideas quickly and putting them into to different groups, right? I do that too. As we both know, some of those ideas get kept, some get quickly thrown out. For me too, this is also usually an important step on the "long and painful R&D road". However what I see lots of beginners doing (and I used to do it myself) was working and re-working the optimizer to get the "best" solution and then imagining that I had something real ... instead of just a useless illusion based on over-fitting. From your follow-up comments, you are obviously way past that. Nowadays, after a few decades, I like to work some of the conventional ideas of trading system design & development from a "backwards" perspective, i.e. filtering to get rid of as many bad ideas as fast as possible, rather than filtering to try find good ideas. Its mostly just a matter of perspective & efficiency, and in the end I think we are at a similar place.

I concur with all your comments about your own findings, with just one possible exception. That is the topic of conclusively identifying variables that drive performance. In addition to the fact that this may change over time, particularly with regard to Style (e.g. one year "Growth" is the most fashionable, another year it is "Value", etc), this problem mostly (I think) comes down conceptually to identifying Information Content based on some form of Feature Extraction process. Yes, I'm in this for the money too, not for intellectual wanking. In the past I played around a lot with NN, GA, different forms of ML, PSA & other stuff in addition to conventional optimization. I never got as far as I had hoped in conclusively identifying drivers in non-linear systems. If you have any comments or would like to discuss & share ideas offline, yes, please. "Over coffee in your office" might perhaps be a bit difficult as I choose to live on Penang Island, Malaysia, but please feel welcome to contact me if you want: [email protected]
Cheers, all the best, TonyM.

Interesting stuff Tony.

I'm definitely going to steal your "intellectual wanking" phrase.

The backwards r&d concept is something i like too. 100% agree with that. It's why I always start by coding up a batch of different ideas first, so that I'm not married to one single improbable idea early doors.

Wish I lived on Penang island. I bet the food is awesome.

Hello, I am very interested in the conversation.
In the end, is it possible to perform parameters' optimization as it is done with Easylanguage in Tradestation or Multicharts?
If it is not possible to do in Q specifically, do you guys know of to do that in Python other libraries?
Thank you very much.

My input: parameter optimisation is not different to choosing factors. Are you seriously going to argue that you can overfit factors? Yes, in both cases you can. I'm not going to go into details but creating algorithms is all about rigorousness of your testing and selection of both parameters and factors.

With that, I am keen to hear if there is a sensitivity analysis that can be done on parameters.

This is referred to as Hyperparameter Optimization in the ML and optimization fields (I work in both) and it's a critical part of developing models in ML and effective algorithms in optimization. I'd imagine it's also an important step in building a robust trading strategy. The issue of overfitting is present also in ML and optimization, there are standard ways of mitigating it, such as cross validation, regularization, and holding out data. I realize that this would amplify the compute time that people need. However, since this is likely to result in significantly improved models IMHO it should definitely be part of the Quantopian platform.