"Net results from my shorts tend to be negative" Yes, that was my first observation. Get the return on those shorts up and you may see a big improvement.

I could be way off here (not knowing anything about your algo) but here are some high level ideas.

First, look at a typical way to determine securities to long and short. Often one takes a single factor (or group of factors), then long everything that does well and short everything that doesn't. However, not all factors are 'symmetric' like this. Consider the case of stock buybacks. It may be the case that stocks with high share buybacks do well, but the converse, companies with no share buybacks don't necessarily do poorly. So, if this is the case in your algorithm, it may be helpful to use two factors, one for longs and one for shorts.

Second, it looks like the short positions are held longer than the longs. That may be what's driving those losses? Maybe try putting in some stop loss logic for the shorts or even a brute force 'max hold' time. I've used a 'blacklist' type of logic in the past. Again, not sure how your algo works, but I generally think of a factor as providing insight into not only the absolute value of future performance (ie price up or price down) but also some expectation of timeframe. A factor predicts a price change within x days. It looks like the median duration of a trade is 17 days or so. Maybe close positions after 2x17 days and 'blacklist' those so they cannot be held for another x number of days. The logic here is that those stocks aren't behaving the way one expects. There may be something 'odd' going on like a news event or product review that the quant data doesn't capture.

Just some thoughts.

Good luck!

Hey Joakim,

Great strategy. You're starting from the right point and thinking about exactly the right issues, these kinds of discussions are exactly what we want to encourage in the forums. Dan already contributed some good thoughts, here are some of mine.

1. "Sharpe dipping below 1 (and zero) a bit too frequently and for too long. Does this automatically disqualify me for allocation consideration?"
   At a quick glance, your Sharpe looks good to me. Real algos will have real performance, and real performance means that your day over day Sharpe will move around a lot. I don't see anything there that would be automatically disqualifying.

2. "It tends to have a 'Sell in May and go away' tendency where it doesn't do too well for a few months from around May each year. Not really sure how to neutralise or factorise this however. Maybe add a seasonal short factor with a long hedge?"
   It seems like the turnover is consistent through May, so I guess you have seasonality in the factor such that the model stops being predictive around then and you lose money on trading costs. With any kind of factor combination like this you want to understand each factor's behavior as best as possible and then make an intelligent decision about how to combine/hedge. Hedging adds estimation risk into your algorithm, so another approach might be to have a multifactor model in which the model which seasonal bias is weighted near zero in the months where it contributes noise.

3. "Net results from my shorts tend to be negative. Should I try to identify (using Alphalens?) the factors that contribute the most to the short losses and try to reduce the short weights for those factors? How does one go about doing this? Or am I running a risk of overfitting it too much then?"
   As with the last point, you should use alphalens to try to really understand the behavior of each factor independently. Understand the potential biases and try to use economic rationale to understand why they might exist. Each factor has noise, so it's totally possible to overfit by trying to overcorrect for biases that aren't there. Best practices to reduce overfitting here are to not tweak too much, come with a hypothesis about behavior before you look at the data, and check what the data is telling you against economic rationale behind the factor, and of course as always out of sample testing. If your factor is predictive on longs but anti-predictive on shorts, then you could even try splitting your model and using different factors for long and short weights.

Long story short, use alphalens to really understand what's going on with each factor independently, then make intelligent decisions about how to combine. Each factor will be good at predicting some things, and bad at predicting others. It's better to allow each factor to work only in the universe in which it does well. That can mean weighting factors differently depending on what time of the year it is, whether you're weighting longs or shorts, or what time horizon you're trying to predict on. I believe it's fairly common to have models which are better at predicting longs vs. shorts, so one approach is to construct a strategy on long only positions, and then do an equal weighted short basket on random stuff from the Q1500 for the rest of it. You could even rotate your shorts far less frequently than your longs to reduce trading costs.

I hope the performance holds up out of sample, the approach you've taken to being risk constrained and thinking about factor performance could really pay off. If any of this was unclear or there are more things I can help with let me know.

hi Joakim,

Looks great. A couple quick points and I'd also be happy to connect offline to discuss.

I know it is painful to think about 'waiting for out of sample' to accumulate, a good way to try to get around that for your own work is to hold data back from your model development process and then cross check or cross-validate when you feel our modeling is pretty much complete so you can try to avoid overfitting (yes, I know we could make this easier for users - its on the to do list!). In the meantime, if you'd expect your algo to perform comparably over a prior 36-48 month period but you didn't use data from that period to develop the algo you can do a check now to see if you would see a similar (or at least positive) Sharpe over say 2010-2014.

Second Q is just to confirm you're using the fixed slippage model at 5bps of cost - I've found that to be the most valid, albeit a bit conservative, cost tool we've made available to date and we use it internally (as well as in the contest).

Please feel free to ping me at [email protected] and I'd be happy to chat more about your strategy's economic rationale and our diligence process. Best, jess

@Dan,

Thanks so much for your feedback! I really appreciate it! It's all spot on too. None of my factors are very symmetrical, and in general I do tend to think of 'Long' ideas, and then short the opposite. I might try to think of what stocks I'd like to short instead (over-valued, highly leveraged, low return on capital, etc) and long the opposite instead. As Charlie Munger has been known to say: "Invert! Always invert!"

Having different weights for longs and shorts for each factor has also been on my list of things to try, and with the (round_trips = True) switch on in the full tear sheet I think it's evident that it might indeed be something worth exploring (minor tweaks mostly though as I worry about overfitting if I do too much).

Stop-loss orders I've tried, but I don't think Optimise API supports them (I can understand it would be difficult to implement). Blacklisting though is a great idea though which I hadn't of and which I'd like to try. Just need to figure out how to best do it first. :) Doesn't sound like it should be too difficult but I'm a bit of a Python novice still.

@Jess & @Delaney,

Thank you so much for taking the time to provide feedback. I really appreciate it! I've only been on Quantopian actively for about 3 months now and it's been a steep learning curve (which I enjoy) so it's really encouraging to hear that I'm on the right track.

@Delaney,

Great suggestion on #2. As you said, I'll need to spend some more time with AL to analyse each factor individually to try to see if I can find any seasonal trends (and predictability of longs vs shorts), and try to adjust the weights accordingly without overfitting. Thanks also for your comment on #1. I'm glad you don't see it as a big issue. My aim is still to try to have sharpe remain above 1 consistently, even if it means the peaks will need to come down a bit. The challenge as always will be to not overfit (even if unconsciously and unintentionally). I'm hoping this will be possible if I can reduce the impact of the short losses by adjusting weights.

Your comments and suggestions on #3 is spot on too. I think I have enough factors to work with, and I will indeed spend more time with AL to understand the strengths and weaknesses of each one of them and hopefully be able to make rational adjustments to reduce losing trades (and potentially trading costs). Would you recommend picking a sample period where the algo doesn't do too well in terms of sharpe or drawdown? For example April 2013 to April 2014 when sharpe is tanking? Or end of 2013 to end of 2013 based on the Underwater plot?

Also, would you be able to expand on your comment below? I understand the concept but not how to implement it when using Optimise API and Maximise_Alpha? How would I go about doing this? Could you point me in the right direction please?

You could even rotate your shorts far less frequently than your longs
to reduce trading costs.

A separate question: while I've specified to rebalance daily XX min after the open, does Optimize API automatically VWAP large orders over the day in order to get best fill prices (I imagine your PB would do this)? If not, would you recommend me implementing this to be able to handle more capital and get better fill prices? Looking at the tear sheets it seems to me that it's already doing this, or am I wrong? If that's the case, would it not be better to start directly from the open (MOO essentially, volume weighted) as volume tends to be highest at the open (and around close)? Or is the potential slippage cost from the higher volatility right after open not worth it?

@Jess,

Neither Commission nor Slippage was specified in this backtest so it should be using the default ones, which I believe is indeed 5 bps cost at 10% volume per minute bar. Just to be sure though, I've now specified this under Initialize and I can't really see any difference in returns.

def initialize(context):  
    set_slippage(slippage.FixedBasisPointsSlippage(basis_points=5, volume_limit=0.1))  

Awesome tip on the 'historical Out-of-Sample' test as well! My research was mainly between 2014-2016 and then 2016-2018, so the attached (2011-2014) is somewhat out-of-sample, and it shows... While not completely horrible, the returns are not nearly as good, and unfortunately sharpe does indeed go quite a bit into negative territory for a while. This I'll need to address.

I would have liked to run backtests from further back (especially during GFC) but unfortunately I get the below error when I try anything earlier. I think it's because one of the Fundamentals from Morningstar that I'm using may not be available then. Does that sound correct?

TypeError: MaximizeAlpha() expected a value with dtype 'float64' or
'int64' for argument 'alphas', but got 'object' instead.

Though it may be of less interest now until I've addressed the negative sharpe, I'll still send you and Delaney a direct mail explaining my high level rational behind the factors.

Attached tear sheet has slippage specified to 5 bps as above. I'll specifically specify this going forward. I'll attach another one for the full period as well, with slippage specified to 5 bps. Also, I wanted to show you full tear sheets with 100mm and over of initial capital, but unfortunately I run out of memory whenever I try this, even after killing all active notebooks and restarting kernel. Would this be related to amount of capital deployed or just a coincidence you think? As far as I can see, there's not much impact on returns with 10x more capital.

Again, thanks so much for the feedback! I find it all extremely helpful. I'll have plenty to work on for a while now but if you or anyone else have any more feedback, I'm all ears. Cheers!

PS: Something funny is going on with the Common Returns from end of 2013. Is this a bug perhaps?

Same backtest as the first one, but with slippage specifically set to 5bps at 10% volume.

Full period backtest for as far back I'm able to go. Slippage set to [email protected]% volume/minute.

In your backtest 5af8f212a8c1f844ac870815, the Average Risk Factor Exposure is 0.00 across the board. This seems kinda anomalous to me and possibly over-constrained. What constraints did you use for the optimizer (assuming that you used the optimizer to constrain the risks)? Are you using RiskModelExposure(risk_model_loadings, version=opt.Newest)?

You might consider the turnover. I've tried the optimize API turnover constraint, by iterating over the max turnover:

    turnover = np.linspace(0.05,0.65,num=100)  
    for max_turnover in turnover:  
        constraints.append(opt.MaxTurnover(max_turnover))  
        if context.init:  
            constraints = constraints[:-1]  
            context.init = False  
        try:  
            order_optimal_portfolio(  
                objective=objective,  
                constraints=constraints,  
                )  
            record(max_turnover = max_turnover)  
            return  
        except:  
            constraints = constraints[:-1]  

The above code avoids the crash that sometimes occurs when using a fixed turnover constraint.

Grant, that's some awesome code. Now I wish there was some better place to post these snippets. I'll see if we can get it integrated into some of our templates if that's okay with you.

Joakim,
One point about #1, Sharpe ratio isn't really a metric you should be trying to 'optimize' in an engineering sense. The way you want to use it is as a comparison. Test multiple models, measure their information coefficient and other metrics, and then intelligently choose the best one. Then when testing different algorithms, see which one has the highest/most consistent Sharpe ratio. It will lead to a lot of overfitting if you think about trying to change/optimize/reach a certain Sharpe threshold. It's not a metric which can be changed, it's a response to the quality of your hypothesis/model, which is what you should be changing.

In general it's a good idea to focus on where your model doesn't work, as that's where you'll learn the most about why it and your hypothesis was wrong in some way. You should also be careful about getting obsessed with time periods over which it doesn't work, sometimes it's just random as to why it stops working and you can overfit to that period. I would just look at it overall, and then maybe each year individually and then focus in on certain time periods where it doesn't. The main question is whether there is a reason you can identify that it doesn't work. If you know the reason you can either change the model or choose to not use the model during those conditions.

"Also, would you be able to expand on your comment below? I understand the concept but not how to implement it when using Optimise API and Maximise_Alpha? How would I go about doing this? Could you point me in the right direction please?"

I just meant using your faster moving long factor to assign weights to your longs, and then use a pretty slow moving factor for your shorts. The optimizer will definitely muck with your shorts a bit and affect turnover, but that's life. The other thing to worry about is whether your factors are all scaled in the same space.

order_optimal_portfolio just uses market orders for the sake of simulation. How it works when you receive an allocation is slightly different. In that case the order_optimal_portfolio computes the target portfolio and deltas, then sends the deltas to our trading team. The deltas are then routed through our prime broker's execution algorithms. I recommend just using order_optimal_portfolio at the start of end of day depending on your model. Vary the cost and slippage assumptions in your backtest to determine how robust your algorithm is to cost and slippage, if you determine it's a big issue, but the model is predictive, then you can ask us specifically if we think the model is still viable in a different execution environment. However as Jess said the best approach is probably just to use the 5bps model and go from there.

@ Delaney - sure, you are welcome to build on the concept (your Python whizzes can code it better, I'm certain). Another thing to consider is adjusting the turnover based on a factor. For example, when the market is more volatile, then perhaps use higher max turnover, but reduce the max turnover when the market is calmer (I have not tested this...just an example).

Hi Grant,

Yeah, looks a bit suspicious doesn't it? I do indeed use that exact code for the sector and common risk factor exposures, but of course I use much tighter parameters than what the default ones are in opt.Newest.

I do also restrict turnover, but I only use the below. I picked 20% because it seemed to allow for most of the natural turnover (roughly 2 std from the mean - estimated, not calculated), and hopefully limits that top 5% that may be overtrading. I also have 10 fingers and 10 toes that helped me determine that this was a good number :) . It did help improve returns a bit (from less trading costs I believe), which perhaps means I've overfitted again? I haven't experienced a crash yet due to this constraint, but I might still try your code to see what it does. Hopefully it will at least help me learn numpy more, and what that method does (currently I have no clue).

MAX_TURNOVER = 0.2

    constraints.append(opt.MaxTurnover(MAX_TURNOVER))  

Hi Delaney,

Awesome feedback as always, and point taken on the sharpe ratio and period picking. I've been pretty paranoid about overfitting, but I now believe I've been overfitting quite a bit since I was always thinking of how I can minimize volatility and maximise returns. I need a bit of a mindset change I think and tackle it from a different angle, the way you've been suggesting. Easier said than done, but that's my goal now at least. Thanks for pointing it out.

Having a code snippet library (Grant's original idea?), organized into relevant sections, I also think is a great idea. I hope it happens.

I'll keep using 5bps slippage too as my default to manage my own expectations. I'd rather be positively surprised than potentially disappointed from poor fill prices.

Off to research-land now to try to gain a better understanding of each factor. Thanks again for all your help!

Hello Joakim -

I'm curious what algo performance you get with RiskModelExposure(risk_model_loadings, version=opt.Newest) without changing the defaults. If you run 5af8f212a8c1f844ac870815 but comment out any default risk model over-rides, what does the tear sheet look like?

@ Delaney/Jess - Based on the contest rules, and the fact that they reflect what is required for a decent Q fund algo, my understanding is that using the default risk model constraints should be sufficient, correct? In other words, if the default constraints are met, then one is at the point of diminishing returns in trying to further isolate specific returns? Or is the idea that algo authors should strive to get common returns to zero?

@ Delaney - Regarding "Now I wish there was some better place to post these snippets" I'd recommend thinking this through a bit, and doing some systems engineering to get the solution right the first time. If you think the turnover code above has legs, then you could stick it here for starters. I and other users could help with the systems engineering that needs to be applied to the more global problem of code being all over the place on Quantopian, and difficult to find (and knowing if it is current and will work with the present Q API).

@Grant,

With the default constraints, the tear sheet reveals more than I'm willing to share unfortunately, but in summary (I would attach screenshots if I knew how to), during the same period:

Total Returns
38.27%

Benchmark Returns
57.06%

Alpha
0.07

Beta
0.07

Sharpe
1.80

Sortino
2.76

Volatility
0.05

Max Drawdown
-3.56%

Not nearly as good in other words. Roughly half of these Returns are from 'Common Returns' and the other half 'Specific Returns.' Beta to SPY moves around much more, mostly on the positive side, and just barely stays under 0.3. Some cut and paste stats from the tear sheet (can you paste in tables so the formatting is nicer?) below. Shorts still perform poorly, though slightly better perhaps due to a few big winning short trades (I'd rather have lot's of small ones).

Summary stats All trades Short trades Long trades
Total number of round_trips 86729.00 46703.00 40026.00
Percent profitable 0.51 0.39 0.64
Winning round_trips 43882.00 18409.00 25473.00
Losing round_trips 42847.00 28294.00 14553.00
Even round_trips 0.00 0.00 0.00
PnL stats All trades Short trades Long trades
Total profit $3467543.30 $181500.75 $3286042.55
Gross profit $24847603.90 $13287403.23 $11560200.67
Gross loss $-21380060.60 $-13105902.48 $-8274158.12
Profit factor $1.16 $1.01 $1.40
Avg. trade net profit $39.98 $3.89 $82.10
Avg. winning trade $566.24 $721.79 $453.82
Avg. losing trade $-498.99 $-463.20 $-568.55
Ratio Avg. Win:Avg. Loss $1.13 $1.56 $0.80
Largest winning trade $57867.80 $33865.51 $57867.80
Largest losing trade $-48027.16 $-48027.16 $-25062.32
Duration stats All trades Short trades Long trades
Avg duration 53 days 06:33:36.622594 54 days 20:44:21.915594 51 days 10:00:43.177359
Median duration 23 days 23:57:00 27 days 00:02:00 20 days 00:00:00
Longest duration 1064 days 23:59:00 1064 days 23:59:00 768 days 00:00:00
Shortest duration 0 days 10:25:01 0 days 10:25:01 0 days 10:27:01
Return stats All trades Short trades Long trades
Avg returns all round_trips 0.00% 0.00% 0.00%
Avg returns winning 0.00% 0.01% 0.00%
Avg returns losing -0.00% -0.00% -0.00%
Median returns all round_trips 0.00% -0.00% 0.00%
Median returns winning 0.00% 0.00% 0.00%
Median returns losing -0.00% -0.00% -0.00%
Largest winning trade 0.50% 0.26% 0.50%
Largest losing trade -0.48% -0.48% -0.23%

Performance Relative to Common Risk Factors
Summary Statistics
Annualized Specific Return 3.89%
Annualized Common Return 4.39%
Annualized Total Return 8.43%
Specific Sharpe Ratio 1.34
Exposures Summary Average Risk Factor Exposure Annualized Return Cumulative Return
basic_materials -0.00 -0.28% -1.10%
consumer_cyclical -0.02 -0.40% -1.59%
financial_services 0.08 1.29% 5.26%
real_estate -0.04 -0.24% -0.97%
consumer_defensive 0.00 0.03% 0.13%
health_care -0.02 -0.16% -0.63%
utilities -0.01 -0.06% -0.23%
communication_services -0.01 -0.04% -0.17%
energy -0.04 0.34% 1.38%
industrials 0.04 0.58% 2.33%
technology 0.02 0.58% 2.34%

@Grant,

Regarding your question to Delaney/Jess, I'm not sure you'll get a response but have a look at the Capital - Allocations page and I think you'll find the answer there. Below are the relevant sections with my highlight in bold. The way I read it is that the default constraints are just the very bare minimum and far from ideal (note that it seems to start becoming more interesting for an allocation at about half the default constraint values). They probably have the defaults this wide not to prevent or discourage too many people from entering the contest.

I think the more capital you have the less tolerance for Risk you have (in general). Not all my algos are constrained this much, but I wanted to see if I could find some 'Pure Alpha' (not really, but a step closer at least) by minimising Risk exposure as much as possible (what I call Risk Parity - both terms shamelessly ripped off from Mr. Bridgewater). 'Optimal' constraints for this algo is probably somewhere in between these extremes.

Low Exposure to Sector Risk We want algorithms that limit their
exposure to any single stock market sector. While there are valid
strategies that may be confined to one of a few sectors, or that
specifically seek to predict and profit off of differences in
performance across sectors, in all other cases we are looking for
algorithms to limit their sector exposure. As a rule of thumb, we are
looking for algorithms that seek to limit their maximum exposure to
any single sector to less than 10%, and *ideally target an average
exposure of as close to 0% as possible*.

Low Exposure to Style Risk We

want algorithms that limit their exposure to well-known investment
styles such as investing in small market capitalization companies over
large companies. There are five investing styles we track: size,
value, momentum, short-term reversal, and volatility. While there are
valid strategies that exploit consistent exposures to these investment
styles, we are looking for algorithms that limit their exposure as
much as possible and seek to exploit novel relationships in the market
to achieve consistent profits. As a rule of thumb we prefer algorithms
that limit their maximum exposure to any single style to less than
20%, and *ideally target an average exposure of as close to 0% as
possible.*

Thanks Joakim -

The page https://www.quantopian.com/allocation was updated at some point; I didn't realize that it would be advantageous to have risk exposures of ~ 0%. This had not been clear to me; I was under the impression that a little "breathing room" would be allowed, and perhaps encouraged, based on the contest rules and ranking criterion of a simple return to volatility ratio. In other words, no penalty for an allocation would result, so long as the algo stays within the default risk limits (including beta), and has a high return to volatility ratio (not exactly Sharpe ratio, and in fact, Quantopian shouldn't be using the term "Sharpe ratio" since most folks assume that the risk-free rate is subtracted, when in fact, it is not).

It would be interesting to hear from Jess/Delaney on this topic, since I have contest algos that has been doing pretty well (consistently in top 10), but now I'm wondering if I need to revisit them to attempt to beat down the risk exposure to ~0% (and probably other stuff, like understanding the "economic rationale"). Since I submitted my entries to the contest, I have done almost nothing on Quantopian, thinking that I'll wait 6 months, and then if I have a case to plead, I'll get in touch with them for feedback. As Jess offered to you, I'd be glad to get feedback sooner. Maybe I should be working to improve my algos, but it would be good to have some sense if it'll be worthwhile, beyond making money on the contest.

@ Joakim - I've followed your lead, and posted a tear sheet here:

https://www.quantopian.com/posts/contest-algo-feedback

Your feedback would be welcome.

Cool! Will have a look over the weekend and give you my feedback.

Based on the contest rules, and the fact that they reflect what is required for a decent Q fund algo, my understanding is that using the default risk model constraints should be sufficient, correct? In other words, if the default constraints are met, then one is at the point of diminishing returns in trying to further isolate specific returns? Or is the idea that algo authors should strive to get common returns to zero?

I would advise not spending time trying to over-optimize, and potentially overfit, the risk model constraints. We provide the default contest constraints as the box we would like every model to fit in. If you take a model and constrain it to those thresholds without destroying the signal, that's a good indication that there's good alpha there. There may be a good reason based on your model to try to constrain the risk even more, but generally think of it as a binary check. Returns positive? Check. Information Coefficient positive? Check. Retains signal when risk constrained in default box? Check. Performs out of sample? Check. That's mostly what you should be concerned with, after that you should just be intelligently combining more and more models to achieve the best signal/noise ratio possible.

Regarding "Now I wish there was some better place to post these snippets" I'd recommend thinking this through a bit, and doing some systems engineering to get the solution right the first time. If you think the turnover code above has legs, then you could stick it here for starters. I and other users could help with the systems engineering that needs to be applied to the more global problem of code being all over the place on Quantopian, and difficult to find (and knowing if it is current and will work with the present Q API).

We did try this a while back, but we didn't publicize the repo and didn't put any effort in beyond the original batch, so it didn't work out. I actually would like to try an experiment. Can you PR a new folder into research_public which contains this code snippet as its own file, maybe with a very short docstring description? In general I'm happy to try letting users use that repo as a way to swap code snippets. It works for bacteria so why wouldn't it work for quant finance :) ? If the experiment goes well and people find that helpful we can re-evaluate then.

@ Delaney - I submitted the PR to research_public, as you requested. If y'all want to improve it before posting, go for it.

@ Delaney - There are a number of topics above that are analogous to hyperparameter optimization from the ML domain--e.g. turnover, algo run & order submission scheduling, number of stocks in portfolio, optimize API constraint values, etc. They are all parameters that interact with the alpha factor, but are not the core of the strategy. Within the context of the Q platform, and the development of type of algos you need for the fund, there could be a better defined, systematic approach, that would limit the risk of over-fitting, but at the same time allow for a full exploration and optimization of the hyperparameter space, in an automatic fashion. When I get the chance, I'll add my input to https://www.quantopian.com/posts/what-type-of-content-would-you-like-to-see.

Hey Karl,

Thank you - I appreciate and value your feedback! Good spotting on BPOP-1062. I had a look at the stock and it seems to meet at least some of the criteria in the strategy. I don't normally look at the individual stocks it trades, especially for this algo since concentration risk is pretty low. Do you think I should investigate further, or be concerned if the algo favour a certain stock?

Yes, I've tried several different concentration levels, including 200, which as expected, gives me higher returns but higher volatility as well. For this one, I wanted to have as low volatility as possible, without impacting returns too much. 400 seems to give me a good balance. Optimal concentration is probably a lower number of max holdings though as you suggested.

Hi All,

Just wanted to revisit this 'older' post to see how my original PARTY algo has performed in the last 4 months' with true OOS 'live' data. Any and all feedback welcomed!

To me, it appears that it's actually performing 'better' since May 16, 2018 than during the 'in-sample' period. The 'red-live' trading line appears to be moving along the upper bound of the bayesian cone in the 'Cumulative Returns' graph... Is this a 'bad' thing, and does this mean that the strategy is more likely to perform worse going forward, to average out the recent good performance? Or am I falling victim of the old 'gamblers fallacy' with this type of thinking? I honestly don't know as I'm no expert on probability or statistics (or gambling :)).

Also, I'm curious if the 'red peppering' of OOS returns data outside of the in-sample boxes is a bad thing? Or is it still somewhat normally distributed?

Hi Joakim,

I think you're getting great OOS, probably a bit on the "lucky" side but still within expectations. Your last graph seems to confirm something I noticed in the tearsheet... which is that your algo seems to be getting more and more volatile since around 2017. That could explain the red dots that are out of the rectangles.

Edit:

Forgot about the rollling volatility graph ;). Took a look and it's above average in 2018, but not "more and more volatile". Sorry about that. That could still explain the red dots.

Thanks Charles! I appreciate and agree with your feedback.

I do find it interesting that the strategy's returns tend to be more volatile during low volatility markets. I'm not really sure the reason for this or if it's a good or a bad thing (I'm leaning towards 'bad' though).

I ran this backtest ID through the Risk On NB (see attached below) to see if the strategy prefers a low or a high volatility market. It appears to be preferring a high volatility market. Again, not really sure if this is a good or a bad thing but leaning towards bad (perhaps indifferent in a portfolio of uncorrelated strategies).

Perhaps this could be one reason why the strategy has been more volatile since 2017, when we've had a relatively low volatile market?

Looking again at the rolling volatility plot, and I don't think I see a negative correlation between your algo's returns volatility and the benchmark. I wouldn't be worried. It was a good idea to run the Risk-On notebook! It basically shows that your algo performs very well in both regimes. Really impressive, and all that with pure specific returns...

The only thing I would maybe be worried about is over-optimization. I've had some curious results on my side when using the Max Turnover constraint, and squeezing the risk-model constraints. It might not be the case at all with your algo, but I'd love to see how it performs if you relax the risk model exposure constraint a bit, and lets say, average your last few alpha vectors before sending them to the optimizer, instead of using the max turnover constraint.

Cheers!

Thanks Charles, I’ll see what I can do. From what I can remember, relaxing the Risk constraints increases both Returns and Volatility a bit. Optimal Sharpe might be with a bit less Risk constrained.

Removing MaxTurnover doesn’t impact too much from what I remember.

What do you mean by this:

average your last few alpha vectors before sending them to the
optimizer

Would you be able to provide a code snippet example?