1. Economic Hypothesis:

This strategy will attempt to find alpha in the well known GARP (Growth At Reasonable Prices) investing strategy/style, successfully implemented and made famous by Peter Lynch at Fidelity's Magellan Fund in the 1980's.

The Long hypothesis is to buy good quality, compounding growth companies when they are selling at attractive prices (e.g. due to a short-term correction, earnings disappointment, scandal, etc.).

The Short hypothesis is currently TBD, and may be completely different from the GARP strategy altogether. Or it may just be the bottom quantile of the GARP strategy, if it turns out to be somewhat symmetrical and predictive on the short side as well (pending Alpha Factor research in #4).

Further information on GARP can be found on Zacks, Investopedia, Investorguide, and SeekingAlpha, and many more places I'm sure.

Why am I expecting to find any real alpha in something so well known and studied as GARP you ask? Perhaps we can find an answer to that from the ever quotable Yogi Berra, hypothesising that "[GARP is] so crowded, nobody goes there anymore." :)

(2.) Data Selection to follow...

2. Data Selection:

This one's pretty easy. Since I'm currently not willing to pay for any datasets, I'm primarily going to use the free Morningstar growth and value related Fundamental data. I may also use the free US Equity Pricing from Quantopian (OHLCV) to try to identify longer term growth trends, and shorter term value opportunities (hopefully anyway).

It would be nice to also use some of the EventVestors, Estimize, or Zacks datasets (e.g. earnings guidance, estimates, and surprises), but since I can't use them in the contest without paying for them, I'll leave them out for now.

This is one major reason why I'm so excited about Quantopian's recently announced partnership with FactSet. Yes, international equity markets is exciting as well, but I'm even more excited about using FactSet's vast datasets, for free essentially but with limitation on the time-series available (e.g. up until 12 months ago perhaps). This 'limitation' can actually be seen as an advantage, as it forces us to 'hold back' OOS data when researching Alpha. Data which WILL be available to Quantopian internally (according to Jess in the recent tearsheet review webinar), and which Q can then use to analyse how 'overfit' a FactSet based strategy is. Very cool!!

(3.) Universe Definition to follow...

3. Universe Definition:

Another 'easy' one. On surface anyway. Going back to my 'Primary Objective' to minimize the likelihood of overfitting, I want my base_universe of stocks to trade to be as wide and diverse as possible. At this point, I don't want to 'fit' my strategy to only trade certain sectors, only large/small caps, only high volume stocks, or only low volatility stocks, etc.

However, I also don't want to train my model on listed securities where the model is not intended to work, e.g. ETFs, ADRs, OTCs, etc. (I suppose I'm 'fitting' here a little bit, but that's unavoidable and the reward/risk ratio is massively net-positive in my view). I also want to exclude very illiquid stocks, where I may not get realistic fills and prices without 'moving the market'. Quantopian's fantastic built-in filter QTradableStocksUS() takes care of all of this! ThanQ! :)

Now, I said 'on surface' because while I now have a solid 'base_universe' defined, I may look at further narrowing down the Universe in #4 (which follows), and run factors on a more select universe of stocks (intersected with my base_universe) where I believe each factor is (more) predictable. Here I would be 'fitting the model' of course so I will need to be mindful and careful not to let too much 'overfitting' creep in. Any tips from anyone on how to avoid this, I'd be all ears!

I won't be going in to any more details on any further universe narrowing as it's starting to get in to 'secret sauce' territory, but it's not exactly rocket science either... :)

(4.) Alpha Factor Research & Discovery to follow...

4. Alpha Factor Research & Discovery:

Here comes the hard part!

I anticipate to spend most of my time on this step, with multiple posts, over the next few weeks. A lot of alpha research, and a lot of failures, but hopefully a bit of alpha factor discovery as well in the end, otherwise I can't really continue to the next step. Understandably, I won't be posting any discovered real alpha, but I'll probably post a few alpha research failures.

I do plan to share and attach an Alphalens notebook 'template' that I'll be using to research and hopefully discover alpha, but before that, I'd like to do some initial research to see if I can plot and 'visualize' any potential correlation between top and bottom factor quantiles, and the 'walk forward' cumulative future returns for the quantiles during the in-sample training period.

Take the PEG Ratio for example, perhaps the most commonly used value-to-growth ratio for a GARP type strategy. Before running it through Alphalens, I'd like to see if I can plot future cumulative returns (e.g. in a bar chart, where each bar represents the cumulative future returns for stocks in each PEG Ratio quantile/decile).

Before looking at any of the data though, based on my 'Economic Hypothesis' I would expect a lower PEG Ratio to be correlated with a higher cumulative future return. I am NOT going to look at the data and then 'fit' my economic hypothesis to reflect what the data is telling me (that to me is 'data mining' and would be very prone to overfitting).

However, I would also expect that at some point, say around a PEG ratio of 0.8 and lower, to gradually perform worse than stocks with a PEG ratio of say around 1. For this one, I would like to first look at what the data is telling me before determining where to cut off my universe of stocks to trade based on too low (or too high) PEG ratio. Maybe I'm datamining and overfitting by doing this?

In short, this is what I'm planning to do in a Research notebook:

1. In a pandas dataframe, import the PEG Ratio (from Morningstar) as of the starting date of my in-sample training period (06-15-2015), for all stocks in my base_universe (QTradableStocksUS()).
   
2. In the same dataframe, get the 'walk forward' future cumulative Returns, during the in-sample training period (06-16-2015 to 07-15-2017) for each of the stocks with a PEG ratio (survivability bias prone as I'd be excluding stocks without a PEG ratio?)
   
3. Divide up all stocks into deciles based on the PEG ratio.
   
4. Plot the 'walk forward' future cumulative Returns for each decile in a bar graph.

Please bear with me as I attempt this, because even for something as simple as this, I don't actually know if I can do all of this (or how long it will take me) with my limited pandas and plotting knowledge. If I do get stuck, I may ask the Community for help, or I may also just skip it altogether and just plug in some inverse version of the PEG Ratio (e.g. 1 - PEG ratio) in to Alphalense and go from there. I believe it should be fairly simple though, and that I should be able to figure it out and learn something in the process.

To be continued, hopefully in the not too distant future...

[Placeholder for my initial Research Notebook.]

I'm putting this on hold for a while as it's taking me a bit too long to figuring out, and I'm eager to jump in and use Alphalens to test factors that is in line with my 'strategic intent.'

[4. Alpha Factor Research & Discovery (Cont'd)]

Alphalens Template:

Well, my intention was to attach a basic Alphalens (AL) Template that I would be using to hopefully find a few decent GARP alpha factors from the free Fundamentals and OHLCV datasets. However, I keep getting the dreaded 'Sorry, something went wrong on our end.' error message no matter what I do (closing and killing all notebooks, reloading this post, etc), so this will have to wait for a bit as well (maybe 'user error' or maybe I'm just trying during 'maintenance windows'?).

The AL template uses the PEG Ratio, and as @Blue Seahawk pointed out, there seem to be something strange with this ratio, as it's not really updating very frequently for about half of the symbols in the QTU. For a ratio with 'price' in it, I would have expected it to update daily for most symbols.

It may therefore be worthwhile to create some sort of custom Value-to-Growth ratio (or the inverse really, so one doesn't have to 'flip' the factor), using data that can be verified to be more reliable.

For any alpha factor I might discover, I won't be sharing the notebook, but I'll share some key statistics and how I've interpreted them. For all factors, I will be focusing on a 1-day holding period, using a p-value(IC) cutoff of <0.05. I've also realized that I will need to 'eat into' a bit of the timeseries that I had set aside as OOS, for AL to use as OOS. I will update my first post above to reflect this.

Factor1 discovery analysis with key stats to follow...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor1 analysis:

Factor1 uses the free Morningstar Fundamentals dataset exclusively, and is based on a GARP-type 'economic rationale.' The below stats figures are all for 1D holding period (5 quantiles), using the above 2 year in-sample 'training period' with 2 months of AL OOS test period:

• p-value(IC): 0.005 (below my cutoff)
  
• IC Mean: 0.012
  
• Risk adjusted IC: 0.123
  
• Ann. alpha: 0.216
  
• beta: -0.107
  
• Mean Period Wise Return Top Quantile (bps): 1.416
  
• Mean Period Wise Return Bottom Quantile (bps): -3.405
  
• Mean Period Wise Spread (bps): 4.722

I didn't include the IC skew and IC kurtosis because honestly I don't understand them well enough yet.

That all looks quite good to me (let me know if you disagree), and the graphs all look pretty good too. It almost all looks a bit too good to be honest, so I might look to 'cross validate' this factor on an earlier time-series, to ensure I haven't overfitted too much to this specific in-sample training period.

For now though, I'm hoping that the somewhat 'imperfect' graphs might improve if I combine this factor with a different, hopefully somewhat uncorrelated (and hopefully equally strong) factor, using a different dataset (OHLCV). Note also that, for now at least, I'm not excluding any 'bad' sectors where the factor don't seem to be working, as I'm worried that I may be overfitting to this specific time-series too much if I do.

Factor2 discovery analysis with key stats to follow...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor2 analysis:

Factor2 is using data exclusively from the free OHLCV dataset from Quantopian. My reason for using a completely different dataset for this factor is so that hopefully it won't be too correlated with the first factor.

How am I supposed to get a GARP type factor from using only historical OHLCV data you ask? Basically, without going into too much detail (not exactly rocket science either), I'm hoping that I can find both growth as well as value 'priced in' to stocks' historical price series.

Below stats are based on the same time series, holding period, # of quantiles, etc. as Factor1:

• p-value(IC): 0.000 (below my cutoff)
  
• IC Mean: 0.019
  
• Risk adjusted IC: 0.156
  
• Ann. alpha: 0.077
  
• beta: 0.017
  
• Mean Period Wise Return Top Quantile (bps): 4.782
  
• Mean Period Wise Return Bottom Quantile (bps): -2.664
  
• Mean Period Wise Spread (bps): 7.406

Ok so that looks quite good as well (to me anyway - let me know if I'm wrong), and the graphs look pretty good too. A bit different from the first factor, which I actually think (hope) might be a good thing.

Both IC Mean and Risk Adjusted IC are higher than for factor1, but the Annual Alpha is quite a bit lower. I'm not worrying too much about the difference in Ann. Alpha for now, 'hoping' that this might just be due to random-walk noise. Also, the 'Mean Period Wise Spread (bps)' is quite a bit higher than for the first factor, so I'm not really sure why the annual alpha is so much lower for the second factor (maybe the Ann. Alpha for the first factor was inflated somehow, e.g. due to outliers, and unrealistic going forward?).

Since the first factor had a negative beta, I'm glad this one has positive beta, which hopefully overall will result in a more neutral beta value.

Based on the 'Mean Period Wise Return' from the Top and Bottom Quantiles, it looks like this second factor is stronger for 'longs' than for 'shorts' whereas the first factor was 'stronger' in the bottom (short) quantile than for the top (long) quantile, which I'm interpreting as a good thing.

For the third (and possibly final) factor, I will look to use a combination of different data from both the Fundamentals as well as OHLCV, to hopefully find a factor that's somewhat uncorrelated to the first two factors. Fingers crossed!

Factor3 discovery analysis with key stats to follow...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor3 analysis:

As mentioned, this factor uses a combination of both Fundamental and historical price data (OHLCV). In order to avoid too much correlation overlap with the first two factors, I've tried to use different type of data from the two datasets. I wish I knew how to measure correlation between factors (or alpha decay for each factor), but currently that's a bit too over my head unfortunately. Hopefully I'll get there in the not too distant future...

• p-value(IC): 0.002 (below my cutoff)
  
• IC Mean: 0.015
  
• Risk adjusted IC: 0.134
  
• Ann. alpha: 0.087
  
• beta: 0.082
  
• Mean Period Wise Return Top Quantile (bps): 2.786
  
• Mean Period Wise Return Bottom Quantile (bps): -3.999
  
• Mean Period Wise Spread (bps): 6.781

I'm hoping the relatively high (positive) beta will help balance out the relatively high (negative) beta of my first factor, to make the combined factor close to 0 beta.

As I don't have too much experience yet using Alphalens, I don't have too much to compare with for any of my factors, so I don't know if these values are (too) good or bad. For now though, I'm going to assume that they are a bit 'too good' so I want to find some assurance that I haven't been (unintentionally) data-mining this in-sample period too much, and effectively fit these factors to random-walk noise in the in-sample training time series.

In order to do this, I'll need to run these factors over some earlier OOS time period. It might be interesting to see how the factors held up during 2008 for example (GFC). By 'cross validating' each factor against a different time period, I'm hoping to also be able to detect any 'false positive' factors. However, if any of the factors get a p-value(IC) above my 0.05 cutoff from the new OOS test period, I may choose to test it a third time, to ensure the second test didn't return a 'false negative.'

In other words, I may need to test these factors up to two times OOS, to ensure I have either 'true positive' factors (which I'll keep), or 'true negative' factors (which I'll discard).

I may try to find a few more factors first though, incase some of my factors turned out to either be 'false positive' / 'true negative', or didn't perform very well during the OOS test period.

To follow: TBD

Good going, Joakim. Do you prefer to combine the factors inside Pipeline, or outside, and which is better?

Thanks Karl!

I haven't gotten to combining the factors yet, but I was planning to do it inside of Pipeline (normalized equal weighting), just because I would have thought it's faster than outside(?). I don't quite see the benefit of doing it outside. You know more about this stuff than I do though.

[4. Alpha Factor Research & Discovery (Cont'd)]

Trying again to attach my Alphalens (AL) template. This one first creates a 'summary' output of all the key stats of the factor during the training period, and then the full tearsheet, grouped by sectors. This is pretty much the one I've been using, and will most likely be using going forward.

I've also realized that I'll probably need to use up more of my historical data for 'cross validation' of my factors during different market periods (non-stationary timeseries). Once I've tested a factor during a past OOS time-series however, I will no longer consider that time-series to be OOS and it will instead become part of my (growing) training time-series. I will eventually update my original post to reflect this. One may cry out 'look-ahead-bias' or 'benefit of hindsight' but I'm seeing this as part of my own learning experience in going through the process of designing a more robust L/S strategy that's less prone to overfitting.

I'm also realizing that this exercise is mostly trying to address Fawce's 'bad data hygiene' example of overfitting in his recent excellent post on Portfolio Structure and Overfitting. However, I believe most of my strategies are also quite prone to the 'Excessive precision without accuracy' example of overfitting as well. Eventually I would like to address this by making use of Kalman Filters (if that's even possible), but for now, that's quite a bit too complicated for me. So for now, I will just rely on having a good 'rationale' for the parameters I'm choosing. For example, for any look-back windows, I'll use whole weeks/months/years instead of just an arbitrary number of days (even if it's dynamic within a range). With the caveat perhaps that I may try to 'front-run' other algos using 'standard' lookback windows. Or front-run the front-runners, etc. :) If anyone has any better idea/suggestion on how to deal with 'parameter overfitting' I'm all ears!

In the meantime, I'm doing a bit of Python training (and trying to learn more about key relevant stats concepts in the lecture series), so if there's no update to this thread the next few days, that's what I've been doing. :)

[4. Alpha Factor Research & Discovery (Cont'd)]

Ok, so next I'd like to test these factors:

• Correlation to each other during the training period.
  
• OOS to see how well/poorly they hold up individually out of sample (i.e. how overfit each of them are to the training period)
  
• Correlation to each other during the OOS period.

The OOS time period is exactly 8 years before my training period, and is the same length as my training period. I chose this period because I want to see how well/poorly my factors hold up during the GFC period.

OOS test period: 2007-06-15 - 2009-07-15 (plus 2 months of 'future returns')

First though, let's see how correlated my three factors are during my initial training period. Thanks to Delaney's excellent notebook, I'm now able to do this. This is the correlation matrix I get during the training period for my three factors:

[ 1.        ,  0.44395672,  0.45470046],  
[ 0.44395672,  1.        ,  0.37135153],  
[ 0.45470046,  0.37135153,  1.        ]

Since this is really my first time doing this, I don't have any previous experience to compare to, but my initial reaction is that this is probably pretty good. All of them are somewhat equally uncorrelated.

Next I will test them individually during the OOS time period, and later also the correlation of returns during the OOS period. Honestly I'm expecting at least some 'overfitting' on all the factors, but I'm hoping not too much. Let's see...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor1 OOS testing: 2007-06-15 - 2009-09-15 (inclusive of 2 months of 'future returns'). This is the 'Fundamentals only' factor:

p-value(IC): 0.001 (below my cutoff)
IC Mean: 0.013
Risk adjusted IC: 0.151
Ann. alpha: 0.033
beta: -0.079
Mean Period Wise Return Top Quantile (bps): 2.776
Mean Period Wise Return Bottom Quantile (bps): -2.028
Mean Period Wise Spread (bps): 4.718

OOS testing of Factor1, comments: Since the p-value both during my in-sample, and OOS is below my cutoff of 0.05, I am going to 'declare' this one as a 'true positive' predictive factor. Furthermore, the (too good to be true) Annual Alpha has come down quite a bit, but the rest has either stayed mostly the same, or even improved slightly, which I'm positively surprised about.

I'm thinking the significantly lower Ann. Alpha is probably due to 'overfitting' but since everything else seems fine, I'll leave this one as is for now, rather than 'tweaking' and retrain on both this (GFC) time-series and my original in-sample period, and then test it OOS during a different time-series altogether. Let me know if I'm making a mistake with this decision.

Factor2 OOS testing to follow...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor2 OOS testing: 2007-06-15 - 2009-09-15 (inclusive of 2 months of 'future returns'). This is the 'OHLCV only' factor:

p-value(IC): 0.009 (below my cutoff)
IC Mean: 0.013
Risk adjusted IC: 0.114
Ann. alpha: 0.055
beta: 0.017
Mean Period Wise Return Top Quantile (bps): 1.907
Mean Period Wise Return Bottom Quantile (bps): -1.525
Mean Period Wise Spread (bps): 2.923

OOS testing of Factor2, comments: The p-value is again below my cutoff of 5% so I'm going to 'declare' this one also a 'true positive' factor. Everything else has mostly deteriorated somewhat, perhaps mostly due to overfitting, though perhaps some variance can be expected? Funnily enough, the beta to market value is exactly the same during both periods... Scary!

Since the factor (in my opinion anyway) didn't perform 'horribly' OOS I'm going to leave this one too as is. What I might do instead is to try to find a fourth (or more) factor trained on this (2007-2009) period, and test it OOS on my original in-sample period, hoping that the additional factor(s) may weigh up / neutralize the overfitting of the other factors somewhat. I may look at doing a final OOS test of the final combined model on a completely new time-period, to see if my 'finger-crossing-method' worked. There's probably a better way of doing this - please let me know if you know of one.

First though, let's see how my third factor performed OOS...

[4. Alpha Factor Research & Discovery (Cont'd)]

Factor3 OOS testing: 2007-06-15 - 2009-09-15 (inclusive of 2 months of 'future returns'). This is the 'Fundamentals & OHLCV combined' factor:

p-value(IC): 0.011 (below my cutoff)
IC Mean: 0.011
Risk adjusted IC: 0.113
Ann. alpha: 0.108
beta: 0.099
Mean Period Wise Return Top Quantile (bps): 4.673
Mean Period Wise Return Bottom Quantile (bps): -3.015
Mean Period Wise Spread (bps): 6.852

OOS testing of Factor3, comments: P-value is again below my cutoff, so again (thank goodness) another 'true positive' factor (hopefully). This one too had some 'variance' in the OOS test period, but it doesn't appear too overfit to me. Unfortunately I don't know how much 'normal variance' I should expect from OOS testing, and how much is required in order to say a model is way overfit.

Even though none of the factors performed 'horribly' OOS (in my opinion anyway), I'm going to assume that they are all at least somewhat overfit to my initial training period. I'm hoping that if I try to find a 4th factor, trained on this new time-series (2007-2009), it may help to 'neutralize' the overfitting effect somewhat. Hopefully I'll be able to find one that's also not too correlated with my other factors...

4th factor research (trained on 2007-2009 data) and discovery (hopefully) to follow...

[4. Alpha Factor Research & Discovery (Cont'd)]

Sorry, first here's the correlation matrix for my first 3 factors during the OOS time period (2007-2009).

[ 1.        ,  0.19727104, -0.14067721],  
[ 0.19727104,  1.        ,  0.33425805],  
[-0.14067721,  0.33425805,  1.        ]

I have no idea what to make of the negative correlation between the first and the third factor... I'm hoping it might be a good thing. If anyone knows, I'd be all ears.

@Joakim

Correlation goes from -1 to 1. 0 means that there is no correlation at all (which is barely never the case, even with some white noise, you will always find a bit of correlation). Myself I would state that 2 sets are completely uncorrelated if -0.05<corr<0.05.
A positive correlation between A and B means than if A is positive B is likely positive. A negative correlation is the opposite. It does mean that if you have two factors which are negatively correlated you have factors which tells you the "opposite" on is saying long, the other short... Might be a good thing in fact, because you might want to discard such trade. If it was very high anti-correlation that would be bad, as you would have to variable (which should be descriptive) which tell exactly the inverse of each other!

But I say this with no financial background but just what I know about correlation function.

Thanks @David,

I appreciate your help! What I don't quite understand is how two factors that both seem to have some predictive power of future returns, and both which have positive alpha, can have negative return stream correlation to each other. Perhaps the negative correlation is coming mostly from the 'noise' part of the factors, i.e. the parts that are not predictive (most of it).

For now, with my limited understanding of it, I'm happy with them not being too correlated (positively or negatively). In other words, the closer to 0 absolute value correlation the better (as with beta to the market).

I think it make sense.

The point is 0.14 is quite low. There is a bit but not so much anti-correlation. I would interpret it int the following way: There is moment where factor1 works fine a does the good prediction. There is moment when factor2 works well too. Those moment are not overlapping, but are shifted one in respect to the other. Then the negative correlation might be a sign that there is a biggest amount of return/lose when the two factor are predicting the opposite. Might not be the length of the period, but just that at the moments it does not agree the lose/return are stronger. You need to think about the sum in the correlator, there is something that make this sum to be negative... But indeed it must come from the part which is not predictive, otherwise it does not make sense at all.

0.14 is quite low! white noise versus white noise (generated by a uniform random genrator give easily something like 0.2 if one use only 1000 points, and goes to 10e-5 when you use 10e12 points..... Will be hard to find something less correlated than white noise :-). There is always some correlation that comes from whatever/whoknows.

Bah your post really motivated me to find out how to extract meaninful imformation on correlation of factors! Ill let you know how it goes, but hell I have to fight with panda which I really have never used!

Thanks David! I'd certainly be interested in what you might find out. A bit too difficult for me to tackle...

Could the correlation figure be 'skewed' by return 'outliers' perhaps?

I also wonder how much 'random variance' one could expect in the correlation figure. Do you think it would be safe to say that any figure between -0.5 and 0.5 indicates that the factors are mostly not correlated?

Hi Joakim -

Regarding your comments on universe selection, I think the prescription from the Q team is to write factors that run over the entire QTradableStocksUS (QTU), versus restricting the universe on a per factor basis. I'm not sure I understand this guidance yet. For example, say I restrict the universe for a given factor to the most volatile stocks in the QTU, and it works well. Well, instead of restricting the universe, I could instead build a penalty into the factor that shifts low-volatility stocks toward zero alpha value (assuming that the alpha vector is de-meaned). From an architectural standpoint, perhaps this is better, since I then don't have to keep track of which universe applies to which factor. However, in effect, one is still restricting the universe--the universe-limiting operation is just applied within the factor.

So, I think the way to go is to write factors that run over the entire QTU, and effectively select a QTU subset within the factor itself.

It would be interesting to hear from Q on this point. It goes without saying that the best factors will be the ones that are the broadest, and thus rank across the entire QTU. But then this would say that the hypothesis for every factor would need to be based on some broad market inefficiency, which doesn't sound like the way to go--with 200,000 quants, I'd think you'd want them trying to find idiosyncratic alpha that when combined provides an advantage over your typical brick-and-mortar hedge fund.

So first step in my factor correlation notebook.

Here is a small function which return the correlation between 2 factors. The correlation is computed on a daily basis.
Next step, compute the correlation on a asset basis.

Hope it helps

Hi Grant,

So, I think the way to go is to write factors that run over the entire
QTU, and effectively select a QTU subset within the factor itself.

Yes, this is basically what I'm doing. Your first sentence and paragraph seems to contradict this though, or am I misunderstanding it?

Basically, here's my rationale for universe selection:

• Start with as broad a universe as possible (the QTU is great, and is always my 'base universe')
  
• Ideally, as you said, a 'strong' factor should work fine on just the QTU itself. However, those factors are quite hard to find in my experience?
  
• If I have a good 'economic rationale' for restricting the universe further, and both my in-sample and OOS data supports my rationale, it should be fine, as long as I don't restrict the universe too much. That said, I try to avoid looking at my in-sample data before selecting a more restrictive universe, so I don't just 'data-mine' and overfit my model. For example, if I see that my model doesn't appear to be very predictive for stocks in the Utilities sector, even if it's supported by OOS testing, I try not to exclude the Utilities sector from my factor universe unless I have a good rationale for, and think I understand fully, why my model is not predictive for stocks in that sector. In short, I don't want to 'fit' my rationale to the data (I'm very likely overfitting if I'm doing this).
  
• I don't want to restrict my 'factor universe' too much. For example, if I plan on having 800 positions (400 long/short each) in my strategy, I wouldn't want to restrict my factor universe to anything lower than 800 stocks. I'd rather have a buffer of 200, so a universe of 1000 stocks (or more) might be fine.
  
• If one of my factors uses only half of the QTU stocks, but another factor uses the other half of the QTU, then effectively in aggregate I'm using the full QTU, right, as long as the factors are equally weighted?

In short, unless I have a really good economic rationale (and think I understand it well) for using a subset of the QTU universe, I just use the QTU universe.

I've probably contradicted myself there a bit as well, and may also be making 'obvious' mistakes, but that's one big reason for why I'm doing this - to hopefully learn from mistakes I'm making as I'm going through the process. :)

Hi David,

Thank you! Definitely helps!

Do you know if, when we say 'correlation' here, it's really mean correlation over the period tested that we're talking about? If so, there should also be a 'variance' from the mean correlation that could be measured? Would you know how to do this?

Let me know if I'm completely off the mark here... :)

@ Joakim -

The basic idea I was trying to get across is architectural. Each Pipeline factor gets the entire QTU as an input, and anything that limits the universe or weights it would be self-contained within the factor. I had started down the path of a code structure that would allow a different universe to be applied to each factor, via Pipeline masking (e.g. for some factors, use only the most volatile stocks, and for others, the entire QTU). One ends up having to match factor_1 to universe_1, factor_2 to universe_2, etc., which is not hard, but realized that it doesn't follow the architecture on https://blog.quantopian.com/a-professional-quant-equity-workflow/ which shows a single universe as an input to multiple alpha factors.

The other thought I have is that effectively one is applying a kind of transfer function in limiting the universe, and that it probably should be smoothly varying for continuous variables (e.g. volatility).

@Grant,

Ah, gotcha, now I understand. That might indeed be a better and more proper way of doing it. Honestly I don’t know. For now though I’m sticking with masking with either full QTU or a subset universe specific to the factor. For me, this is much simpler, and I like to keep things as simple as possible (but not simpler :)).

As long as my subset universe is still fairly large I think this is fine, though I have been known to, on rare occasions, be wrong before. ;)

Just for info, two timeseries can have a correlation of -1 and still have a very positive average. You must remove the drift.

[4. Alpha Factor Research & Discovery (Cont'd)]

Just a quick update that I've found what appear to be a number of 'false positive' factors. They looked fine during the new in-sample training period (2007-2009), which I'm using for trying to find a 4th factor, but then 'fall apart' when testing them on OOS time-series. Note that I'm using my original in-sample training period, 2015-2017, as OOS for the 4th factor, which may not be 'best practice'.

I'll keep training and testing (conscious not to 'data-mine' too much), to hopefully find a decent 4th factor that's not too correlated with the ones I already have...

@Joakim

When I we speak about correlation there is no mean or variance. You basically compute a quantity of the form:

$$\sum_n x_n y_n$$

So you take the sum over n. I my example I compute the sum everyday and n labels the assets. One could think to do the opposite. Computing the sum for each asset with n labeling the dates. That would be completely different things.

Then if you look at the output of my Notebook. As we have the correlation at every day, one can evidently compute the "mean correlation" and its standard deviation, or every statistical quantity that one could build on some data :-)

@Ben Lieblich

Just for info, the correlation function is independent to the mean (I expect that what you mean by the average).

Thank you David, I really appreciate it! Please let me know if you think there’s anything I can help you with.

Bah lets explain why it is independent of the mean.

One can think of the correlation function as the angle between 2 vectors. The vectors components are each of your data point. So if you have n data point, your vector is in a n-dimensional vector space. Then the correlation function is simply the \(cos(\alpha)\) with \(\alpha\) been the angle between the vectors. So for any dataset you can find a second dataset with any means that give a given correlation (if your dataset have at least 2 points). The point is that you have 2 equations with n unknown, this is always solvable and therefor you can arbitrary (so as large as you want) set the mean of your second dataset.

Thanks for the offer :-). Right now, I am reading some books, then I might have some question. I am new to all this, so I still need to learn both panda and finance lol

[4. Alpha Factor Research & Discovery (Cont'd)]

Training a 4th alpha factor on my original OOS dataset, and testing it on my original training dataset, has proven to be quite difficult. As I would like to move on to the next step (5. Factor Combination), I'll go with a factor that's quite correlated with my 1st factor.

Factor4 In-Sample testing: 2007-06-15 - 2009-09-15 (inclusive of 2 months of 'future returns'):

p-value(IC): 0.003 (below my cutoff)
IC Mean: 0.008
Risk adjusted IC: 0.129
Ann. alpha: 0.518
beta: 0.004
Mean Period Wise Return Top Quantile (bps): 2.327
Mean Period Wise Return Bottom Quantile (bps): -2.839
Mean Period Wise Spread (bps): 5.719

Factor4 OOS testing: 2015-06-15 - 2017-09-15 (inclusive of 2 months of 'future returns'):

p-value(IC): 0.017 (below my cutoff)
IC Mean: 0.008
Risk adjusted IC: 0.104
Ann. alpha: 0.018
beta: 0.148
Mean Period Wise Return Top Quantile (bps): 0.881
Mean Period Wise Return Bottom Quantile (bps): -2.490
Mean Period Wise Spread (bps): 3.375

Correlation during IS time period:

[ 1.        ,  0.24745288, -0.1784698 ,  0.63263895],  
[ 0.24745288,  1.        ,  0.35489379, -0.15707995],  
[-0.1784698 ,  0.35489379,  1.        , -0.26880649],  
[ 0.63263895, -0.15707995, -0.26880649,  1.        ]

Note: The slightly different correlation figures for the first 3 factors (compared to the previous table above) is most likely due to my widening of the individual universes just slightly for two of the factors.

Correlation during OOS time period:

[ 1.        ,  0.413451  ,  0.45278498,  0.8448823 ],  
[ 0.413451  ,  1.        ,  0.42739565,  0.03151596],  
[ 0.45278498,  0.42739565,  1.        ,  0.43143329],  
[ 0.8448823 ,  0.03151596,  0.43143329,  1.        ]

I'm not too happy with these numbers, especially that this last factor appears to be quite correlated to the first factor, but I'll go with it for now as I'd like to move on to the next step.

(5.) Factor Combination to follow...

5. Alpha Factor Combination:

I'm not really sure I can do anything fancier than just summing the zscores of the factors with equal weights, but I would like to explore the following:

1. Summing equal weights of factors' zscores (my baseline)
   
2. Use Machine Learning, as outlined in this 3-part series, to assign more weight to recently more predictive factors.
   
3. Some other dynamically weighted factor model (maybe)

The first option will basically be my baseline; any other weighing scheme would need to prove to be better than just summing the zscores. Essentially, this is my 'worst case' scenario, which hopefully won't be TOO bad.

I haven't had much experience with ML at all, and I've just briefly played around with Thomas' notebooks and algo templates (with not much success), but I would like to see if I can make use of his templates to come up with a superior factor combination.

The third option I may not do at all, will see how I go.

Baseline, normalized equal weights, factor combination (on new OOS time period) to follow...

You could also consider unequal weights of the factors. For example, you could weight the sum of the z-scores by the inverse volatility in the returns of each factor. More weight to less volatile factors might up your SR.

I don't know how to do this in Pipeline, but I'm guessing it is doable.

no such thing as interesx or not.

Thanks Grant! Yes, that was basically my plan for #3 above, which I'm not sure I'll get to at all because I don't know if I can figure out how to do it... :(

I was basically thinking of giving more weight to recently better performing factors based on recent risk-adjusted-returns. Maybe a 63 day rolling window of absolute returns / 63 day rolling window of std of those returns (as in the 'contest score'). A contest-score 'momentum-based' factor weighing scheme essentially. Or the opposite with a shorter look-back window, so a 'reversal-based' weighing scheme. Or some combination of this. All of this sounds to me like it may be quite prone to parameter over-fitting however...

It doesn't feel like it should be too difficult, but I may need to reach out to the Q Community for help if I get too stuck...

[5. Alpha Factor Combination (Cont'd)]

Following will be some screenshots of AL tearsheet outputs, from the combined (summing of zscores) alpha factor on a new OOS time series. This will be my 'baseline' essentially that any other factor weighing scheme would need to beat.

The new OOS time-series I chose is: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns).

I'm not sure what to call this period, maybe either 'Greenspan's Goldilocks Economy' or 'Housing Bubble Buildup'? :) It's mostly a 'bull market' which my original training time-series (2015-2017) was as well, but hopefully distant enough to be dissimilar in terms of any other common market regime. I haven't measured this though, which may not be a bad idea.

Let's look at the stats tables first... Many thanks to @Grant for instructions on how to share an image on the forums, and how to quickly store images on Imgur without an account.

[5. Alpha Factor Combination (Cont'd)]

1. Summing of zscores Factor Combination

OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

The first thing that stands out to me is that the Turnover Analysis table looks really bad (way too high turnover, right?). Maybe Autocorrelation as well? Unfortunately I don't quite understand these very well yet, and obviously didn't pay much attention to them during my Alpha research, so that's probably a big lesson for me to learn from.

The rest looks ok to me (beta is a bit high, at least perhaps in relation to alpha?), but please let me know if you think something else looks bad, or if you have any other feedback on the Turnover and/or Autocorrelation numbers.

[5. Alpha Factor Combination (Cont'd)]

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

This one looks ok to me. Perhaps one thing to say is that the 'alpha' appears to be quite fast, which I'm not sure is a good thing? Anyone know of a way of measuring alpha decay?

Perhaps the spread of the 1 day returns between the first and the 5th quantile is not wide enough to survive trading costs?

[5. Alpha Factor Combination (Cont'd)]

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

This one looks really bad to me. Ideally I would have like to see the mass of the violin plots to be significantly below the zero line for the first quantile, and significantly above the line for the 5th quantile. And also not so narrow and long... I can't really tell much from this graph unfortunately. Sigh! Maybe it's time to go back to the drawing board... :(

[5. Alpha Factor Combination (Cont'd)]

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

This one I'm fairly happy with (could be a bit smoother and consistent though). :)

[5. Alpha Factor Combination (Cont'd)]

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

Another graph I'm fairly happy with (though could be better of course). Good thing because I was starting to get a bit depressed after looking at the Turnover table and Violin plots... :)

[5. Alpha Factor Combination (Cont'd)

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

Far from perfect, but ok perhaps for just a 4 factor (some fairly correlated too) equal weighing combo? Ideally the smoothed-out moving average line should all be positive. Let me know what you think.

[5. Alpha Factor Combination (Cont'd)

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

Also pretty happy with this one. Or should the IC mean be higher and IC std be lower? Let me know what you think.

[5. Alpha Factor Combination (Cont'd)

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

Another two graphs I can't really tell if what they show is good or bad. I'm leaning towards bad. The normal distribution curve I think could be narrower and higher (a lower std), and more clearly on the positive side. The Q-Q graph, I think, should show more of an S-shaped curve ideally? Let me know if I'm wrong. Are the outlier dots good or bad?

[5. Alpha Factor Combination (Cont'd)

Summing of zscores Factor Combination
OOS time-series: 2005-06-15 - 2007-09-15 (inclusive of 2 months of forward returns)

This looks pretty good as well I think. At least during this time-series. Note that I didn't exclude any sectors for any of the factors in my alpha research, even if they showed negative IC during the training and initial OOS time-series. Maybe I should have, but I was too worried that I might be overfitting if I did... Thoughts?

Hi Joakim,

Many thanks for sharing your method of designing a good, competition-grade algorithm. Would you perhaps also be willing to tell us whether your existing algorithm that has been doing so well in the competition is also based on HLOCV data and the fundamentals? As you have mentioned yourself, using any alternative data sources involves payment, so I have wondered for a long time now whether one can create a top-notch algorithm based on the freely available data here at Q only. If his is indeed the case, I would be much heartened, I must say.

Cheers,

Tim

Joakim,

Looks pretty damn good to me, other than turnover. I myself have been struggling with turnover in my research... can never seem to get a good trade off between alpha decay and turnover. One theory of mine is that the more factors you have ranking your universe, the higher the turnover, as many of the ranks I use are subject to daily changes.

I have been planning on testing a method of combining similar factors (i.e quality, value) into single factors within the pipeline, and ranking by the average of that sum (thus reducing turnover?). Hopefully this method will reduce turnover and still provide a good signal via the 'comprehensive' factors. I am not sure if that is applicable to your research, but I will be sure to report back if I find anything signifcant.

Thanks for sharing your process, it is very insightful.

@Joakim,

From above post, you stated that:

Take the PEG Ratio for example, perhaps the most commonly used value-to-growth ratio for a GARP type strategy.

Perhaps you are aware of problem-with-peg-ratio-in-fundamentals

If you are still using peg_ratio as a factor, how do you account for this in your OOS time-series: 2005-06-15 - 2007-09-15 when it seems like Morningstar only started providing this data starting from 2014-06-03. Did you do a workaround in calculating the ratio for your OOS data?

@Tim,

Thanks, but honestly, my original PARTY (Pure Alpha Risk pariTY) algo's recent performance could just be pure luck. Too early to tell in my view. Perhaps it just does well in the recent market regime. That said, I'm hoping that it might be less overfit than some of my later algos. It does use only free datasets, and mostly fundamentals and historical price data.

In terms of just winning a bit of cash in the contest, I'm hoping it might still do reasonably well in the next month and a half, since the strategy's returns were fairly flat 2-3 months ago (till around 16th of July), so I won't lose any of those returns in the score. See screenshot below. Orange Possum's score is very impressive though and might be out of reach for quite a while. If that algo is less than 3 month's old, his/her score may be rising still...

If I ever finish this one, I plan to research a few premium datasets, and if I find any decent alpha there, use some of my contest winnings to subscribe to those datasets.

@Luke,

Thank you, and yes, turnover is a major concern for this one I think, and something I've clearly overlooked. Before proceeding, I think I may need to go back and investigate the individual factors again to better understand what the heck is going on with turnover. Worst case I may need to remove one or more of the factors, or at least modify it/them to bring overall turnover down. Ideally I would like to have average daily turnover less than 20% (before constraining MaxTurnover).

@James,

I hadn't seen that post before (thank you for linking to it), but yes, I'm aware of the issues with the peg ratio. A few posts down I had stated:

The AL template uses the PEG Ratio, and as @Blue Seahawk pointed out,
there seem to be something strange with this ratio, as it's not really
updating very frequently for about half of the symbols in the QTU. For
a ratio with 'price' in it, I would have expected it to update daily
for most symbols.

I'm not using Morningstar's peg-ratio in any of the factors. Basically I just wanted to use 'some' GARP-type factor, one which I didn't think had much alpha in it, in the Alphalens template I shared. Obviously I'm using only data that's available from at least 2005. :)

I was basically thinking of giving more weight to recently better performing factors based on recent risk-adjusted-returns. Maybe a 63 day rolling window of absolute returns / 63 day rolling window of std of those returns (as in the 'contest score'). A contest-score 'momentum-based' factor weighing scheme essentially. Or the opposite with a shorter look-back window, so a 'reversal-based' weighing scheme. Or some combination of this. All of this sounds to me like it may be quite prone to parameter over-fitting however...

First, I'd consider some sort of weighting based on factor volatility only. As I recall, there is discussion of this approach here:

Systematic Trading: A unique new method for designing trading and investing systems
by Robert Carver
Link: http://a.co/d/1DvH4Ls

Rather than computing the returns volatility per factor within the algo, one could just estimate it from Alphalens (and assume, relative to other factors, it will persist). I'm not sure that AL spits out the right number, though. I think one would want something like the standard deviation of the "Factor Weighted Long/Short Portfolio Cumulative Return" but I don't see it. I'm guessing it is accessible, though.

Thanks Grant, I appreciate it! I might try that too if I can figure out how to get the factor returns volatility from either AL or from within pipeline.

Note that the sum-of-factor-z-scores alpha combination model is implicitly dropping higher order terms in a polynomial expansion of z-scores (i.e. one could consider the full response surface up to a given polynomial order).

"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." John von Neumann