Got excited with the new release of the FactSet Fundamental Dataset and quickly whipped up an algo that is very simple yet robust with low volatility over a long backtest. I fondly call this algo OTP (One Trick Pony)!
Question, how do I get feedback on the results of the 1 year holdout period?
Here's the short NB:
@ James, well done without wasting any time setting the bar high for the rest of us haha. Did you notice any surprises in the Factset data set which were not a part of the original Fundamentals? I.e were there particular ratios or inputs used that were not available previously?
@Daniel,
Thanks. To be honest, I haven't gone through the reference documentation thoroughly yet so I can't really give you meaningful answers. I just replaced the fundamental factors from an old algo to what is more or less the Factset equivalent. For this particular algo, there is a slight improvement.
Hey James,
"You down with OTP? Yeah you know me!" :)
Regarding this:
Question, how do I get feedback on the results of the 1 year holdout
period?
I think it would be cool if Q could provide some sort of 'lagged overfit feedback' maybe on contest algos that have been running live for >63 days? Then we could make a better informed decision to pull or not to pull the algo from the contest. I don't think they could provide this feedback too often or too soon, because then we would eventually just tweak/fit our models based on the feedback, and the 'hold-back' OOS period would soon become part of our training set. With a 'feedback lag' however, that would be much less likely I think, though maybe 21 days would be long enough (or even shorter)? Ideally some sort of more granular 'overfit range' would be cool i think, rather than just binary overfit/not-overfit.
Just thinking out-loud again... :)
Hey Joakim,
On pondering about my question further, I think one can get "instant" feedback on the 1 year holdout period simply by entering the contest and passing it's contest constraints and threshold tests. I believe Q runs the mandatory latest 25 months backtest for their pass/fail test. If this is correct, then one can see the results of the 1 year holdout period, if it passes the constraints test by running backtest on contest panel. This is under the assumption that Q releases FactSet data on 1 year holdout period when it does it's pass/fail test.
But that said, measurement of overfitting is another matter because of complex considerations like length of holdout period, alpha decay, luck factor, regime changes, etc. Live/real money trading results over time would be the most accurate indicator of overfitting/not overfitting, that is and by itself, a risk even after exhausting all the tools and techniques it applies to OOS data to determine overfitting.
Yeah, very true. I just submitted a FactSet Fundamentals based strategy to the contest, so will find out soon enough. Will report back here when I do. :)
Hey Joakim,
Just joined you on the "Hunt For Red October":-) Submitted my FactSet entry. We'll see if we get results on 1 year holdout upon approval of entry.
Contest algorithms using FactSet fundamentals should be evaluated from 2 days prior to the submission date through today, just like any other submission. If your result indicates otherwise, please let us know!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Hi Jamie,
Did you mean 2 years prior? :)
Indeed, that's what I see. This one doesn't look too overfit to me, so I'm quite happy with it. It got a score of 0.114 on the first day in the contest too, which I'm also quite happy with. Just need to do that every day and I'm golden... :)
@James,
Sorry, didn't mean to hijack your thread... Looks like your theory was correct though, which is cool!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.
In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
