Hi Grant,

My understanding of this requirement is just for Q to ensure that the author understands and has a rationale for why the strategy has worked in the past, and why it should continue to work in the future. For example, if I create a strategy without any sort of rational idea behind it as to why it should work (e.g. data mining), the strategy is a lot more likely to be overfitted (fitted on noise from the 'training period' essentially) and likely not work very well outside of the in-sample training period.

More of a general 'qualitative' requirement, but also something more specific than just 'buy low sell high,' which they may want to confirm with the author before allocating any capital to it, again to reduce the likelihood of allocating capital to strategies that are more likely to be mostly trading (successfully perhaps in the past) on market noise.

Hey Grant,

You make a very valid point, Without the benefit of seeing the code, Q will have a very hard time verifying the truthfulness or veracity of the author's expressed strategic intent. But maybe, Q just wants to poll and account for the similarities and/or uniqueness of the different strategic intents to ensure diversity. Say, they find many strategies are of statistical arbitrage, they can put a cap on that particular strategic intent to accomodate for other strategic flavors.

Another issue Q is not clear on is the execution intent of the strategies and the subsequent author's payout. I have read in other forum posts Q staff mentioning that the execution of these portfolio of independent L/S market neutral strategies would be to leverage them many times over. This is consistent with Steve Cohen's (the principal investor) fund execution strategy as articulated in this article cohen-point72-s-reveals-high-leverage-as-firm-recruits-new-money So my question is, will the author be paid 10% of net profits based on one unit leverage as is by contest design/evaluation or 10% of net profits as executed at many times leverage?

Hi James -

I'd forgotten about the leverage part. I've asked this question before, and the answer was that the author gets 10% of the net profits after leverage (e.g. see Dan Dunn's response here: https://www.quantopian.com/posts/how-much-can-the-top-player-slash-developer-can-earn-in-quantopian-1#57f7b941d1c12eb54800004e).

One risk to the fund of the Strategic Intent requirement is that it potentially introduces bias. Rather than just going with the data, it introduces a qualitative judgement call on what will be admitted. The right set of words may bias the decision. Here are some words/phrases to pick from:

• Alternative data
  
• Multiple alpha factors
  
• Based on published research
  
• Factors studied extensively with the Quantopian research platform
  
• Uploaded novel, proprietary data

By just going over the Quantopian material, I think one could reverse-engineer a nice Strategic Intent statement that would match the current thinking/bias of what would be best-suited for the fund.

The Strategic Intent basically suggests that Quantopian can't evaluate black box algos on their "exhaust" alone--it would be too risky, and so a qualitative judgement needs to be applied, based on self-reporting by the quant. This would seem to have the potential for injecting bias into the 1337 Street Fund, right?

To borrow from Jess's remarks in the webinar - they need to be convinced that there is predictable positive alpha in the approach. I guess they want to make sure it was not just some luck factor strategy arrived at by trying numerous combinations.

Well, it would seem that if the backtest is long enough, combined with a long enough out-of-sample period, then explaining the performance (particularly without the accompanying code), could add risk/bias rather than reduce it. Another thought is that it may be a kind of false assurance to customers that they aren't investing in a cobbled-together black-box fund. Further speculation is that there may be some regulatory requirement to provide an explanation of what is under the hood--gotta say something in the documents provided to investors, right? Or maybe Point72 simply requires it? Whatever the case, it is not obviously a good thing...

Exactly!

In a coin-flip prediction contest, out of 1 million monkeys, 500,000 will correctly 'predict' heads or tails. Out of these 500,000 winners, 250,000 will again correctly 'predict' heads or tails. After 3 flips, 125,000 monkeys will remain that have correctly guessed heads or tails correctly in a row. After 10 coin flips, roughly 976 (on average) monkeys will have correctly predicted heads or tails IN A ROW!! Very impressive!

I believe the Strategic Intent criterion is intended to reduce the likelihood of Q allocating capital to one of these monkeys.

( :(|)

I think the problem is that monkeys, being monkeys, can be deluded, or just plain make stuff up. And the folks listening to the monkeys may believe them, or start to consciously or unconsciously apply their own judgements, feeling they know what will work in the future and what will not. I'm thinking Q should just "do the math" and not monkey around with this qualitative Strategic Intent mumbo jumbo. Shame on them, with the name Quant opian (I suppose they could always change the name to Qual opian...doesn't exactly roll off the tongue).

Math is powerful for sure, but it can sometimes be easy to be Fooled by Randomness.

Have a look at some of these apparent correlations. Do you think any of them are predictive of the future?

The idea I'm trying to convey here is that it is not clear what the author's Strategic Intent statement does to mitigate risk, and it may inadvertently add risk. One risk would be some form of cognitive bias on the part of the Quantopian judges. There's perhaps no substantive risk to the author, which makes the whole exercise even more suspect, but the risk to the fund may be real. I think the eggheads at Q need to noodle on this point, and consider confirmation bias that may creep in, along with other cognitive biases. They have the opportunity to do blind evaluations (anonymous author, anonymous strategy description), which is perhaps challenging/impossible in traditional hedge funds; why squander the advantage?

Hi,

Before I address the concerns raised here, I’d like to note that successful authors have a clear strategic intent. Besides being part of our criteria, defining your strategic intent is simply good quant practice. Your strategy may be economic, statistical, behavioral, or any other field from which you wish to draw. Whatever it might be, you are more likely to succeed if you write down your strategic intent before you start exploring it.

The theme of the concerns raised above is that Quantopian’s selection process will become qualitative and biased as a result of using strategic intent. I also read the criticism as Quantopian will somehow choose strategies with a compelling intent, in spite of failing other criteria. This simply isn’t the case. Our selection system begins with totally automated checks of the structural requirements in-sample, and cross-validation our-of-sample. Only strategies that pass these tests are considered for the portfolio, and at that point checked for strategic intent, as an additional guard against ovefitting.

thanks,
fawce

Whatever it might be, you are more likely to succeed if you write down your strategic intent before you start exploring it.

Well, you store the code of every backtest, so that it can be re-run, out-of-sample (e.g. after a minimum of 6 months of "aging"). Maybe full backtests that authors would like to be considered for the fund should also capture a strategic intent statement at the point of origination? This would preclude cooking up some explanation that attempts to explain the performance after the fact. Whatever statement the author puts down would be the one used for the fund allocation assessment, with no augmentation. Then you could do true hypothesis testing (e.g. the algo will do Y because of X, over the period T); the hypothesis can't be written after the test is performed, which is what you are allowing now.

I'd note that authors can do this voluntarily, by adding the strategic intent to their code. Then, if the algo is picked, they can simply give permission for Q to read the relevant lines of code.

Our selection system begins with totally automated checks of the structural requirements in-sample, and cross-validation our-of-sample. Only strategies that pass these tests are considered for the portfolio, and at that point checked for strategic intent, as an additional guard against ovefitting.

Sounds good, except that I'd still be concerned with injecting bias. If you have enough out-of-sample data, then there's no need. There will be bias if you allow qualitative words which are not formulated for a rigorous hypothesis test (before the test is run!); the subconscious is a powerful thing. There's also the strong influence customers and your VCs could have, should they express preferences--of course you'll tend to pick what they say they want, which may not be what would actually be in their best interest. And then there's whatever is in the "ether" of the hedge fund world--the risk of following the herd, thinking its the safest route (e.g. I'd heard that the 2007 quant meltdown was due to this kind of behavior).

Of course, if you aren't waiting long enough for out-of-sample data, then the strategic intent statement might mitigate the risk of over-fitting, for the worst cases ("I just sat down in my monkey suit at the keyboard and hoped for the best!").

Hi @Grant, i enjoyed reading your comments above, and i'm pretty much aligned with most of your thinking on this topic of so-called "strategic intent".
...... however, assuming that I really am clear on my strategic intent and happy to divulge it, then I have 2 questions for Q,

@Fawce
1) Ref. https://www.quantopian.com/get-funded, Strategic Intent: ..... "We’d like you to be able to explain the original idea and intent of the strategy to us before we license it". Please can you advise what is Q's recommendation / requirement for where, when and how this should be done?

2) I wonder what exactly does Q intend to do with this info? Even if it might have some use in categorizing algos, I certainly hope that it would NOT be used in any way as a means of determining the similarity between algos. Surely that should be better done on a purely objective basis using actual performance statistics?

Thanks Tony -

Very good point. The anonymous, black-box algo should be melded into the existing fund in a simulated fashion to determine if it would add anything, and a decision made, before the strategic intent is assessed, for the final check against over-fitting (I'm not convinced that it is effective, but I suppose Q could collect data in the coming years to support their process). It probably wouldn't be a good practice to know about the strategic intent prior to this point, and it should only be used to exclude algos that have already been accepted (e.g. the author spouts off some ridiculous strategic intent statement).

As a footnote, one thing I've realized is that the same analysis for the risk factors could be used to determine if a new algo X is already represented in the fund, and to what extent (i.e. how much of the returns are already due to X), without actually "knowing" what the algo does based on the strategic intent.

Yes, regarding your footnote, i think its a bit like listening to what people SAY or write about their beliefs or whatever vs. watching what they actually DO in practice in their lives. Q's Performance metrics and Risk Factors are a good, objective, albeit still partial & incomplete, set of yardsticks against which to measure what any algo ACTUALLY DOES, irrespective of whatever its author might believe or say.

One thought would be to develop a survey interpretable by a ML routine to systematize the Strategic Intent requirement. This would make it less loosey-goosey and more scalable to the hundreds/thousands of algos that potentially could be funded. The survey could be required to enter the contest and to be completed prior to a full backtest on any algo to be evaluated in 6 months.

One would have to think about how to obtain training data for the ML routine. It could take a year or two I suppose. No free lunch.

Hi Fawce -

In the context of the quant world, any information given up about a strategy is giving up IP. I'm sure that the NDAs that quants sign at hedge funds view things this way, so I'm thinking that you are basically asking for IP with the Strategic Intent requirement, right?

So, one take on this is that you are basically requiring members of the community to give up some of their IP, in exchange for a shot at making money. There's a kind of underlying monotonic relationship between the probability of getting funded, and the amount and quality of IP given up (picture an S-curve, for example).

So, what I suggest is having a look at your terms of use, to see if you can incorporate this basic exchange into the "5. Proprietary Rights" section. It's just fine to have the exchange part of your system, but it should be spelled out in the terms of use.

The exchange is not necessarily unfair, but it just needs to be spelled out to community members that they'll need to give up some IP to get funded (or wait N years for enough out-of-sample data so that the over-fitting risk has been mitigated).