So, I guess I should probably make some suggestions of my own first. I've been thinking about this and here's what I got so far.

Firstly, we can consider every asset in terms of it's expected returns, volatility and correlation to every other asset. Cash is just another asset that has 0 expected return, 0 volatility and a measurable correlation to the other assets.

Secondly, every trade should have a defined entry and exit signal. For these event-based algorithms, the entry signal would be when you detect the event. For ranking algorithms the exit signal is a time limit. You've owned the security for a month, so time to rebalance, right? But other exit signals might be a significant enough reversal, for example.

Putting that together, you could start out with an all cash portfolio. You want to maximize your returns given some maximum tolerance for volatility. So, when you detect an entry event, you find an optimal weighting for the new position, given the current positions (including cash), such that you maximize expected returns given your tolerance for volatilty. The volatilities and correlations would be inputs and I think you could express it as a linear or quadratic programming problem.

Then when you detect an exit signal, you would probably have to rebalance the existing positions to make sure your volatility tolerance is not violated.

I guess a question is whether, when you're adding the new position, you should consider rebalancing the other positions or only consider the problem in terms of adding the new position. Does it make sense to add to a position in the middle of period of being held, or should size be set only at the beginning. With rankings, you can always rebalance to the best possible positions right now, but an event-driven position might not be worth as much 10 days into holding it, right?

I guess another problem is estimating the expected return for a position. How do you do it? Do you base it on historical returns for events on securities with similar volatility? Is there another cost function you could use instead?

Anyway, just some food for thought...

Hello Rudiger -

I saw your post, and have had some of the same questions. My sense is that as the framework is laid out, the associated API improvements and discussions are not orthogonal to event-driven trading. The institutional-grade workflow "is a loop and the cycle time could be once a week, once a day, every 15 minutes, etc." The alpha factors will be on widely different time scales, so the cycle time would need to accommodate the shortest one. You might have "When the sun is visible in lower Manhattan, go long sunglass manufacturers, and when it is cloudy, go short." Every 15 minutes, you'd need to check the outdoor conditions, and crank through the alpha combination step. On a much longer time scale, there would be alpha factors based on company fundamentals, for example. Intraday, they would just spit out the same results over and over again, until new fundamental data became available (e.g. quarterly/annually).

The ranking (or ranking plus ML) is a way to combine a relatively large number of alpha factors (N >= 30?) in a systematic way, in the hope of getting steady (1+r)^n compound interest returns (that will need to be leveraged, since r will be relatively small). The portfolio construction step is just concerned with nudging the M-dimensional portfolio vector to point in a new direction, subject to constraints (where M is the number of individual stocks in the portfolio). This is what Scott Sanderson is doing on https://www.quantopian.com/posts/request-for-feedback-portfolio-optimization-api . It is event-driven, in the sense that the output of the alpha combination step includes all of the alpha factors on their various timescales. If I'm understanding the workflow correctly, some of the alpha factors could be binary--if certain conditions are met, then the forecast return will be Y = R (entry point), otherwise, Y = 0 (exit point). As long as the conditions hold, then Y = R, but as soon as they fall apart, then Y = 0. My sense is that if the workflow is constructed properly, even if you predict Y = infinity, it'll get converted to a rank, and then the portfolio construction step will further mute its effect. The ranking is a simple way of dealing with outliers, as I see it, but I'm still getting a feel for when one should convert to ranks, how it transforms the data, and the effect on performance. Information is being ignored to some benefit; apparently it is standard practice in the industry, but it is not clear to me why.

The question I've posed is that with the emphasis on writing factors in pipeline, it kinda rules out what a lot of folks would consider "event-driven" trading anyway. With pipeline, the timescale is daily, so one is really talking about pretty gross-level "events" that in my mind are more like trends. It won't be amenable to anything close to a 15-minute timescale, if pipeline factors are used exclusively.

Hey Grant. Thanks for your response. You've definitely brought up some important points.

One thing is that I've been thinking of this problem very abstractly, whereas in reality Quantopian doesn't have the means to write an algorithm that listens for specific events and reacts. And, as you mentioned, the amount of polling you can do is limited by the fact that you can only run the pipeline on a daily basis.

So, in light of this, I should probably rephrase the issue. The issue is that I might have a single boolean factor in my pipeline that is very sparse. Let's say on any given day, all stocks are "false" except for maybe one or two. The question is how to use a factor like this. If you just rank and select the top N, it wouldn't make sense, because you'd have to choose quite a few stocks that are "false" for the factor. You'd have to use a second factor as well. So, to make an algorithm that uses a factor like this effective, you'd have to accumulate positions over time, instead of rebalancing completely every so often.

As an aside, I was thinking about how these ranking algorithms that have periodic rebalancing are really an event-driven strategy where the number of positions is constrained and the exit signal is based on how long you've held a position. I wonder if you actually miss out on things that happen between your rebalancings. What about, instead of having a holding period be the exit criteria, you had the ranking decide when to exit? You could, for example, say if a position goes below rank R, then sell it and buy a higher ranked stock instead. I guess this would be more like indexing, but you could pick R to higher than the number of stocks you hold in order to let your winners run a bit before exiting.

But, back to the sparse boolean factor algorithm... I was thinking maybe it makes sense not to try to estimate the expected returns. Instead you try to maximize exposure, constrained by a maximum position size and a maximum volatility (based on the correlations and volatilities of the positions). So, each day you could see which new "events" had fired (i.e. which stocks are "true" for the boolean factor) and then see how you could incorporate the new positions into your current portfolio given the constraints of position size and max vol. The existing positions would be resized if the get too large and you'd have some kind of exit criteria to remove positions (and resize the others). I think this generalizes and would work for a regular numeric factor as well.

maybe it makes sense not to try to estimate the expected returns

I think you end up doing this implicitly. If you have N stocks that have been triggered as long/short, but do not have return forecasts for each, then you are just saying that the relative forecasts are the same.

Gotta run, but this is an interesting topic. The proposed multi-factor, multi-equity long-short approach would seem to be more akin to continuous-in-time asset allocation than even-driven trading (entry/exit points), but my guess is that the two are not mutually exclusive.

You might be right. But somehow I think it might be more of an assumption that high volatility stocks will get better returns than lower volatility stocks. My version of the daily algorithm tries to maximize exposure subject to a max volatility. So, in a sense you're trying to put on as much volatility as tolerable in the hopes that the risk will result in rewards. I could be way off on this.

I think with a multi-factor approach, you gain rankability. You can use ML to predict alpha for a security and then your optimization can be to get the best possible alpha subject to some rules regarding volatility and correlation. The only think I'd caution against regarding ML is to not use a linear model. The problem is that two factors can work well in isolation, but when mixed they don't do as well. So a linear combination of the factors would not produce the desired result. A possibly bad example of this might be value and momentum. I think you'd need something like random forest, where the decision trees can distinguish between different combinations of factors being good or bad together. The other thing I wonder about the ML approach with many factors is how far into the future you predict your alpha for a given security. What if some things (as you've suggested) work in the short run and other in the longer run. Lot's of things to consider.

Anyway, I'd love to learn more about the management of optimal portfolios, managing risk and other things relating to the practical problems that are not about finding sources of alpha (I think this is what people focus on too much). Does anybody have good sources of information on this (other than Quantopian's own, excellent lecture series, of course)?