Hi Kyu,

I got at least a start going for you. I started with the previously shared hard coded list of 100 sids (if you'd prefer, you can swap in the set_universe() function to grab an unbiased dynamic universe of stocks), though I ended up just running it with the first 10 because it was a bit slow and I was just looking for a PoC to share (you'll see the larger list commented out and you can run the full test if this is what you're looking for).

Each day at the open (or as close as we can get at 9:31am) the algo will invest $10,000 per name in each stock where the "gap" is < (-1) * stddev for the prior 90 days of close to close percent changes. I didn't get the ranking in here yet - but is definitely something we can add.

Each day at the 'close' (3:59pm) the algo closes out all existing positions.

Also of note, this looked pretty bad when I ran it with the defaults for slippage and commissions costs in place, in order to see whether the underlying strategy works, I zeroed those out. But obviously doing 10 round-trips per day will certainly have the potential to racking up a lot of t-costs depending on your cost structure.

Hope this is a helpful start and I'm happy to iterate it further with you. Best wishes,
Jess

Jessica,

Your code looks amazingly compact and good. Thanks.

Please add how to code for ranking top 10 (or 5) list.

Question:
As this algo takes a little long to compute relatively simple math for 100 stocks (even with 10 stocks), I'm wondering if this algo can get the right number of stokcs in the list at 09:32 as the becktest results show. What if it takes longer than 1 minute (because this algo starts running at 09:31 ONLY) and thus it cannot buy any stocks then?
Basically my question is that why does it take so long?? It is not complicated computation though..

BTW, I will add more conditions and logics later on this. We will see.
Please help.

Thank you.

Anyone cloning the algo above should use sid(1406) for Celgene in place of sid(40207). My attempt at a list of NASDAQ 100 SIDs is here: https://www.quantopian.com/posts/list-of-nasdaq-100-sids-to-use-in-your-algo

P.

@Kyu,

Are you trying to bet on mean reversion within a day? My observation is, individual stocks can have trends that don't get reversed. They are even more unlikely to reverse if the gap is news driven. May be you could consider running it after the effect of sentiment has been removed. And sentiment is not just keywords. Sentiment in high beta stocks can get affected by macro news / price action as well.

@bharath,
Thanks for your comment.

What I am tryng here is that just to create one of the popular models we quant use. Especially, this algo (well, it's not completed yet, far far yet.) is based on the rationale that on days when the stock index futures are down before the open, certain stocks suffer disproportionately due to panic selling at the open. But this panic selling is over, the stock will gradually appreciate over the course of the day. (of course, this is not for all the stocks.).

Yes, this is a mean reversion strategy intraday. And of course, we need to play with this adding more features like you mentioned. My co-workers use event-driven algo as well. Or even in the opposite way, we can apply this strategy concept for trend following strategy, which I saw worked well too.

We can do anything we want.

But ladies & gentlemen, trust me, there is no single strategy winning all.

From my experience at hedge fund and all my co-workers' experience, there is no such a thing. Some strategies work well for three-four years. And suddenly they don't work for next three - four years. Why? Because market changes all the time. And traders' & investors' behavior change and finally technologies develop and change entire market action. In my career, I've seen a lot of times QPM fail as they just stick to one or two strategies.

I respect all the members here and all the quants trying to develop winning strategies. But please do not waste your time on digging just one single strategy that you think or believe will win all. In this complex and rapidly changing market, we all need to deploy all reasonable (not a fantastic) strategies with statistically solid rationale, and also very importantly combine them with well-planned money management. And then keep doing research and develop new one, and adjust your strategies until you quit this job. That's all we quant do.

Again, thanks.

@Peter
Thanks for your Nasdaq 100 listing.

@Jessica, how to do ranking?
And as I asked above, why does it take so long to compute this relatively simple computation?
With this, can this system buy stocks in the list at open (09:32)?

Thanks.

Hello Kyu,

Have a look at https://www.quantopian.com/posts/how-do-i-shift-the-moving-average-of-an-imported-csv-chart where AM ranks securities by standard deviation of gap size. You get a mention.

I don't know much about timing python code. Have you tried?

import time

def handle_data(context, data):  
    start = time.clock()

    ''' Do some stuff here with 100 SIDs '''

    elapsed = (time.clock() - start)  
    print elapsed  

P.

@Kyu,

I believe there is potential in this mean reversion strategy. The problem lends itself well to be modeled as a Markov Chain. Like you said there is no one size fit all. We can do 1 of the following things.

1) Build a model and try to find the asset classes for which the model works.
2) Select an asset class and try to build model for it.
3) Build a model and keep learning the changes in the market microstructure. This is easier said than done !

@Kyu

There are two issues w.r.t. performance and execution times actually.

First, the fact that you are seeing your first execution times at 9:31am is actually the earliest execution time that you will see, please refer to my detailed answer to Erik on execution times here: https://www.quantopian.com/posts/at-jessica-stauth-question-about-trade-at-specified-time

Second, we are always working to make our backtester faster - what tends to slow things down is a combination of universe size and time range/resolution as you might expect, so an algorithm that uses the upper bound of our allowed universe size for 3 years of minute level data will certainly take longer to run than a pairs trade tested with daily data over the same time period. That's even before you worry about whether you're running anything computationally intense on a per stock basis.

The best way to deal with performance woes IMO is to start off getting your algo working with a small test range of dates and stocks. Then scale up to your desired real test once you know the logic is solid, since we run this for you in the cloud, you can always kick off a backtest and come back later to get your results.

One other note: I did take a look at getting the ranking step you asked for working, but haven't had a chance to finish it. If you or anyone else is looking to work on that in the meantime I think what needs to be done is to use Python's in-place sorting method .sort() to do an in-place sort on the "gap" array that my code assigns in line 43 above. see: https://wiki.python.org/moin/HowTo/Sorting I got stuck because that method expects a series and for some reason the .diff() method handed back an array with an extra row of NaNs in there. Argh.

Hi Kyu,

I had a chance to revisit this algo today and to add the ranking and screening steps you outlined in your original post. You can see the new logic in the code with the attached backtest.

One thing I did change from last time, if I follow correctly what you are going for with step 4 (below), then I think the gap needs to be computed as the percent change in price between yesterday's close and the current (open) price, if we want it on the same basis to do z-score like comparison with the trailing close to close returns standard deviation. Might be missing something, but you can take a look at how I've implemented it here and let me know.

"4). First Find the stocks with gap low at open. If opening gap (open price - close of yesterday) is lower than one standard deviation with period of 90 days.
This stddev should be computed using daily close -to-close returns of the last 90 days."

Also note, I added a custom variable to record how many stocks pass the gap screen on each day, using a 1% dollar volume universe that number is often below 10 stocks in the last year or so - I interpreted your logic such that the algo only buys stocks that pass the screen of being more negative than -1 * std dev, so in those cases there are fewer than 10 stocks bought, and on many days no stocks passed the screen.

Hope this is helpful at least as an example to work from, doesn't look like it does well with this particular universe of stocks, but perhaps with a different stock universe it will work better - I'll leave that part of the magic up to you :)

Best regards, Jess

Thanks Jessica,
Your code looks great. Although the strategy is a bit different. I will modify it later on the top of the one you did. Thanks a lot.

@Peter, Thank you again.

@Anony, I don't know what the source of your plot is. But there are a lot of serious researchers who have been doing great jobs working on the opening gaps (Julie Dahlquist, Richard Bauer, Stephane Reverre, etc..). And also there are many successful QPM who have been using the opening gap strategies and making profits consistantly . I respect them and their work. There are many different strategies regarding gap strategies as you might know. So I am not going to make any comment on your comment, but thank you.

Thank you for the explanation, Anony.
Your plot says something valuable too although it's just for a 5-day return analysis.

I wanted to say that there are so many researchers out there and they have been doing great jobs with enormous data and putting endless effort in the analysis of opening gap strategies. From their work, I just wanted to introduce you guys several successful algos that work in real market. (This one is originally based on SP500 stocks.).

Here in Quantopian, we have some limitations including order generating system. This opening gap strategy (it's actually not completely introduced yet.) should be computing, sorting, picking, and sending buy orders at market open, usually at 09:30 (EST). But Quantopian system cannot do so. The earliest time to send orders is 09:31 and orders get filled at 09:32. Yes, in real market, it could be also hard to get orders filled at 09:30 too. But we (algo) do all the work before 09:29 and send orders at 09:30:10. So we may have different result from the work done originally in real market.

Also, when you guys work on with daily data, here your orders get filled at next day close. I personally think that it is non-sense. You better implement your work with the daily data to minute data.

Anyway, keep up the good work and keep challenging.
I will come back later after adjusting this strategy.

Thank you all.

just my 2 cents on the algo why not going long and short at the same time to be market neutral => if gap up short the stock & long SPY. if gap down go long and short spy ? decoupling from market condition the algo

I apologize for re-hashing this post as it's a few years old. I'm trying to reverse engineer this and I have a few questions. As I've been devoting a lot of time to learning algo coding.

In Jessica's first code. I'm not understanding why in the for loop at the end of the day she used data in "for stocks in data"? I don't understand the "data" part of this. Why didn't she go in the direction of something like "for stocks in openpositions"...or just loop through the open positions in the portfolio. I'm just not seeing where "data" comes into play. Came someone expound on this?

Another question. In Jessica's second code...I see she created a dictionary of sid's in universe....context.sid was defined; however, I don't see where "context.sid" was used in her code? Am I missing something? I'm assuming that anytime she used "context" in a argument of her functions...it automatically used context.sid in the function? I don't know, I'm lost on that.

Lastly, I see Kyu mention his co-workers use event-driven algos. Question, is it possible to have an algo determine if a stock is gapping with news or if it's gapping on fluff?

Unfortunately, you're working from a pretty old algorithm. Our API has gotten a lot more powerful since 2014, and had a major revision almost two years ago now. Some of the questions you're asking about how to think about lists of stocks is covered in Quantopian 2.

In general, I'd suggest that you take the idea you're looking for and start it again, or find a more recent example. Trying to learn a strategy and trying to understand partially-obsolete code at the same time is a painful task.

As for gapping on news - Quantopian doesn't have a direct news feed. But we do have some social media sentiment data that you might be able to apply for that purpose.