Here's the result with margin. Seems too good to be true. Grant

I am not sure how easy it is to short these inverse/leveraged ETFs. Good avenue of research though!

EDIT: I should add, the reason the SH has a persistent downward trend is due to the geometric return penalty of rebalancing the ETF's position, which worsens linearly with the variance of the asset and the square of the leverage (if I remember correctly).

I'm curious - why did you bring dollar-volume into the equation as part of your mean reversion scores?

EDIT: also, it's so hard to tell with this IDE, but it looks like what it's doing is calculating the z-score of price*volume, using only that of the second stock (SH), and when that is higher than normal, it goes long SPY and short SH. I'm not entirely sure what you are aiming for here (I mean no disrespect!), but it looks like this strategy boils down to buying the dips (and shorting the equivalent SH pops) in what has been essentially a low-vol bull market for the last few years?

Thanks Simon,

Good point on shorting SH; I've wondered how a broker would treat it.

Regarding the dollar-volume, I started playing around with just prices, but it wasn't productive. I did some work with a cointegration test to understand what might be going on (https://www.quantopian.com/posts/augmented-dickey-fuller-adf-test-spy-and-sh-dollar-volume). I can't say I have a good feel for the fundamentals yet (and Quantopian isn't really a platform for data exploration).

I'll have a look at the code again. The algorithm should compute z-scores of the dollar-volumes of both SPY and SH. Then, it should compute the difference in the z-scores and use the difference to determine if they are "out of whack."

Grant

Agreed re: data exploration. What is the rationale for dollar-volume though, is it a mean-reverting process, and if so, does it revert via price or via volume?

I recall reading a paper which mentioned that the shorting/borrow costs for leveraged ETFs was in the neighborhood of 50%/year. I thought it was http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2211387&download=yes but it doesn't seem to be in that one.

Also:

http://falkenblog.blogspot.com/2011/10/shorting-leveraged-etf-pairs.html
http://www.math.nyu.edu/faculty/avellane/LETF_Dobi_Avellaneda_Sept2012.pdf
http://blog.quantumfading.com/2009/08/17/leveraged-etf-myths-1-shorting-both-bull-and-bear/

Lots of information out there about this strategy, in general!

Thanks Simon,

At this point, I don't have a rationale for the dollar-volume, other than a guess. My intuition is that the volume provides a weighting, so that only significant departures from the norm are used as a trading signal. But I haven't dug into the details yet.

Grant

Hello Simon,

Here's the backtest using volumes only--appears to be the dominant factor. I'll try prices next and post the result.

Grant

Here's the result run on prices (I changed the threshold to 5 to get a decent return). --Grant

Here's a long-only version with no margin that goes in and out of SPY. --Grant

Here's the same algorithm as immediately above, but with both long and short SPY positions allowed. --Grant

Interesting, I'll have to look more closely at this, definitely seems to be be something there...

Are you sure which stock's dollar volume you are using for the zscore? Eyeballing it looks like it's using SH. If so, then a rise in the volume of a short ETF might be a plausible indicator of market bottoms/turning points.

Thanks Simon,

I appreciate your input. I've gotta put it aside for now, but hope to get back to it shortly. My intent is to compute the SPY-SH z-score difference as an indicator, but it is possible that I botched the code. I'll have a look.

Grant

Hi Simon,

This code looks correct:

    # normalize dollar volume with z-score  
    dv_z = stats.zscore(dv, axis=0, ddof=1)  
    # compute z-score difference  
    end = dv_z.shape[0]-1  
    delta_z = dv_z[end,1] - dv_z[end,0]  

Why do you think otherwise?

Grant

I must simply not understand what you are trying to do, you're multiplying the mean of the price with the the sum of volume of each, so you end up with a matrix of P-V x stock, which you z-score for each stock. Then delta_z is supposed to be the difference between the P-V z-score of SPY vs the z-score of SH? So, if the P-V of one is an outlier but the other is not, you enter? Or is delta_z supposed to be the difference between z-scores of now vs last period for SH? (it's not clear to me what the dimensions of these axes are and I haven't had time to look into it).

The delta_z is supposed to be the current SPY-SH dollar-volume z-score difference (using the pandas rolling stats to reduce the data, prior to computing the z-score). My thinking is that the z-score difference should stay within a fairly tight range, unless something is amiss. So when the z-score difference is large, I treat it as a signal to trade. Unfortunately, the whole approach is murky at this point, since Quantopian doesn't allow free-form analysis of their proprietary data. I'll think about how we could get a more fundamental understanding of the mechanism.

Yeah, there definitely seems to be something there, I'll play around in matlab when I have some time, but swamped with work and studies at the moment.

Simon,

Here's a first attempt to get a feel for what's going on.

Grant

Hello Grant,

I checked this idea in my own system and discovered a significant difference between quantopian and original (from bloomberg, yahoo) data. The difference in traded volumes significantly decreases performance of strategy. Probably quantopian uses some adjustments for data, do you know about it?

Hello Ilya,

I am unaware of adjustments, but the Quantopian folks tend to be pretty responsive, so hopefully they'll have a look at your other post and help sort it out. Kinda strange that the difference would be so large.

Grant

Hi Grant,
Sorry, I'm new here and do not have some knowledge of the codes written in Python. (I rather understand things in C++ better...)
Can you briefly explain your strategy on this SPY & SH pair trading?
What time frame is this based on? 1 min.? 5 min.? and Intraday only?

Thanks.

Hello Kyu,

The basic idea is to normalize the dollar-volumes of both SPY & SH with a z-score. The z-score difference is used as an indicator.

if delta_z > threshold:  
        desired_amount = positions_value/data[context.stocks[0]].price  
        diff_amount = desired_amount - current_amount  
        order(context.stocks[0],diff_amount)  
        context.new_day = False  

    elif delta_z  

Kyu,

Sorry, part of my post got chopped off...I'll fill you in later.

Grant

Kyu,

When the z-score difference is above a threshold, SPY is bought, and when the z-score difference is below the threshold, SPY is shorted.

Trading is limited to once per day.

Hope this helps.

Grant

Grant,
So do you actually trade SPY only? But you are using SH with a z-score as an indicator?
What time frame is this based on?
Does this include transaction costs?
Have you tried this strategy with other pairs? If so, how did it go?

Thanks.

Hello Kyu,

I can't say that I have anything spectacular here. The trading is very infrequent, and for more recent timeframes, I haven't had much luck. I have tried other pairs (e.g. SPY & BND) but with the same mixed results. If you have any insights, please post them.

Grant

Hi Grant,

I've got one question here, why would you "skip tic if any orders are open or any stocks did not trade" ? (line 49)

Qingwei

Qingwei,

That is just a sanity check. If you have an open order, you can't place another order. If a stock does not trade, you can't trade anyway.

-Huapu

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2362971

wrt short selling strategies and borrow costs

Hello Qingwei,

Regarding your question, as Huapu said, the idea is to avoid placing new orders if submitted ones have not been filled. The check to see if actual trade data exists is a bit more subtle. Quantopian forward-fills missing bars (see the discussion https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps), so the check ensures that only historical trade data are used; if any filling occurred, the tic is skipped.

Grant

Grant,

I still didn't completely understand the logic of your algorithm. You will buy SPY and short SH if the z-score of SH's price-volume product is larger than that of SPY by 1.5, right? As you mentioned before the dominant factor is volume and price doesn't matter much. Then your algo basically is saying that if the volume of SH has a much larger increase than that of SPY, you should buy SPY and short SH? Logically it doesn't make much sense. If it is price rather than volume, it actually make more sense logically.

Also buy SPY and short SH may double your risk. If you buy SPY and SH at the same time, it is kind of like pair trading and it should reduce your risk. But if you always buy SPY and short SH, or the other way round, aren't you basically doubling your earnings/risk? Why you didn't try to pair trade SPY and SH, but the other way round?

Hi Grant, Simon & Huapu

Thanks you guys! I've got it! I never take it into account when writing my algo.

Hello Huapu,

Thanks for your comments. You are likely correct regarding shorting SH, and I abandoned the idea awhile back (above, you'll see algorithms that are long only or long/short SPY, with no trading in SH). One approach that might be viable would be to go long in either SPY or SH, for a long-only portfolio (rather than shorting SPY, for example).

Grant

Does this work well with other ETF pairs (i.e. SPDR Sector ETFs, country ETFS, etc.)?

Hello Nihar,

It's been awhile since I played around with this, but I do recall trying other pairs. In the end, I concluded that backtesting probably isn't the best approach to gain insight. An offline statistical analysis would be nice (which I have not done). Nevertheless, you could clone the algo and try alternative securities and settings.

Grant

unfortunatly i've needed to start using multicharts because of this exact reason. quantopian doesn't provide me enough detail on what's going at minute level. Seems like this should be trivial to add to quantopian. :(

I've been playing around with variants of this algorithm and noticed that the results vary widely depending on what rebalancing point you pick throughout the day. For example, rebalancing at 10am or 3pm produces wildly different returns (both short term and long term) than rebalancing at noon. Was there some logic to the choice of 12pm ET as the rebalance trigger?

Hi Matt,

I don't recall any logic, other than noon ensures that if the market closes early, a trade can still be executed.

Grant

Hi Grant,
Are you trading live with this strategy?

Will

Nope. --Grant

Hey guys!

I tried backtesting Grant's original strategy with margin, except I used UPRO and SPXU for the etf's. This should have the effect of tripling returns, right?
I still used the volume and price data from SPY and SH, however, because they work better for the signals.

How does this look?

I ran a clone of this on some other time-frames, and its performance was dubious. Caveat emptor.

This backtest uses all the available data for UPRO and SPXU.
The 66.8% drawdown is pretty scary.

This is what I got...

Your backtest is different because you changed the time of executions to 3:45 instead of noon. I also get these terrible results when executing at that time. Noon appears to work best, though I have not thoroughly optimized it.

Now that is interesting.

I'm not sure exactly what influence it would have, but the volumes stocks trade is quite different through the day (even ignoring the first 5 minutes of the trading day).

EDIT: I got it wrong - most stocks have the least volume midday. SPY has least volume from 12:45 to 13:15

Has somebody traded this system in live trading successfully?

Hello Javier,

I have not traded this live, and I haven't heard of anyone else doing so.

Grant

Interactive Brokers lists 750,000 shares of SH are available to short, any security can be looked up on their list here:

https://www.interactivebrokers.com/en/index.php?f=shortableStocks&p=shortable

[Edited] Could be helpful for someone to cover how any shorting limits could be addressed in code? At some point in buying them back, do margin limits apply?

I am not sure I follow, what are you talking about? To short shares, you need to have shortable shares from the broker, you need to abide by the current uptick rules (not even sure what they are right now), pay the short interest rate and pay any dividends. To cover, you just buy them back? Is there any more to it than that?

Thats is all there is.

If you're shorting in Q, you have to monitor your own margin intraday, they don't do it for you. I also don't see a way to model margin interest yet, but IB only charges 1.56% APY, which from an engineering perspective is essentially zero, so I'd be comfortable ignoring it a model and just focus on proof of concept.

Also, Q will not tell you that you do not have sufficient initial margin available to initiate a short sale of a particular size, you have to track that on your own.

Keep in mind as well that the proceeds from the short sale show up in "cash" in the Q model, which my be reinvested, as long as maintenance margin requirements are still satisfied, which, once again, you have to monitor on your own.

Hi everyone,

Shorting is nothing more than what Simon talks about (more or less), the only drawback could be that a)there are not enought shares to short (which is not the case because we have more than 15 USD million in IB from SH) or b) big interest charged (this will depend on Fed funds rates and also is dependent on leverage, making leveraged ETFs more expensive to short, which is not the case because SH is not leveraged but inverse). Finally regarding margin requirements, if you leverage your account proportionally to the drawdown of the algo, you shouldnt have to suffer much thinking in potential margin calls.

Taking this into account, is there any reason for not trading this algo live? I can't see any at the moment (even including commissions and slippage).

Thanks.

Here would be my top reason:

We don't know why this algo is successful. It is using the volume of ETFs, which doesn't seem like it ought to have a lot of explanatory power on the face of it. So, that fact needs to be isolated. Furthermore, it's very sensitive to the time of day the trade is placed at, which again is a mystery - is it exploiting some intraday seasonality? Is there some sort of late-day reaction to volume in the past 24 hours? Is this related to the effects of leveraged ETFs rebalancing?

Without knowing why an algo works, it's impossible to know when it has stopped working, which won't inspire confidence when it's in an expected drawdown or might incite hope when it's broken for good...

Well, I think that most patterns in financial markets (specially intraday patterns in price and volume) dont obey to a fundamental reason, but just to the strategy that big players are using to make money. One example is the Gold Silo strategy (short intraday long overnight), the strategy had amusing returns for about 10 years until the author of zerohedge made the strategy public and suddenly it became unprofitable (the reason is that Deutsch Bank among others retired from the strategy when it bacame public). Is there a fundamental reason to justify Gold´s higher price during the overnight session? Not at all, its just market manipulation, banks observed that lots of retail traders bought gold during the day session to sell it at the end, so banks decided to play the opposite game, move the price down intraday (when theres lots of retail volume) and make the money overnight (when there is no retail volume), this implies that banks would keep most profit from gold momentum for them.

So basically in my opinion, the fact that I dont find the fundamental logic behind the process doesnt let me down because the strategy seems to be robust. The strategy is robust simply because the out sample performance seems to be like the in sample one, therefore I believe the strategy is profitable (taking into account we have more than one year of out sample performance).

Agree?Disagree?

I have to agree with Simon that this algo would benefit from more development, including establishing some sort of working model that explains the performance. This way, one can apply the scientific method of collecting evidence and either confirming the model, modifying it, or scrapping it for a new one. In my mind, the algo results above are more of a clue that it might be worth doing a thorough study. My suggestion is that if anyone has time on your hands, give it a go and publish your results here for review.

My 2 cents:
The market price is determined by millions of humans. Some of them think the market should rise for reason X, others think it will fall for reason Y, and some don't care which way it goes so long as you can make money from it. I don't think there is anything wrong with having a trading system which works for unknown reasons so long as the risk can be capped. One day the system will stop working, but that is probably true of most systems. So rather than let your brain demand the satisfaction of a supposedly logical answer, just trade it and realise the clock is ticking (It has been posted publicly, after all). Even if you get an answer to why you think it works, there's no way of knowing for sure.

I agree with you James. I believe this system has proven enough to give it a chance, of course limiting the risk you take based on how the system performs up to date. Also, most systems before they die they show some symptoms which can give us a clue that some unusual bahaviour is going on which should trigger our alarms, at that point I would stop trading the system without waiting for it to fail completly. The problem is some systems use such a high leverage that, when one of those negative symptoms show (eg.a drawdown never seen before or an unsual increase in volatility), the account is already blown up. Managing the risk you take (eg, using margin as a function of volatility) is always crucial when trading whatever system.

When I run this backtest, the first transaction (2014-02-03) is selling 143 shares of SPXU that are not held. The market wouldn't allow this. Why does quantopian? Maybe that is an API bug. Anyway on to the problem.

In your order command, it seems as though the "position" value even when 0 causes some shares to be sellable.

order(etf,(-long_short*shares)-position)  

Am I reading that correctly?

Daniel - presumably it is short-selling?

SPXU is an ultrashort etf.... would be strange to sell the ultrashort instead of buying the SPXL. Shorting leverages etfs is very expensive. if you set set_long_only() it wont short

There is no API bug. Just like in real life, you are responsible for verifying that you are able to short the shares. Checking and monitoring your own initial margin available is simple enough.

80k shares of SPXU are shown as available on IB, It's possible, but limited.

Hi Daniel,

I have hacked the algo such that is does not sell short but buys the opposite. still decent performance but the code is really a hack so caveat emperor

Peter, it looks like your algo is functioning correctly for the +1.5 z-score threshold; albeit strangely in that is executes two UPRO orders individually (i.e., in the enclosed interval, 2014-02-03 two orders are placed for 59 shares instead of one, 118 share order).

However, at the -1.5 threshold the algo sells UPRO, again in two orders, but does not buy the short, SPXU. Instead it holds cash until the next +1.5 event. Just an FYI.

Hey guys I know this in an old thread, but I had some crazy results with dank meme's original leveraged etf post when I switched to SPXS, SPXL. Can someone help me update this to the most recent API. I want to integrate this strategy into robinhood after I limit the margin account

Ryan,

The algo is pretty dated. I'd recommend starting with a fresh sheet of paper and doing some major re-factoring (I have some other stuff to do). You might take a more fundamental approach to understand what might be going on in the first place, by using the research platform to understand if there is actually any information to trade off of.

Grant

Thanks Grant, I just started writing this weekend, so I'm having hard time understanding what is the theory for the original quant. To my best understanding, it buys/sells based off minutely data with portfolio%= (price-mean/std)/1.5 for long and short positions respectively? And then normalizing the z score (price-mean/std) with dollar volume?

Ryan -

There was no "theory" behind my hack. I just decided to try something. The basic idea was that since SPY & SH are trading off of the same underlying S&P 500, they should track closely on a dollar-volume basis, and when they do not, maybe there is some useful information. If you are just trying to get a feel for the platform, it is as good a starting place as any, but there was no deep thinking or careful analysis involved in the original formulation. And as other posters point out, in practice, it may not work at all.

Grant

Thanks Greg,

This is a pretty old thread, or I wouldn't change the subject, but I think I have stumbled onto a useful indicator formula.
Check it out and see if you can find it useful, but it kind of goes like this:
Dma= day moving average
std = 50dma standard deviation
(20dma - 50dma)/(2*std ) = x

2*std is the bollinger band, by dividing it I can detect squeezes. For SPY, a large value above around 1.9 seems to detect large random drops, but has to take a while to reset and find the next drop. Buying below zero seems to fair well as it is below the 50dma. Shorting with SPXL and closing after 5 days on those drops really seemed to boost returns significantly. It did detect one instance when it wasn't supposed to, but if you set a timer with that counts days holding cash/60 that increases the buy threshold to a max 0.5 it will buy back in accordingly.

Just what my project has been for the last couple of days. Had bugs in my minutely versions and might scratch it for a new one. Just thought I'd share.

Best,
RC

ha just realized I made a z-score

I think this falls into the general category of "ETF arbitrage" (e.g. http://www.investopedia.com/articles/investing/032615/how-etf-arbitrage-works.asp?lgl=bt1tn-above-textnote ), in case you are wanting to dig into it.

Ryan -

FYI -

https://www.quantopian.com/posts/analysis-of-minute-bar-trading-volumes-of-the-etfs-spy-and-sh

There's probably some way to apply the Quantopian alphalens thingy here, as well, if you formulate the "alpha factor" appropriately:

https://www.quantopian.com/posts/alphalens-a-new-tool-for-analyzing-alpha-factors

I also don't want to change the thread topic here too much. One theme however, throughout, is why dollar-volume would have any predictive ability with ETFs? Grant found an interesting correlation.

One way to look at index ETFs is that their price is simply a convenient way to track an index. The market makers for an ETF do their best to keep the price tracking pretty close to the underlying index or basket of stocks. One could create a factor to calculate this same thing directly from the stocks (in theory). Does the the price of an ETF therefore have any 'new' informational value? No. Also, the price is completely historical. For the technical traders out there, that is all that matters and past prices DO predict the future. As for me, I'm a little skeptical.

Note therefore, that index ETFs don't follow the same rules of supply and demand as regular securities. If there was a huge demand for a security it's price would go up. Conversely, if there was little demand, the price would go down. Index ETFs, however, have their price set 'artificially' to an external index. Index ETF volume is independent of price. If more people buy into an index ETF it's price doesn't necessarily go up (and vice-versa). There is no 'feedback' to influence the volume. Volume becomes almost a true 'independent' variable. Index ETF volume IS new information.

If a lot of people think a stock will go up (or a group of stocks like the S&P500), then they would all buy in hopes of making a profit. However, that profit is put in check by the Adam Smiths supply-demand rules. As they buy the price goes up. As the price goes up their returns go down.

Now, think of the case of an index ETF. A lot of people think the S&P500 will go up and buy SPY. As they buy there is no impact on SPY, or for that matter, no impact on the S&P500. The returns on their bet the S&P500 will go up aren't impacted by this volume.

I conjecture that index ETF volume is a good predictor of market sentiment and could be a future indicator.

As an aside, volume seems to be an under-utilized metric in general. Looking over the algorithms posted on the forums here virtually none use volume to derive factors other than in conjunction with price to get dollar-volume. I randomly took some of my algorithms and globally replaced 'close_price' with 'volume' - even for a factors like returns or volatility. The results were never too dissimilar and, in many cases, actually better. Go figure.

Dan -

I also don't want to change the thread topic here too much.

As the originator of the thread, I'd say your comments are right on the mark! For example, in the limit that SPY volume goes to zero, or captures all of the market volume, there would be some information, I'd think. So, more modest changes in SPY volume (perhaps relative some other indicator) might carry information, as well.

Grant, do you know why there are so many unfilled orders eod in the backtest?

Tony,

No, not sure why there are unfilled orders. You might try turning off the slippage.

Some tools added to the original code. Partial fills are visible in the logging window.
Move trading earlier toward market open to reduce those.

Ryan, some fixes you requested plus scheduling. A start to simplify zscore normalization.

p = pd.rolling_mean(context.prices,window_roll)[window_roll-1:]
v = pd.rolling_sum(context.volumes,window_roll)[window_roll-1:]

are going away - how do we replace these?