Thanks for sharing a clean code.
I think context.leverage is reserved as a variable, better use something different.

Alex, I am especially grateful, having asked you personally about this. And, again, think it's so generous of you to freely share such valuable work with others.

Bravo!

On first blush, the algo looks great. (Going to play with it now!)

Stephen

Alex,

I'm told the model works with Total Return instead of Excess Return, effectively setting the risk free rate to zero. You might consider that implementation:

# Scoring  
#  
# excess return momentum  
mom = (hm.ix[-1] / hm.ix[-2]  - hb.ix[-1] / hb.ix[-2] + \  
       hm.ix[-1] / hm.ix[-4]  - hb.ix[-1] / hb.ix[-4] + \  
       hm.ix[-1] / hm.ix[-7]  - hb.ix[-1] / hb.ix[-7] + \  
       hm.ix[-1] / hm.ix[-13] - hb.ix[-1] / hb.ix[-13]) / 22  
mom[mom < 0] = 0

Stephen

Stephen,

You're right, we'd need a Total Return time series. AFAIK the Q neither provides TR data nor programmatic access to dividend payouts yet. When this becomes available we can back-fit this algorithm.

I think we need the total excess return over risk-free for the price momentum score. Using short-term bond momentum is kind of a hack. In the paper, section 4.6 provides a condensed version of the "EAA recipe":

ri is computed as the average total excess return over the last 1, 3, 6 and 12 months (not 9 months) where excess return is defined relative to the (13 week) TBill yield.

Actually, it should be possible to fetch and integrate dividend data from Yahoo. This would be a great improvement over the existing code base. Maybe someone feels adventurous. :-)

What about using SHV or SHY dividend adjusted price from Yahoo as approximation?

VY

I think they're both good options. SHV, since its inception in 2007, has certainly lived up to its safe haven status. Its price has been very stable and SHV should be minimally vulnerable in a rising rate environment. Either one makes good sense.

This is interesting--the strategies with the best equity curves are the ones from my blog =P. Thanks for the advertising once again ^_^

-Ilya (author of QuantStrat TradeR)

VY, I'm sure SHY or SHV would work fine. I've looked at the Fetcher docs, but didn't have the time to dig into them yet. If you have experience getting Yahoo data with the fetcher, I'd appreciate your input.

Alex and VY,

I'm more of a database programmer ...a mere fledgling wanna-be in python. But here's code I wrote to bring over yahoo data for a database when I wanted to snag a LAST price for a ticker I wrote into a field:

PHP_Execute( "

$getfield = 'GetField ( \"Asset Components::Ticker\" )'; $ticker = fm_evaluate($getfield);

$link = \"http://download.finance.yahoo.com/d/quotes.csv?s=\" . $ticker . \"&f=l1gh&e=.csv\";

$set = ini_set('memory_limit', -1);
$curl = curl_init();
curl_setopt ($curl, CURLOPT_URL, $link);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec ($curl);
curl_close ($curl);


echo $result; " )

You guys might look at that and go, "Ahhh."

Hope it helps.

Stephen

As many of you know, it’s the ability to adhere to a strategy that counts …in the long run. And what makes it easier to adhere to a strategy? Minimizing drawdowns.

Look at the relative performance of EAA from 2008 to early 2009 and the latter half of 2011. I think this is where the EAA recipe really shines.

I believe this may help:
https://www.quantopian.com/posts/yahoo-and-fetch-comparing-q-and-y-for-12-mth-rolling-return

and a strategy based on either yahoo or quantpian data: https://www.quantopian.com/posts/yahoo-slash-quantopian-using-y-adj-as-signal-for-swtich

Note that fetcher only allows 5 xls fetch, thus you may need to write an external fetcher consolidator.

Florent,

That’s a really useful implementation and a great article on paired-switching. Now, if we could snag implied volatility data out of yahoo, we might exploit iv’s predictive value. (…lots of articles supporting this in SSRN eLibrary under the search terms “implied volatility predictive”)

Your algo also has a handsome performance plot. Do you think this approach could be expanded to three risky assets, drawing from a small universe and rotating around a volatility forecast?

Stephen

Florent, nice! I'll incorporate your fetcher code when I have some more time. It might mean cutting down the number of assets to 5 for now, but let's see.

Alex,
Hope you won't mind the intrusion here, but I read through your scripting, and a few papers you posted on EAA. I must say- what you have here is rather remarkable.
Though, may I make a suggestion, to introduce slippage into the model. It would more accurately show past performance. I'm working on doing that myself (but I'm rather new to Python).

Corey - Quantopian includes a volume-driven slippage model by default unless instructed otherwise. See https://www.quantopian.com/help#ide-slippage

This is a great defensible, robust algorithm. I really like it. The code is clean and very well documented. Thanks very much for spending the time to do so.

Since Quantopian offered us minute-resolution data, I tried using the same strategy and re-balancing the asset every 30 minutes. The preliminary result is not at all satisfactory, perhaps the play between momentum/volatility/correlation is a whole different ball game for intraday data.

Edit 14MAR2015: the effect of constant negative return is caused by slippage, once slippage and commission are neglected, backtest performance improved.

@Yoyo T., you'll find that commissions are eating into that strategy. Trade too often without much profit and you'll just piddle your account away. The original poster's monthly rebalance periodicity reduces these costs to a fraction of your's.

The strat runs fine on daily data too due to the fact of the EOD rebalance schedule. Try starting the strategy in the early part of March 2009 -- you might get conflicting results.

@Market Tech. Thanks for your insight! I thought about commission too and had set the commission to zero to in the code I posted earlier. I was hoping to get results at least as good as yours, however, the return was negative. I think the framework of the algorithm is really nice, just have to figure out an appropriate definition for z.

I spent some time experimenting with Florent's fetcher code and found a way to get up-tp-date adjusted data from Yahoo into the regular history data-frame format. Using adjusted prices makes quite a difference depending on the instruments used, e.g. with REITs. You can switch between Quantopian data and Yahoo data with the "use_adjusted" flag. If you use adjusted data, the fetcher limits you to max 5 different instruments.

(I hope the traffic from repeated back-testing doesn't get The Q blocked from Yahoo servers)

Alex, They say, “The enemy of good is better.” Well, you just disproved that aphorism.

Implementing Florent’s lead towards dividend and split adjusted prices, v0.6 shows a substantial reduction in risk without sacrificing returns: Sharpe improves by 34%, max drawdown decreases 33%, and portfolio level volatility comes in at a very respectable 10%. This version is also a truer estimate for a real-world application. Nice work Alex and Florent!

Great post Alex.
I've added a new term to the momentum score (z) which tries to capture the intraday volatility (iv),[1] which I will call the Golden Lama EEA:
w ~ z = ( r^w_r * (1-c)^w_c ) / ( v^w_v * iv ^w_iv)
where iv is defined as:
iv = mean(daily_high[-10:]-daily_low[-10:])/mean(daily_high[-280:]-daily_low[-280:])
The return improved a bit as compared to the Golden Offensive EAA, while volatility and max drawdown remained relatively unchanged.

-- Golden Lama --
Total Returns: 217.2%
Sharpe: 1.71
Volatility: 0.10
Max Drawdown: 11.9%

-- Golden Offensive EAA --
Total Returns: 197.5%
Sharpe: 1.65
Volatility: 0.10
Max Drawdown: 11.7%

[1] https://cssanalytics.wordpress.com/2015/03/13/using-a-self-similarity-metric-with-intraday-data-to-define-market-regimes/

Hey Ted, nice mod. I looked at replacing daily "vol" with intraday data as well, with mixed results. Your "Golden Lama" score seems to be quite a bit more stable than my initial attempts. Thanks!

Alex, the cash proxy is IEF in this implementation. Last night, ahead of the Fed's announcement, I noticed that flows into IEI were up 535%. Recently, it looks like institutions use SHY and IEI as cash equivalents when moving out of equities.

the algorithm seems break after platform upgrade, when backtesting it responsed following:

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.
ValueError: No axis named 1 for object type
There was a runtime error on line 143.

Did anyone implemented the improvements described here :

http://indexswingtrader.blogspot.co.il/2015/01/a-primer-on-elastic-asset-allocation.html

its a Hedged, Offensive settings for the keeller algorithem which provides higher returns and more important, none of the years has a loss and the biggest monthly drawdown -%6.5 versus ~-%11 in the offesive EAA.

http://a.disquscdn.com/uploads/mediaembed/images/1618/9084/original.jpg?w=800&h

@Novice TAI
I ran into the same problem, as well with all iterations of this program when forward testing. I've been working on it for the last few days, to no avail.

@Corey, Novice TAI: Looks like the upgrade broke my sketchy workaround (and one of my contest algos). I'll look into this when I have some time for TheQ in the next couple of days.

@Joe Lee: When you look at the source code, you'll find the "Weights" part of the configuration section. "Equal Weighted Hedged" is the scoring setting you're looking for.

A plot of cumulative return of the ETFs used in the default strategy, along with SPY.
This demonstrates how well the strategy is, i.e. minimal drawdown with the final return (198% or 217%) close to the best performing ETF (250%) used. Note the different cumulative returns for SPY shown in the image (~200%) with those in Quantopian(~130%), which is likely due to my return is the simple cumulative return, when Quantopian likely accounted for slippage/commision...?
image here

%matplotlib inline
import matplotlib.pyplot as plt  
import numpy as np  
import pandas as pd  
from zipline.utils.factory import load_bars_from_yahoo

start = '2003-12-31'  
end = '2014-12-31'  
ticks = ('MDY','IEV','EEM','QQQ','XLV','IEF','TLT','IEF', 'SPY')  
data = load_bars_from_yahoo(  
    stocks=ticks,  
    start=pd.Timestamp(start, tz='utc'),  
    end=pd.Timestamp(end, tz='utc'),#pd.Timestamp.utcnow(),  
    indexes={})  
prices = data.minor_xs('price')

log_returns = np.log(prices).diff()  
compound_returns = (1 + log_returns).cumprod()  
compound_returns.plot(figsize=(14, 6))

changed the code so that he non-fetcher version works again

Hey Peter, thanks for the quick fix.

Looks like there might be a pure pandas solution for the fetcher problem with multi-indexing. If you're feeling adventurous: http://pandas.pydata.org/pandas-docs/stable/advanced.html#advanced

I wrote a small app you can read multiple tickers via one call with fetcher so we are not limited to one fetcher per ticker .
The output is something like this:

Date    open_vti    high_vti    low_vti close_vti   volume_vti  adj_close_vti   open_efa    high_efa    low_efa close_efa   volume_efa  adj_close_efa   open_icf    high_icf    low_icf close_icf   volume_icf  adj_close_icf   open_tlt    high_tlt    low_tlt close_tlt   volume_tlt  adj_close_tlt   open_ief    high_ief    low_ief close_ief   volume_ief  adj_close_ief  
2/01/02 106 106.34  104.61  106.25  1627600 41.55   119.85  119.96  118.96  119.96  1215900 28.74   84.5    84.85   83.09   84.85   35200   24.44  
3/01/02 106.46  107.39  106.35  107.39  354400  42  120.1   121.04  120 121.04  971400  29  85.23   85.3    85.07   85.22   35400   24.55  
4/01/02 107.8   108.41  107.29  108.1   487200  42.28   122 122.07  121 121.84  377400  29.19   85.3    85.32   84.53   84.76   9600    24.41  

We only have to extract adj_close_SID in the algo

Don't call it very minute though ;)

http://peterbakker.pythonanywhere.com/?stocks=VTI,EFA,ICF,TLT,IEF

the flask code to generate the csv:

from flask import Flask, make_response,request  
import pandas as pd  
import pandas.io.data as web  
import datetime as dt  
import io

app = Flask(__name__)

# Route '/' and '/index' to `index`  
@app.route('/')  
@app.route('/index/<stocks>')  
def index():  
    #output = request.args.get('stocks')  
    # Get data from fields  
    stockstr = request.args.get('stocks')  
    stocklist = stockstr.split(",")  
    StockData = []  
    startdate = dt.datetime(2002,1,1)  
    enddate   = dt.datetime.utcnow()  
    DataCollection2 = None  
    i=0  
    for stock in stocklist:  
        StockDF =  web.DataReader(stock, "yahoo",startdate , enddate )  
        #print >> sys.stderr, StockDF  
        StockDF = StockDF.rename(columns = {'Open':'Open_'+stock,'High':'High_'+stock,'Low':'Low_'+stock,'Close':'Close_'+stock,'Volume':'Volume_'+stock,'Adj Close':'Adj_Close_'+stock})  
        StockDF.columns = map(str.lower, StockDF.columns)  
        StockData.append(StockDF)  
        if i==0:  
            DataCollection2=StockDF  
        else:  
            DataCollection2= pd.merge(DataCollection2, StockDF, left_index=True, right_index=True, how='outer')  
        i+=1  
    buffers = io.StringIO()  
    DataCollection2.to_csv(buffers,encoding='utf-8')

    csv = buffers.getvalue()

    if csv==None:  
        csv='error generating content'  
    response = make_response(csv)  
    file_name = 'DATA_'+stockstr.replace(',', '_');  
    response.headers["Content-Disposition"] = 'attachment; filename="'+file_name+'.csv"'  
    return response

I was playing a lot with in sampling the algo with various parameters, there is great weakness of the algo in the year of 2004, basically the algo return is minus 11% for almost a year the while the benchmark goes down ~%3.5. the main risk is that the algo goes down the 10% in 14 days only between april 2 to april 14, this is a great volatility and risk. it seems the algo , due to its monthly rebalance, didnt catch the turmoil of its ETF's, Attach is the backtest with charts of the algo ETF so you can see they werent going down per the algo.,

if anyone have an idea how to detect and eliminate this fast turmoil, please help.

Extremely nice concept. Would it be possible to explain the crash protection mechanism?

@Bharath. The existing crash protection is very elegant, it allocates more 'cash' when more 'z's are set to 0, due to negative 'mom's (weighted returns).

@Joe That is a good pick. You can always use schedule_function to proactively rebalance the portfolio with your ideal trigger and frequency, below is just a not-so-elegant example. Use of volatility maybe another option for the trigger. Would be great to share your results if you figure something out!

# your trigger here  
def mama_bear_is_here(h_price):  
    ret = h_price.pct_change().ix[-1]  
    num_neg = ret[ret <= 0].count()  
    N = len(context.active)  
    cpf = float(num_neg) / N  
    if cpf > 0.95:  
        print "look at that bear."  
        return True  
    else:  
        return False

#call below function with schedule_function for proactive portfolio management.  
def rebalance_now(context, data):  
    h_price = history(5, '1d', 'price')[context.active]  
    if mama_bear_is_here(h_price) is True:  
        reallocate(context, data)  
        rebalance(context, data)  

Thanks Ted

Thanks Ted and Bharath for responding. I adapted the solution above and did massive tests.
Observations:

1. This type of CPF wasnt enable to eliminate the 12% downturn occured in less than 14 days in 2014 and last the whole year. I played with various CFP thresholds as well as tried to check CPF and balance every day, week, and bi-weekly

2. Checking and rebalancing more than bi-weekly creates a signifcant reduction in overall performance and gain (11 years period). Regardless, the CPF system didnt help.

3. something might be wrong with the history function , the values I am getting for the active stocks doesnt seem similar to what yahoo/google shows

conclusions:
1. The CPF system calculates how many of the stocks in portfolio were negative, but it doesnt know if it was very small decrease or very high in %. It seems we should try momentum indicator like RSI or ADX to detect large downturn,
2. Need to check the history function validity + why more frequent rebalance cause such a big reduction in overall perfromance (10 years period)

attached is the up to date with the new crisis rebalance and logs. If someone can help me and continue the work it will be great.