Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
The Dogs of the Dow

This is the "famous" Dogs of the Dow strategy. I mainly continued the version posted by @ Jack Wood.

What's the Dogs of the Dow? Investing in the top 10 highest dividend yielding Dow stocks and hold them for a year. You can find out more about it here:
https://www.investopedia.com/terms/d/dogsofthedow.asp

UPDATE: Working version at the end of the post

What works:
Everything works fine if the backtest is run year by year, so the rebalance is made just once during a single backtest.

Problem:
If the backtest is run over multiple years, then the stocks are not chosen correctly.

Does anyone have a clue of why that happens??

Thanks.

7 responses

Some logging that might help.

Click a line number in the margin for line{s} of code to set breakpoint(s), run it, then type or copy variable names in the console prompt that appears and hit [Enter] to examine them.

1969-12-31 16:00 get:110 INFO get(self, 2016-12-27 00:00:00+00:00),    year 2016  
stocks ['MMM', 'AXP', 'BA', 'CAT', 'KO', 'DD', 'XOM', 'GE', 'HD', 'INTC', 'IBM', 'JPM', 'JNJ', 'MCD', 'MRK', 'MSFT', 'PG', 'UTX', 'WMT', 'DIS', 'PFE', 'VZ', 'CVX', 'TRV', 'CSCO', 'UNH', 'GS', 'NKE', 'V', 'AAPL']  
2016-12-27 05:45  PRINT .

2016  
    do_pipe, context.year 2016 == dt.year 2016  
    do_pipe, len(context.portfolio.positions) 0  
2016-12-27 05:45 before_trading_start:162 INFO context.year 2016 to dt.year + 1 = 2017  
2016-12-27 07:30 trade:119 INFO trade(),    context.year 2017  
stocks ['VZ', 'PFE', 'CVX', 'BA', 'CSCO', 'KO', 'IBM', 'XOM', 'CAT', 'MRK']  
2017-01-03 05:45  PRINT .

2017  
    do_pipe, context.year 2017 == dt.year 2017  
2017-01-03 05:45 before_trading_start:162 INFO context.year 2017 to dt.year + 1 = 2018  
2017-01-03 07:30 trade:119 INFO trade(),    context.year 2018  
stocks ['VZ', 'PFE', 'CVX', 'BA', 'CSCO', 'KO', 'IBM', 'CAT', 'XOM', 'MRK']  
2018-01-02 05:45  PRINT .

2018  
    do_pipe, context.year 2018 == dt.year 2018  
2018-01-02 05:45 before_trading_start:162 INFO context.year 2018 to dt.year + 1 = 2019  
2018-01-02 07:30 trade:119 INFO trade(),    context.year 2019  
stocks ['VZ', 'IBM', 'PFE', 'XOM', 'CVX', 'MRK', 'KO', 'CSCO', 'PG', 'GE']  

Thanks for the logging and for your consideration.

The attached backtest is run with 11 stocks instead of 10 on purpose.
It's clear that the algo choses the correct stocks the first year (2015). The following years would be correct if "T" wasn't returned as a dog. And that's exactly the problem.

The algo is not broken if run over multiple years, but if "T" is bought. In 2015 "T" was a dog, but in 2016 it wasn't, but somehow it's still returned as a dog. The backtest you attached before never had "T" and the dogs are always correct, indeed.

I have no idea why that happens.

Maybe closer. 'T' drops out in 2016. Main changes marked with #.#

We're definitely getting closer, but there are still some issues.

The actual strategy is to go long the first 10 stocks of the Dow (sorted by forward_dividend_yield), but having a look at the whole Dow is a great starting point to figure out why 'T' is returned and why there are some NaN values - these 2 issues are actually connected.
In the backtest I will hold 30 stocks. Once we address how to fix the universe from which to select the 10 stocks, I will change the algo to hold just 10 positions.

I noticed that in the 'trade' function, 'context.out.index' (which is the output of the pipeline) returns different stocks than 'context.out.loc[dogs].index' (which is the output of Dow30().get( get_datetime() ) called in 'trade').
I don't know why they return 2 different set of stocks, since they should work with the same universe - because they call the same function 'Dow30().get( get_datetime() )' - and since they are both called on the first day of the year. Or am I missing something, maybe about when point in time they are called? Because it looks like the 'Dow30().get( get_datetime() )' returns one set of sids in 'make_pipeline', and a different set of sids in the 'trade' function.

In 2016, actually the only difference is that 'make_pipeline' returns 'T' and not 'AAPL' - which is wrong - while 'trade' returns 'AAPL' and not 'T' - which is correct. However, the fundamentals data are fetched in 'make_pipeline' and that's why 'AAPL' shows NaN values (thanks for the function you added). So the NaN values are not an issue of 'forward_dividend_yield', but they are Nan because 'context.out.loc[dogs].index' - I think - substitues 'T' with 'AAPL' after 'make_pipeline' is called, therefore it's impossible to have fundamental data for 'AAPL'.

So, it looks to me that 'Dow30().get( get_datetime() )' returns the wrong stocks in 'make_pipeline', but it returns the right stocks in 'trade' - only 'AAPL' and 'T' differ.
Do you think this is the issue of the whole algo?

I hope to have been clear until now, if not I'll try to explain myself better (let me know).

N.B.
In the algo, I added some log.info at the beginning of the 'trade' function.
Also, the values returned in context.out are not sorted, but I will sort them in 'trade'.

I think I found the "glitch". Correct me if I'm wrong.

Basically, Quantopian doesn't provide neither DJIA index nor its members, so we call a function each time we need to get the Dow members.
'attach_pipeline' is only called in 'initialize', while 'make_pipeline' is called whenever we want. I think that, however, 'make_pipeline' only uses the data that is fetched at the time 'initialize' is run. E.g. if we run the backtest starting in 2015, 'make_pipeline' will always use the Dow members as of 2015, and will never update in the future. Instead, running 'Dow30()' outside the pipeline will return the correct members.

One solution would be to pass Q500US to Pipeline, so we always get the data for all the stocks, then we select only the members of the Dow outside of pipeline and we should have the members with the fundamental data.

Is there a better solution?
Can @Quantopian create a "dynamic pipeline"?

I will update the algo asap.

We're DONE!
Thanks Blue!

P.S.
Obviously some improvements can be made, but the logic is second to none.