Hope the release of the article goes well
This could be one of those opportunities to be tackling some of the nitty-gritty intricacies of shorting and maybe even some code by someone for handling any extra costs of selling borrowed shares, if that's possible, and/or other things involved in shorting. Maybe time frames? I'm not sure how all of that works, like when settling comes due etc.

Market Tech,

Thanks, that looks really interesting and I think it'd be great if this would happen much more often as it's quite difficult to try and reimplement a strategy from a paper. In addition, it would be very interesting to also publish a continuously updated forward test of the strategy alongside the paper.

@Thomas W., If there were DAG code in Python then the dynamic reconstruction of the DAG group from the correlated pairs would undoubtedly be possible. And advisable. Were such capability to be found or written it sure would be nice to have a better file structure into which one could store often used code modules. I sure hope that feature is at the top of the list for updates. Unless the research facility is going to offer such a capability that code modules can be bolted to the side of any strategy...?

@Market Tech: Do you think that your analysis could be done with the help of the networkx library (https://networkx.github.io/)?

@Thomas W., I see the reference to DAGs in your link: algorithms.dag and I see some of the familiar verbiage regarding nodes (vertexes) and edges. But I'm not sure, on a cursory look, how to retrieve the critical path from the DAG. If you can figure out how to do that then it looks feasible. Correlation is easy to come by I would guess. And that's all you need.
The article relied upon the entire SP500 as its input set but the 200 available from data[] (or more if one uses the fundamental technique Grant discovered), would be adequate. A walk forward reselection + rebalance test would be educational.

doesn't do so well in bear years, try 2007 - 2009

Well, testor testor, I don't doubt it. The intent was not to show the efficacy of the strategy, but to demonstrate the technique of using a DAG generated group. Perhaps if you have a better rebalancing strategy you'd like to share, the groups referenced within the code offered could be tested to see if the technique holds up regardless of the strategy used.

@Market Tech,

I'd suggest thinking about using zipline in the research platform, where there is no limit on the number of securities, and you'll have more flexibility.

There is a limit on RAM, but if you run into trouble, I'm sure Fawce would pay for a few more GB's, given that he's using your work to promote Quantopian (https://twitter.com/fawceisfawce/status/578953140973805570).

Grant

@Grant, Thanks for the recommendation. Coding in python is still like knitting with ovenmitts on for me. Everything I've ever presented here or anywhere I wrote in C# first. Slowly though, I'm starting to think in python, which, as we know, is key to becoming adept in a language. The one thing that Quantopian offers, that I've commented on before, that is nearly impossible on any other platform, is this drop-dead-simple portfolio management and testing.

RE: the articles; the February article was essentially ignored (in the TA world that is (here too though)). The fact that it focused on candlesticks put quants off I think. But the article was all about how to coerce OHLC numbers into statistically measurable patterns that could be tabulated and used as probability fodder. Regardless, it made the cover image! A python slithering over a candlestick chart. That was all tribute to Quantopian (in a way).

April's issue garnered the same treatement! www.traders.com The cover image is the Quantopian referenced article - DAG (nabbit!) I'd write more but, well, it doesn't pay. You should write for them. Seriously. If enough quants here wrote articles S&C might make a monthly Quantopian section...

No articles for me. Got plenty to do. Congrats on getting published. --Grant

Just a quick question: If I run the algorithm in the minute mode, Will the RSI, which is calculated based on the following lines of code, be the RSI over the past 14 minutes??

trendPeriods = 14
rsiIndicator = ta.RSI(timeperiod = trendPeriods)

The reason I am asking this, is becoz when I try to run the algorithm in minute mode, I get a very poor performance. Even, when I use schedule function an only make trade at the end of the day, the performance is really bad and the transaction are so different from the transactions in daily mode...

Is there anyway, I can run the algo in the minute mode (for example for paper trading), and calculate the RSI parameters in daily mode ??

Thanks

@Ali N., I'm going to spend an hour, soon..., and rewrite the strategy so that the methods for rebalance will run on any periodicity. Note that this strategy is only a research tool to gauge relative group performance in the hopes of determining the usefulness of using a DAG style of security selection.

I subscribed to S&C mainly off the back of your last article and this looks equally interesting, thanks.

@ Market tech. I totally understand that at this point the strategy is more or less like a research topic. I wanna do some work on top of what you did. I did lots of research in grad school about DAGs and I believe it can be potentially a promising path. However, first I love to run it properly and then take further steps. Since it is your code, I think it will be easier for you to make the changes for different frequency rebalancing. I take it from there and for sure I will share with you where I will end up too

@Mark L. I'm happy you found the information useful.

Ali N., I'm in the throes of trying to rework this technique such that we achieve the same results as it currently exhibits, but in a modular, extensible fashion.

There are many(*) basket trading strategies here. I would hope that perhaps we could find some of those and test the various groups referenced in this strategy using them. Maybe they work, maybe they don't. But it might give you clarity while I get this redone.

• that's what Quantopian excels at - basket rebalancing strategies.
  

@Market Tech. Sure.. Sounds good. I actually have been trying to rewrite your code since 2 hours ago and I had decent progress. If you are working on it, I can pause and work on sth else ( as you suggested)

However, the issue that I have with the names you selected, is that you selected the names at a specific time, and not necessarily the current time. for testing, it wont matter that much, but I would be a bit concerned about the in-sample or out-of-sample calibration. Do you have the DAG graph code in Python?

@Ali N., Feel free to proceed however you believe you would benefit most. My time is limited at this point...

Yes, this specific test was just a snapshot in time. The correlated pairs were pulled from 2010 (the whole year) and the applied forward and backward. But you're right, to use this properly one would want to recalc the DAG every so often. Thomas W. above found some potential DAG code in a python package. I don't know much about it (or if it's even available here). The C# DAG code is pretty straight forward, but I'm not up to the task of porting it.

Here is a converted version of the DAG research strategy. It generally replicates the original.

A few comments. Commissions are a major issue in this strategy. The Q's exorbitant default of 3cents per share, while conservative, is unrealistic in this strategy. $2.00 a trade, $4.00 a round trip is reasonable I'd wager. So that what this strategy is set at.

Additionally, I have a stinking suspicion that some of the returns in this strategy are due to a dividend or split bug (recently discussed elsewhere here).

Regardless, the new structure provides for other portion calculations to be plugged in and tested. A novel way one might treat these is to somehow monitor all portion calcs for their individual returns, and dynamically choose which version to choose, from time to time.

Thanks a lot. I believe the issue I raised about the minute mode still stands. I think it is more or less due to the minute mode RSI calculation.

@Thomas Wiecki - I think using NetworkX, though there's no "longest path" function (that I've found), you can multiply your weights by -1 and use the bellman_ford function.

I just ran this latest algo through minutely data (last 3 months) and it appeared to work without issue. Not well. But it did work. The RSI calc is still on daily data (as it should be).

@ Market Tech.

I see it is working but the result of the minute date and daily data look very different. I assumed that if we run the algo in the minute mode and use schedule function to rebalance, and make the trade 2 minutes before the closing, then the result of the minute mode and the daily mode should be very similar. But it is not the case..

Well, one thing is that one must remember that when you use history that the most recent period pulled using minutely data is TODAY's price. Which for this treatment would not be the price we'd want to calculate the RSI on. So one has to pull one more day extra then trim the the last period from it.

closeDeck = history(RSIPeriods + 2, "1d", "close_price").dropna(axis=1)[0 : RSIPeriods + 1]  

I'm still researching why the minutely vs the daily diverge so far. (Unfortunately I've seen this many times before and been unable to deduce the discrepancy).

Yeah. I dont understand how it works either. If the difference was slight, it would make sense, but here the differnce is crazy if you do your backtest from 2002 till now. I actually added the slippage mode for the daily backtest and the return is about 1600%, and in minute mode, the return is very small...

How did you determine the direction of an edge between two securities on the DAG? Or is it just an arbitrary direction that doesn't really matter?

Direction should not matter at all. Simply each edge should have both directions

For the article he edge between vertices was correlation. But any common pairs measurement would do. And as such there is no direction of weight or weight during group selection. Trading the group however any divergent measurement would work. And such measurement would have direction - as determined by your strategy. That is, leaders vs laggards. For a mean reverting strat, leaders are directed down and laggards up.

Also, remember this is a DAG so no cycles are allowed in constructing the group's longest path (critical path).

4 years ago the article by Dave Cline "Basket Trading Using A Directed Acyclic Graph" in APR 2015 STOCKS & COMMODITIES magazine bring me to Quantopian.

Despite the fact that "Directed Acyclic Graph DAG - research strategy" by Market Tech has nothing to do with the above article, I ported it in Q2 simplified and modified for my own (not Quantopian) strategy requirements:
Significantly outperform the market with reasonable beta (< 0.25), drowdown (< 0.25) and sharpe ratio (> 1.0) in full market cycle.

# Directed Acyclic Graph DAG simplified and modified  
# https://www.quantopian.com/posts/directed-acyclic-graph-dag-research-strategy#5c5e9e421f902a0046a56511  
import numpy  
import talib  
# --------------------------------------------------------------  
assets, RSIPeriods, lev, h = symbols('QQQ', 'TLT'), 14, 1.0, 0.2  
# --------------------------------------------------------------  
def initialize(context):  
    schedule_function(trade, date_rules.week_start(), time_rules.market_open(minutes = 65))  

def trade(context, data):  
    if get_open_orders(): return

    prices = data.history(assets, 'price', RSIPeriods + 1, '1d')  
    rsiValues = prices.apply(talib.RSI, timeperiod = RSIPeriods)  
    rsiValues = rsiValues.dropna()  
    if (len(rsiValues) == 0): return

    maxRsi = rsiValues.iloc[-1].max()  
    minRsi = rsiValues.iloc[-1].min()  
    avgRsi = rsiValues.iloc[-1].mean()

    if (numpy.isnan(maxRsi) or numpy.isnan(minRsi) or numpy.isnan(avgRsi)): return  
    raw_wt = 1.0 / len(assets)  
    trendAdjuster = .05 if avgRsi > 50 else -.05

    weight = {}; totalWeight = 0.0; wt = {};     

    for sec in assets:  
        stockRsi = rsiValues[sec].iloc[-1]  
        if (not data.can_trade(sec) or sec not in rsiValues):  
            wt[sec] = 0.0  
        elif (stockRsi > maxRsi - 5):  
            weight[sec] = raw_wt*(1.0 - h) + trendAdjuster  
            totalWeight += abs(raw_wt*(1.0 - h) + trendAdjuster)  
        elif(stockRsi < minRsi + 5):  
            weight[sec] = raw_wt*h + trendAdjuster  
            totalWeight += abs(raw_wt*h + trendAdjuster)  
        else: return 

    for sec in assets:  
        if data.can_trade(sec):  
            wt[sec] = weight[sec] / totalWeight if totalWeight > 0 else 0  
            order_target_percent(sec, wt[sec])  
            record(**{sec.symbol: wt[sec]}) 

    record(leverage = context.account.leverage)  
'''
1000000  
START  
06/01/2007  
END  
02/07/2019

assets, RSIPeriods, lev, h = symbols('QQQ', 'TLT'), 14, 1.0, 0.2  
schedule_function(trade, date_rules.week_start(), time_rules.market_open(minutes = 65)) 

Total Returns  
320.07%  
Benchmark Returns  
122.9%  
Alpha  
0.11  
Beta  
0.18  
Sharpe  
1.13  
Sortino  
1.66  
Volatility  
0.11  
Max Drawdown  
-14.45%

'''

Hey Vladimir,
Nice to see this strategy and concept revived.
Four years? Ouch! My, how time does scream by. The DAG was a crazy idea I dreamed up as I learned about the Facebook graph. I'd been deep into pairs and basket trading and wanted to see if they could be combined. The DAG was the answer. Now, I see your code trades only a pair of ETFs. The DAG expects one to trade a basket. But perhaps your code portrays the re-balancing of two of the basket components. Half of the DAG concept is actually creating the basket itself - that is, the network of "friends" of securities. Regardless, thanks for the flashback. (Oh, and Dave Cline and I may be the same person.)

Hi, Market Tech.

Glad to hear from you.
I agree: The DAG expects one to trade a basket of "friends".
But I find out that your strategy especially with my modification works better on pair of "enemies".