HELP! Algo doing the same (simple) thing, done three different ways, produces different results.

This second backtest shows the results after the 'zero weights' are removed, as they were deemed unnecessary.

You can see the results vary from before. Why? There shouldn't have been any variance as the stock ETF weights were zero for everything but the 21d factor.

In this last backtest, I went back to my original algo but weighted the 21d factor to a very very large number, and left the other weights as zero. The results were the same as the original algo. How? The better metric is largely over-weighted while the worse metrics aren't given any weights at all.

The worst part of this is that the better performance is based upon metrics I don't know how to replicated because there is some sort of weird pass-by going on in my algo where something that shouldn't be occurring is.

I'm not only concerned that my algo has a weird bug, but that the above-average results wouldn't be replicate-able because of it.

Thanks in advance for your help! I really appreciate it.

This is an intriguing first for me, seems you're kind of simulating style=StopOrder() so that means it could eventually use frozen for selective orders with with order_optimal_portfolio and the stops portion of this you've created could eventually become contest-worthy, a ways down the road.

I didn't take time to look into the specific things you mentioned, I'm mainly here to help turn on some lights offering tools . About half of what looks like profit there is actually margin so one thing you can try for avoiding that is:

        order_target_value(stock, context.account.settled_cash / len(longs))  
        #order_target_percent(stock, 1.0 / len(longs))

There aren't a vast number of trades so you might be able to see them all with http://www.quantopian.com/posts/track-orders without hitting a logging window limit.
Adding that, besides fill prices, partial fills will be visible, and some minutes with no fill at all. There might be some replacement stocks with higher volume.

Here's a way to see what's in both pipelines, it can be used on the stock and bond pipes one after the other.
https://www.quantopian.com/posts/overview-of-pipeline-content-easy-to-add-to-your-backtest
To increase the use of that info, since it will pick up whatever columns are in the pipeline data, you can add for example some or all of the items that are being summed for score, like score_1yr, to columns. Abbreviate the longer ones for log window reasons, narrow, like '1yr': score_1yr, ... And also add er_126 etc etc. There are some nans to attend to, this is my latest preference, forward filling nans.

Dan Whitnable

It's those darn NaNs.

The following seems like it should return a factor with all zeros.

    weight_1yr = 0  
    score_1yr = weight_1yr * returns_1yr * er_1y

This is true EXCEPT in the case of NaN values. Wherever there is a NaN in either of the factors 'returns_1yr' or 'er_1y', the value will be NaN and NOT zero.

This is the same when all the factors are summed.

    score_1 = score_5d + score_21d + score_63d + score_126d + score_1yr  
    score_2 = score_21d

Wherever there is a NaN in the factors then the result will be NaN. Since there can be some NaN values in the factors there may be cases where a perfectly non-NaN value in 'score_21d' becomes a NaN. Therefore, the two factors, score_1 and score_2, may not always return the same results. There could be more NaNs in score_1.

So, take the second backtest above and add the following code

    no_nans = score_5d.notnan() & score_21d.notnan() & score_63d.notnan() & score_126d.notnan() & score_1yr.notnan()  
    score_rank = score.rank(ascending=False, mask = no_nans)

This filters out any assets that have NaN values. This is effectively what was happening when all the factors were summed because the '.rank' method ignores NaN (ie filters them). Using a filter and a mask just makes it more straightforward.

Attached is algo number 2 with this filter. The results are the same as algo 1 and algo 3.

Hope that helps.

Blue Seahawk, I really appreciate the kind words on the stop loss, but I'll have you look at line 2 of code. It probably looks good because of a little (...or a lot of) help from a certain friend! Admittedly there has been some input here and there, by myself and other, but the base code was born from your help! So thank you for that. And thanks for pointing me in the direction of some useful tools. I will definitely be looking into the 'settled cash' order bit in particular to be able to better discern what is profit and what isn't, but for the most part there can't be a ton of faux profit right? I mean margin only gets to 1.006 at most so if there is some fake profit it can't be much right?...or am I thinking about this incorrectly?

Dan, thanks as always for your help. Your input has been so valuable. Can you explain in a little bit more detail what the

    score = (score_21d)  
    # ONLY CHANGE IS ADDED THE FOLLOWING FILTER  
    no_nans = score_5d.notnan() & score_63d.notnan() & score_126d.notnan() & score_1yr.notnan()  
    score_rank = score.rank(ascending=False,mask=no_nans)

lines are doing as far as orders are concerned? Are we ordering based solely on the 21d score, as desired, just filtering out the equities that have any NaN values? If so, shouldn't I add the score_21d.notnan() to the list of no_nans or is there a reason it's excluded? Just trying to best understand!

Thanks in advance!

Hmm, I see, thanks. Actually, margin hits ~93% of initial cash, I happen to still have some info in an editor, this is before and after order_target_value I think. This'll help there. You're on it tho, the main big thing, the termites, I mean, nans.

2013-03-01 13:00 _ti:713 INFO  82 trading days     0 hr 1.4 min   2018-03-31 23:50 US/Pacific  
    Portfolio:  110589               2012-11-01 to 2013-03-01                  Alpha:  
 Initial Cash:  100000                      Buys:      5 (63 partial)           Beta:    0.78  
  Unused Cash:       0                     Sells:      4 (189 partial)        Sharpe:  
   Max Margin:  -93393               Commissions:      0                    Drawdown:     6.0  
     Max Risk:  203850 (204%)         Shares Now:   3536                   Stability:   0.890  
  Cash Profit: -100477                Shares Val: 111066                     Sortino:  
 Total Profit:   10589                  Cash Now:   -477                    Shrt/Lng Now:  0.00  
     Q Return:  10.59% Profit/Init      Max Lvrg:   1.85  2013-02-01          Max Shorts:  0  
   PvR Return:   5.19% Profit/Risk     PvR %/day:   0.06                      Max  Longs:  203850  
2013-03-01 13:00 ti:894 INFO .  
2013-03-01 13:00 _ti:713 INFO   Sort column 1                 Positions: 1 of 5 traded  
         Profit     Max  Return  Return   Buy    Price    Buy|Sell    Max     Shares  Shares  Partial  
 Symbol    PnL   Risked      %   %/day    Hold  Init|Now   Orders   Shrt|Lng    Now     %      Fills  
  AAXJ    5556   102144   5.44    0.26     0.0   58|60      1|1      0|1753       0     0        14  
   ILF    2790   108192   2.58    0.12    -0.0   44|43      1|1      0|2426       0     0        16  
   IEV    2305   100035    2.3     0.1     0.1   37|39      1|1      0|2675       0     0        22  
  USMV     487   110579   0.44    0.44     0.0   31|31      1|0      0|3536    3536     0        42  
   IVE     -68   110229  -0.06    -0.0     0.0   71|72      1|1      0|1551       0     0        10

2013-03-01 13:00 _ti:713 INFO  82 trading days     0 hr 1.6 min   2018-04-01 00:08 US/Pacific  
    Portfolio:  105157               2012-11-01 to 2013-03-01                  Alpha:  
 Initial Cash:  100000                      Buys:      5 (54 partial)           Beta:    0.34  
  Unused Cash:       0                     Sells:      4 (150 partial)        Sharpe:  
   Max Margin:    -601               Commissions:      0                    Drawdown:     6.0  
     Max Risk:  105757 (106%)         Shares Now:   3367                   Stability:   0.786  
  Cash Profit: -100601                Shares Val: 105757                     Sortino:  
 Total Profit:    5157                  Cash Now:   -601                    Shrt/Lng Now:  0.00  
     Q Return:   5.16% Profit/Init      Max Lvrg:   1.01  2013-03-01          Max Shorts:  -506  
   PvR Return:   4.88% Profit/Risk     PvR %/day:   0.06                      Max  Longs:  105757  
2013-03-01 13:00 ti:894 INFO .  
2013-03-01 13:00 _ti:713 INFO   Sort column 1                 Positions: 1 of 5 traded  
         Profit     Max  Return  Return   Buy    Price    Buy|Sell    Max     Shares  Shares  Partial  
 Symbol    PnL   Risked      %   %/day    Hold  Init|Now   Orders   Shrt|Lng    Now     %      Fills  
   ILF    2654   102701   2.58    0.12    -0.0   44|43      1|1      0|2303       0     0        16  
   IEV    2305   100035    2.3     0.1     0.1   37|39      1|1      0|2675       0     0        22  
  USMV     460   105298   0.44    0.44     0.0   31|31      1|0      0|3367    3367     0        42  
   IVE      -1      497  -0.11   -0.01     0.0   71|72      1|1       -7|0        0     0         0  
  AAXJ      -3       58  -5.34   -0.25     0.0   58|60      1|1       -1|0        0     0         0

What exactly does that mean then, that margin hits 93% of my initial cash? This type of algorithm will be traded manually, not on an automated platform, so as long as the backtest results are somewhat indicative of what I could expect in real life that's fine with me. I don't plan on using any margin or leverage for this strategy in real life, at least not any time soon.

Also, 204% max risk? What's that all about?

...edit...

I think I get the basics of what you're conveying blue, I really appreciate the insight there! But if the max risk and max 'margin' is only instantaneously so (I see it goes back down to reasonable levels just two minutes later), does it have large long-term implications? This algo trades monthly, so if these metrics simmer down before another trade is made, what're the implications?

I'm NOT trying to downplay what you've been so helpful to point out, I'm just trying to make sure I'm seeing straight.

204% max risk means it started with 100k but at some point had invested 204k. It had made some money by then, so that's why margin is only 93k rather than 104k. It only ever invested in 5 securities. Notice the Max Risked column for each of them. IVE & AAXJ apparently happened to be the ones prevented from investing so much (into margin) from the target_value and settled_cash line.

So then is instantaneous or is it a lasting margin? It looks to me like it only lasts .2 minutes before that max risk goes back down. Could that just be something happened in reverse order? I.e. it finished buying something before selling something else?