Take a look at the attached notebook. I believe its doing what you want?
The major problem was the 'window_length' in the 'RSIRank' custom factor. This should be 1 and not 252. While the RSI should have a length of 252 (or whatever you wish) the ranking should only be done with the latest RSIs. You don't want 252 RSIs for each security, you just want the latest one.
The second problem was the use of 'len(r)' to get the number of securities. This actually returns the number of rows. The array is set up with rows being days and columns being securities. 'len(r)' would always return the days or window length which is 1. Maybe use 'np.size(r)' instead.
One final issue is that your custom factor includes NaNs in the ranking. This probably isn't what you want since the top 20% will be predominantly securities with NaN for an RSI. I'd suggest simply using the built in 'percentile_between' method instead of writing a custom factor. This excludes NaNs.