Here is a template for working sector by sector with a quasi S&P 500 pipeline. It starts by filtering the pipeline by top 500 by market cap, then filters out any stocks in the top 500 with low price and low liquidity. Usually returns ~490 stocks. Then in before_trading_start it creates separate data frames by sector.