would b awesome. The primary exchange id works for the nas 100 but not for snp

Hi guys,

Unfortunately, historical index constituent lists for indices like the S&P500 are licensed from the originating data vendor (in this case S&P) and we don't have an agreement in place to offer this data via the backtester. It's been a while since I looked into the costs for historical constituents, I think it would be prohibitive for an individual to license them at the going institutional rate, but if we could find a way to get a better scale deal it might be doable.

If you aren't required by investors to use an index like S&P 500 directly you can always create a proxy for large cap US listed stocks using the Morningstar fundamentals data. Granted that's not perfect and probably also requires a bit of handling edge cases and so on.

Best wishes, Jess

I've read that many times about the licensing and prohibitive costs, but couldn't one fairly easily create the historical constituents from scraping the wiki page like: https://en.wikipedia.org/wiki/List_of_S%26P_500_companies

This gives you (of course you have to trust the source) the current, and all changes back to 2000.
Most of the changes seem to come from this source: http://sandp500changes.whw1.com

(I wonder if this question originated after reading Andreas' latest book out this past weekend, where basically you need this historical constituents list to do the ranking, etc.)

So, what am I missing, that it wouldn't be easy enough to scrape the site with a script and create time indexed constituents listings? At least, that was my plan.

@Ken, seems doable, if you need help, please holler. I'm not a king developer but I have time

Hi Guys,

I wrote a parser for the historical constituents of the S&P 100 list on Wikipedia. I haven't tested it in a while so I'm not sure if it still works.

The first thing you need to do is export the XML from the Special Exports page. Then parse the XML with the code below to pull out the historical ticker list.

import xml.etree.ElementTree as ET  
import wikimarkup  
import pandas as pd  
from pyquery import PyQuery  
from datetime import timedelta

def get_tables(wiki):  
    html = PyQuery(wikimarkup.parse(wiki))  
    frames = []  
    for table in html('table'):  
        data = [[x.text.strip() for x in row]  
                for row in table.getchildren()]  
        df = pd.DataFrame(data[1:], columns=data[0])  
        frames.append(df)  
    return frames

tree = ET.parse('/path/to/Wikipedia-20141121081732.xml')  
root = tree.getroot()

sp100_hist = dict()  
components = list()  
for child in root[1]:  
    for table in child.findall('{http://www.mediawiki.org/xml/export-0.10/}text'):  
        components = get_tables(table.text)  
    if not components:  
        continue  
    for ts in child.findall('{http://www.mediawiki.org/xml/export-0.10/}timestamp'):  
        try:  
            sp100_hist[ts.text] = components[0]['Symbol']  
        except KeyError:  
            continue

symb = pd.DataFrame.from_dict(sp100_hist)

# find unique columns of symbols and drop the rest  
symb_unique = symb.copy()  
for i, col in enumerate(symb):  
    if i > 0:  
        if np.array_equal(symb[col], symb[prev_ts]):  
            symb_unique.drop(col, inplace=True, axis=1)  
    prev_ts = col  

i used kimono labs to get the current list: https://www.kimonolabs.com/api/csv/b0rescwg?apikey=py1Kp8bdN5tjk48mI1vE4WxyFcx9gYBH

@Peter it would be great if you could invest in it a little of your time!!!!

Concerning the sources I have found also
http://faq.library.upenn.edu/recordDetail?id=36133&library=manchester_business&institution=manchester

A CSV file with the list in the format required by the fetcher could be made available to everybody interested, e.g. in Dropbox.
Then we could use the fetcher (fetch_csv) to have a constantly updated S&P500 Universe in our algos.

In the CSV file the full list of S&P stocks should be given only for the days in which there was a change in the index constituents.
If there are no stocks for a date in the file, the stocks from the previous day are forward-filled in the Universe.
The CSV file could be backward filled at least until 2007 according to the WIKI page.

It would be great then to have a software running somewhere once per day that checks if there are changes to the index constituents and if so it automatically adds at the end of the CSV file the new 500 constituents with the current date.

@Nicola, yes WRDS is the database folks commonly point to to answer this question, but this is probably more of what Jessica was referring to perhaps. You have to be in an organization that has access to this, etc.
Hence my suggestion to use a public site like wikipedia to start with. True, it may not stay up to date, but not a bad start.

see my answer here: https://www.quantopian.com/posts/need-list-of-s-and-p-500-stocks, based on the holding of IVV