Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Loading a large number of fundamental data columns from a list or numpy array through the pipeline function

Is there a way to loop through a large list of variables in order to add columns to a pipeline processed dataframe?

In other words, I'd like to avoid adding 1540 financial statement items manually with lines like

Acctg_pipe.add(balance_sheet.accounts_payable_as_of.latest, 'accounts_payable_as_of')  
Acctg_pipe.add(balance_sheet.accounts_receivable_as_of.latest, 'accounts_receivable_as_of')  
Acctg_pipe.add(balance_sheet.accrued_interest_receivable_as_of.latest, 'accrued_interest_receivable_as_of')  

Is there some way to feed the following numpy arrays into a custom function to achieve the same outcome?

CFS_items = np.array(dir(morningstar.balance_sheet))  
IS_items = np.array(dir(morningstar.income_statement))  
BS_items = np.array(dir(morningstar.cash_flow_statement))  

I've tried writing custom functions with python's (*) splat operator; e.g.
function(*CFS_items). But, I get an error with everything I've tried so far.

Another thing I tried was to create lists of strings containing python code that I was planning to execute with eval() or exec(), but the system won't let me use these functions due to security concerns.

'''
creates list of strings containing python code  
'''
BS = morningstar.balance_sheet  
IS = morningstar.income_statement  
CFS = morningstar.cash_flow_statement

CFS_items = np.array(dir(CFS))  
IS_items = np.array(dir(IS))  
BS_items = np.array(dir(BS))  

# retrieve list of accounting items by financial statement  
BS_asof_idx = np.array([item.endswith('_as_of') for item in BS_items])  
IS_asof_idx = np.array([item.endswith('_as_of') for item in IS_items])  
CFS_asof_idx = np.array([item.endswith('_as_of') for item in CFS_items])

# retrieve financial statement items "as of" dates  
BS_asof_items = BS_items[BS_asofDates]  
IS_asof_items = IS_items[IS_asofDates]  
CFS_asof_items = CFS_items[CFS_asofDates]

BS_f_items = [] # financial statement items  
IS_f_items = []  
CFS_f_items = []

BS_asof_pipes = [] # pipelines to create dataframe columns of as_of dates  
IS_asof_pipes = []  
CFS_asof_pipes = []

BS_f_pipes = [] # pipelines to create dataframe columns of financial statement items  
IS_f_pipes = [] # corresponding to as_of date pipeline above  
CFS_f_pipes = []

for idx, item in enumerate(BS_asof_items):  
    BS_f_items.append(item.replace('_as_of',""))  
    BS_asof_pipes.append(''.join(['pipe.add(BS.', item,".latest, '", item,"')" ]))  
    BS_f_pipes.append(''.join(['pipe.add(BS.', BS_f_items[idx],".latest, '", BS_f_items[idx],"')" ]))  
for idx, item in enumerate(IS_asof_items):  
    IS_f_items.append(item.replace('_as_of',""))  
    IS_asof_pipes.append(''.join(['pipe.add(IS.', item,".latest, '", item,"')" ]))  
    IS_f_pipes.append(''.join(['pipe.add(IS.', IS_f_items[idx],".latest, '", IS_f_items[idx],"')" ]))  
for idx, item in enumerate(CFS_asof_items):  
    CFS_f_items.append(item.replace('_as_of',""))  
    CFS_asof_pipes.append(''.join(['pipe.add(CFS.', item,".latest, '", item,"')" ]))  
    CFS_f_pipes.append(''.join(['pipe.add(CFS.', CFS_f_items[idx],".latest, '", CFS_f_items[idx],"')" ]))  

Sample output of one of the lists created above:

["pipe.add(BS.accounts_payable_as_of.latest, 'accounts_payable_as_of')",
 "pipe.add(BS.accounts_receivable_as_of.latest, 'accounts_receivable_as_of')",  
 "pipe.add(BS.accrued_interest_receivable_as_of.latest, 'accrued_interest_receivable_as_of')",  
 "pipe.add(BS.accrued_investment_income_as_of.latest, 'accrued_investment_income_as_of')"]