Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
How to separate data returned by get_pricing by day?
complete_data = get_pricing(sec_symbol, start_date='2012-01-03', end_date='2015-01-03', symbol_reference_date=None, frequency='minute', fields = {'price', 'volume'}, handle_missing='raise')  
print complete_data  

displays the following:

2012-01-03 14:31:00+00:00 15254.0 74.7400
2012-01-03 14:32:00+00:00 140803.0 74.7500
2012-01-03 14:33:00+00:00 21474.0 74.8000
2012-01-03 14:34:00+00:00 20866.0 74.8000
...

Yet when I execute

complete_data = complete_data.as_matrix()  

All datetime data is lost. I want to convert complete_data to a 3 dimensional array where the first axis represents discrete trading days. Is there a simple way to achieve this?

2 responses

The returned 'complete_data' object is a pandas dataframe. It has 2 columns labeled 'price' and 'volume'. It is indexed by the datetime (date and time of day). If you want to change the index into a column use the 'reset_index' method (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html ):

date_and_data = complete_data.reset_index()

This creates a new dataframe with 3 columns labeled 'index', 'price', and 'volume'. You may want to rename the 'index' column to something more meaningful like 'date'. This can be done with the 'rename' method (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rename.html)

date_and_data.rename(columns={'index': 'date'}, inplace=True)

Now the columns are labeled 'date', 'price', and 'volume'.

It sounds like you want to access the days and the time of day separately? I would keep everything in a pandas dataframe and NOT turn it into an array as you were trying to do with the 'as_matrix' method. That's just personal preference but dataframes come with a lot of powerful easy to use methods which one looses by turning the data into an array.

I would create new columns with just the date and just the time. This can be done by accessing the date and time properties of the datetime object (which was the index). See http://pandas.pydata.org/pandas-docs/stable/api.html#datetimelike-properties.

date_and_data['just_date'] = date_and_data['date'].dt.date  
date_and_data['just_time'] = date_and_data['date'].dt.time

Now the date_and_time dataframe has 5 columns labeled 'date', 'price', 'volume', 'just_date', and 'just_time'.

You can sort and slice and dice and compare times across different dates in this format. You may also want to use 'pivot' to easily summarize by date (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pivot.html ). See attached notebook.

Hope that helps.

Thanks Dan!