Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
New Feature: Improved Custom Dataset Dashboard
Jamie McCorriston
edited Jul 30, 2019
Q Updates Self Serve Data

Hey Everyone,

Today, we added a new Custom Dataset Dashboard to help you create and monitor the datasets that you upload to Quantopian using Self-Serve Data. Since the Self-Serve Data introduction just over a year ago, thousands of custom datasets have been added to the platform, several of which have been used in contest algorithms and fund algorithms.

With increased usage came a need to improve the workflow around uploading and managing custom datasets. The new Custom Dataset Dashboard was added to make it easier to work with custom datasets on Quantopian. If you are not yet familiar with Self-Serve Data and you would like to upload a custom dataset to Quantopian, check out the documentation here.

New Functionality

  • Search: Filter datasets by searching all the column data for entered search text.
  • Column sort order: By default the datasets are sorted by Created At in descending order. You can select any column including name to reorder the datasets table.

New Columns

Type

  • historical: Single historical load only, no live updates configured.
  • live: Dataset is configured for live updates.
  • archived: Dataset with disabled live updates, may contain point-in-time data. Note that archiving is not yet available, but should be available soon (see bottom of this post).
  • error: The initial dataset failed during the historical load, import functionality is available but the data typically contains 0 rows.

Status

Hover over the status value to see the last load timestamp and error message if status = failed.

  • pending: a brand new dataset that has been added to the queue, but hasn't started running yet
  • running: an in-process load
  • completed: a successful load, may result in zero updated rows during each live update check
  • failed: the load did not succeed. Use hover to see error message
  • empty: No data records were found during live update

Created At

The timestamp when the dataset first was created in the local timezone and locale date format.
Note: this will be blank for error type datasets.

Coming soon: Delete and Archive (disable live updates) functionality plus an additional column to help monitor the Last Updated Date timestamp for live and archived datasets.

Note: for a short transition period, the old Self-Serve Data page will continue to function. In the near future, we will redirect traffic to the new dashboard.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

6 responses
Samantha Klonaris
Aug 19, 2019

Update: The Delete and Archive functionality are now available at https://www.quantopian.com/custom-datasets, the icons will appear in the far right column when the dataset is in focus.

Archive is only available for Live dataset types. This will permanently stop the nightly live updates for the dataset.
Delete is currently available only for Error dataset types. This will permanently delete the dataset.

We are very interested learning more about your custom dataset use case and self-serve experiences, please fill out this Self-Serve Dataset Survey or contact us at [email protected].

Happy Uploading

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Viridian Hawk
Aug 28, 2019

Great.

Please excuse my basic questions -- I still haven't properly learned Python after all this time. How do I sort the output so that so that I can see the most recent by timestamp?

from quantopian.interactive.data import user_59065fbb5afa7d0010371ea1  
load_metrics = user_59065fbb5afa7d0010371ea1.load_metrics  
load_metrics[load_metrics.dataset == 'my_dataset']
Chris Myles
Sep 3, 2019

Viridian,

The Self-Serve checking upload status docs give a good example of how to leverage odo (in a research notebook) to convert from an interactive dataset to a pandas dataframe (including Ascending=False sort).

In the new dashboard, you can leverage the up/down arrows next to the Created At to sort If you are looking for the last timestamp for historical datasets. For live datasets we'll be adding a Last Updated column, which will include the timestamp of the last data change (vs last live upload attempt timestamp).

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Viridian Hawk
Sep 3, 2019

Thanks. I was just trying to debug why my data was stale. Turns out my data source updates at 7:00am, but Quantopian pulls it at 3:00am-6:00am. I guess not much I can do about that?

Albert R.
Sep 3, 2019

Are algos using self-served data eligible for the contest?

Chris Myles
Sep 3, 2019

Viridian, for now those are the current self-serve update windows that give us enough time to process all the self-serve data for pipeline before market open. In the future, we are looking to extend that window closer to market open and will be adding the ability to push data to self-serve (vs polling).

Albert, yes live updating self-serve datasets are eligible for the contest, https://www.quantopian.com/posts/analyzing-a-signal-and-creating-a-contest-algorithm-with-self-serve-data provides a good template algo with sample self-serve data to get you started.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.