Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Pipeline - Import Datasets

Can you give an example of how to import a custom data set rather than relying on USEquityPricing? This would be very helpful.

6 responses

Hi Anthony,
Right now the pipeline API only works with the USEquityPricing and Morningstar Fundamentals datasets. We are actively working on integrating it with our other partner data.

As part of our planning, it would be helpful to better understand what you are trying to accomplish. Care to share some details?

KR

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

This is something we're working on, right now. Partner data sets like those listed on quantopian.com/data (i.e. under the Data menu in the navigation) will be accessible to you. It's not yet out and available . . . but coming soon. We need to do some data reprocessing and hammer out some bugs to ensure a smooth experience.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Josh - jinx :-)

I am working with my partner on Zipline although I am concentrating on porting our algos into Quantopian at the moment. My partner is working on things we know we need like the ability to trade assets in different currencies and to use different base currencies for reporting / P&L. There are various other things we will need to improve / help you improve / encourage improvement of...such as the way zipline handles de-listed stocks. Probably using a dictionary to mark stocks as "A" for active and "I" for inactive.

We are contemplating using Zipline and / or Quantopian to provide our algos to our clients.
Hence my (unanswered) post:
https://www.quantopian.com/posts/https-slash-slash-www-dot-quantopian-dot-com-slash-algorithm-builder
And our (also unanswered) emails to Fawce and Karen of 27th August 2015.

Pipeline is useful to us.
While a universe of US stocks if fine and dandy, we also need UK and mainland European stocks.

But in the interim I was merely wondering whether Pipeline can be used with some of the other features like fetcher or indeed with a hard copy list of, say, 60 US traded ETFs. The latter being essential to a tactical asset allocation strategy.

Of course I am aware that this can be done with, say, "history" but Pipeline is a lot more convenient.

Please excuse the long ramble but we are currently feeling our way around Quantopian and Zipline and getting a handle on how it all works and whether we can use it.

Hi Anthony,
Thanks for your continued contributions to the Quantopian community. We value the broad variety of perspectives and opinions in the forums and it helps us to build a better product and a better company overall.

There are others, like yourself, working to run their own businesses through Quantopian. We think this is a great idea and encourage it. The pain points that you hit as an investor will help corroborate and feed into our prioritization decisions.

That said, our business strategy will contribute it’s own set of features and needs. Sometimes our business needs will overlap nicely. Sometimes they will not. We do our best to build a product that is maximally useful, but limited time and resources means that we have to make hard prioritization decisions about what we build.

You’ve mentioned a number of things that you would like the system to do, which is great, and your work to add what you need to Zipline is appropriate. Some of your requests are on our list and others are not.

  • Handling de-listed securities is on the short list. Pipeline has made this problem even more apparent and we will hopefully start work soon to manage this.
  • Currencies and foreign markets are on the long list. I don’t know when work will happen on these, but it’s definitely an area where we want to expand in the future.
  • Including fetcher in pipeline is something we’ve discussed, but I don’t have it prioritized at this point. Regarding your use case of importing a hard coded list of 60 ETFs, you could use fetcher to import the list of ETFs, and then gather the data for those ETFs from the pipeline output after it has been calculated across the universe. Here is an example that does something similar with a static list of ETFs. Would this work for what you are trying to do?
  • As for the algo-builder idea, it is certainly interesting. Someday there might be the opportunity for us to bring algo investing to the retail market, so investors can select algo writers and customize, but it’s not a priority of our current business.

As always, thanks for your thoughts.

Thanks for the information and suggestions Karen. I take note of all you say. One point on the algo builder - in fact our client base is institutional not retail. IE authorised fund managers here in the UK who want to offer their clients a quantitative investment approach.

Currently we are providing clients with our algos written in a 3rd party proprietary piece of software called Trading Blox. They take TB + our algos which they lease from us at a certain number of BPS on AUM. It is heavily time intensive to educate the client and it would certainly not be worth our while with retail.

Eventually Zipline will provide a far more attractive option for us. The real problem for our clients is back testing and that is where they require a great deal of time intensive hand holding. A simple on-line facility with a pre-built GUI would make this far simpler for the client.

We do not provide investment advice merely the algos with which the client has to come to his own decision on portfolio and parameters and systems given the framework we provide.

Anyway, Zipline and or Quantopian may well shape us as a good platform for our clients at some stage.

Thanks again for the help
A