Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Pipeline with Datasets that Contain Weekend Data
Michael Matthews
posted May 1, 2018
Feature Request Pipeline

I mentioned this issue at QuantCon this past weekend. I just wanted to document the issue on the forum. Basically, Pipeline is not able to fetch data on weekends for datasets that have observations on weekends. The Stocktwits dataset from psychsignal is an example. see the attached notebook for details and an example.

I'm not really sure of the best way to deal with this issue, since I don't know enough about how Pipeline works on the back-end, but I figured I'd post it in case others had ideas.

9 responses
Luca
May 16, 2018

This is very interesting, thanks.

Doug Baldwin
May 16, 2018

Curious, if you wrote a CustomFactor with only stocktwits inputs and a window_length = 3, would its *inputs array of data also skip non-market days. I believe the answer here too would be it skips, but if not you could compare the current date in the CustomFactor with the market calendar to decide which of the three items in each array to output.

Joakim Arvidsson (Cream Mongoose)
May 16, 2018

Hmm, that’s a real bummer. Do you know if the Q engineering team is aware of this? I wonder if there can be some sort of creative data manipulation or cleaning on the backend to allow this data to be accessible via Pipeline? I would think weekend data would be quite important when it comes to sentiment data... :(

Thanks for highlighting it and sharing the notebook.

Michael Matthews
May 17, 2018

@Doug, That's an interesting idea, but it doesn't look like it works. The numpy array in the custom factor still skips the Friday and Saturday values.

Doug Baldwin
May 17, 2018

I believe Quantopian stages pipeline data for performance efficiency, and would guess trading days to be a strict architectural limitation. You might suggest Quantopian consider creating a new data series containing the Mon-Fri data only, skipping weekend data. Then you could subscribe to both series to access most all the data, some of which would be duplicated across both series.

Josh Payne
May 17, 2018

Thanks for noting this issue. We'll add it to our backlog. Providing different options for the treatment of weekends in lookback windows for pipeline and factor creation is a reasonable improvement request. Some data sets like sentiment could definitely use this feature (where Sunday's sentiment definitely influences Monday) whereas other data sets (pricing data) should only consider market calendar days.

Thanks
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Joakim Arvidsson (Cream Mongoose)
Apr 6, 2019

Hi @Q Team,

Could someone provide an update on this issue please? I just ran Michael's notebook again, and the issue doesn't appear to have been fixed (see attached). For sentiment data, wouldn't weekend data possibly be even more important? As it is, this 'Featured' data-set is only about 70% complete, right? Or is the Friday and Saturday sentiment data included in the Sunday and Monday sets? Thanks.

Ben Hawkins
Nov 8, 2019

Any news on this issue? Best option for sentiment-based strategies is not trading on Mondays

Viridian Hawk
Nov 8, 2019

I'm also eager to hear. My observation with sentiment strategies is that they experience significant alpha degradation on Mondays. I'd be curious to know whether that is due to this "bug".

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.