Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Custom Dataset Upload Error
Michael Matthews
posted May 28, 2020
Self Serve Data

When uploading a historical dataset to Quantopian, for the column containing the data values (type = number), I received an error "invalid digit in double: 756.0\r (line 102)". When I created the csv file, the dataset was written from a pandas dataframe (where the data column dtype was a float) to a csv file. I can't seem to figure out where this extra carriage return character is coming from.

I know this is probably not a problem with Quantopian's tech, but I wasn't sure where else to ask this question, or how to fix it.

Any help is appreciated.

Here is a screenshot of the error: https://ibb.co/syQWVX1

2 responses
Chris Myles
May 28, 2020

Hi Michael,

That is an "interesting" error, especially at the line number that is occurred at. Is there a chance you could send a 150 row snippet of the csv file you are trying to upload to [email protected] ?

Are you on a Mac or PC? Which version of python and pandas are you using?

I suggest trying the following to_csv parameters to see if that helps:

df.to_csv(outfile,index=False,line_terminator='\n',encoding='utf-8')
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Michael Matthews
May 29, 2020

Hi Chris,

I was able to get the data uploaded. I'm still not sure what caused the problem, but here is the info you requested along with the troubleshooting steps I performed. (I also sent an email over with a sample of the first 150 rows of the data (I tried uploading this sample and received the same error).

Here is the info you requested:

  1. PC - Windows 10
  2. Python version 3.7.4
  3. Pandas Verion 0.25.1

Here is some troubleshooting that I have done:

  1. I tried setting the line_terminator argument to '\n but that caused a different error when uploading (although I don't recall the error. I can do it again if you want that error message).
  2. I used a different python and pandas verion (python 3.8.3 with pandas 1.0.3). Still got the error.
  3. I converted the data value column to a string/object dtype. Then, ran .str.strip(r'\r') on the series representing that column, and then wrote it to csv. Still got the error.

None of these worked. I then removed any lines in the dataframe that corresponded to weekends (which removed the problem line), wrote it to csv, and then uploaded it successfully. (I probably should have tried this first).

Another thing that seemed to work was when I simply changed the column header of the data column to a different name. This actually allowed me to upload the original file without the problem line being removed.

Anyways, thanks for your help. I was able to get it uploaded. If you need any more information from me to help you troubleshoot anything on your end, please do let me know.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.