Hi Ed,

You can make up to 5 simultaneous Fetcher calls per algorithm. If you try to use more, you'll run into the error you're seeing. Can you combine the information into 5 or less CSVs? Fetcher can accept large files for import.

Cheers,
Alisa

Hi Alisa,

Unfortunately I can't. I'm getting them from yahoo finance. Does it mean that I'm out of luck here and have to run my algorithm locally using Zipline? BTW, where this strange limitation came from?

Regards,
Ed

Can you get the CSVs from Yahoo and then combine them locally so you can upload them back to a file sharing service for fetcher to download?

I don't think it's that strange. Maybe it's because the service is free and Quantopian has to somehow limit their resource use? Not sure though, but it seems consistent with other services that limit the amount of resources you can ultimately use.

Hi Akram,

Sure I can do some manual work to combine multiple cvs into one. However, that will bring manual step into the workflow and make the code more complex as I'd have to split the data returned by fetcher.

I called this limitation strange because the amount of data is not limited. Fetcher can load all that data if I combine a lot of cvs into one. So, what's restricted here? Amount of TCP connections from Quantopian to Yahoo triggered by one algorithm? I don't think so as multiple HTTP requests can share the same TCP connection. Sorry for being too technical here, but I still don't understand the reason of this limitation.

Hi Ed,

Ah yes, definitely having a manual step in an otherwise automatic workflow is not ideal. I thought you meant that since you were using Yahoo Finance you somehow were having an issue there, but yeah, I agree with you.

You are right about the amount of requests over a single TCP connection, so maybe the limitation is just there to avoid a malformed algorithm to run crazy, say by having a fetcher inside a big loop or something?

I'm just taking a wild guess here since we have no official reason for this, but now that I think more of it it does look like a strange limitation.

This limitation exists from a security standpoint on the platform - to restrict the number of calls an algorithm makes to outside servers and to maintain the stability of our own servers.

I explored other possibilities for reading files, other than fetcher. Fetcher was not appropriate for my intended application. I wrote the code offline twice using 2 other methods. It turns out that they insist that you use fetcher. One of my failed attempts used a library which I later found was not on the approved list. My second attempt utilized the approved panda module. It turns out that even though panda is approved, I got a message saying that the read_csv function is apparently "black listed". read_csv would have been a very elegant solution for me. I suppose that I can understand the limitations from a security point of view. After all, we'll be trading real money.

Hi Alisa,

Thanks for your explanation.
I don't understand what you mean by decreasing number of calls(see my comment about multiple http requests sharing the same tcp connection). However, I've got your idea: I'm out of luck with this and you're not going to increase that number. So, basically for algorithms with number of securities more than 5 fetcher is useless.

Regards,
Ed

Ed, what is it that you're downloading from yahoo? Maybe there is another solution or a workaround that will work in your case.

It's dividend adjusted prices. Unfortunately Quantopian processes dividends in such a way that it's hard to calculate realistic algorithm performance for dividend paying securities.