Hi, is the tool going to be open sourced like the excellent zipline project?

We are building it in IPython, which is already open sourced, and we plan to add back to the IPython project when it's appropriate.

Hello Karen,

Some questions:

• Will the entire Quantopian database be available to the research environment? Or will there be restrictions on the number of symbols that can be analyzed at a given time?
  
• Will the code execute on the user's local machine, or on a server?
  
• If on a server, what is the basic outline of the computing environment? Any parallel processing capability?
  
• Other than doing a 'print screen' will there be any option to save results locally? Remotely?
  
• Will users be able to import their own libraries?
  
• Will it be possible to write out a file (e.g. .csv) that could be read by an algorithm via fetcher?
  
• Will there be a scheduler, so that code can be executed regularly (e.g. nightly)?

Grant

Hi Grant,

With the caveat that we're still in development and everything is subject to change, some tentative answers to your questions:

Will the entire Quantopian database be available to the research environment? Or will there be restrictions on the number of symbols that can be analyzed at a given time?

The only limit here is likely to be what your allocated machine can fit in memory. The data access APIs are built on top of the Pandas object hierarchy, which is very efficient in terms of storage capacity. That said, loading the entire database of minute data into memory at once is probably not yet realistic (though projects like bcolz are making it look like more of a possibility by providing the illusion of holding a large set in memory while actually paging to disk.

Will the code execute on the user's local machine, or on a server?

Python code will still execute on our servers.

If on a server, what is the basic outline of the computing environment? Any parallel processing capability?

The research environment is built on top of IPython Notebook; each user will get access to their own remotely-hosted notebook server. You can have multiple notebooks running simulataneously, and they'll all draw from the same pool of resources. We haven't yet done much investigation into extending the IPython Parallel toolset to use our APIs, but its definitely an intriguing option that could open up a lot of options for computationally-intensive workloads like parameter optimization.

Other than doing a 'print screen' will there be any option to save results locally? Remotely?/Will it be possible to write out a file (e.g. .csv) that could be read by an algorithm via fetcher?

It's unlikely that we'll be able to allow you to save results locally, the main reason being that we can't allow users to wholesale export data from our database. Allowing users to save results on their servers and/or make research results available via Fetcher is a request we've gotten from a few alpha testers, so it's definitely something we're thinking about.

Will users be able to import their own libraries?

This is another question/request we've gotten from a number of alpha users. It's a tricky problem because we need to verify that any imported library meets our security constraints. I think allowing some sort of limited user-written module system will eventually be in the cards (as an engineer such a system would certainly be high on my personal wish-list); but for security reasons it's unlikely that we'll allow you to pip install any package you want.

Will there be a scheduler, so that code can be executed regularly (e.g. nightly)?

This is something we (or at least I) hadn't considered much. I'd be curious to hear what sorts of tasks you'd want to schedule regularly.

Thanks,
-Scott

Thanks Scott,

Regarding accessing a large number of securities, I was thinking along the lines of searching/screening. An example is given on http://blog.quanttrader.org/tag/statistical-arbitrage/ of examining all pairings of the S&P 500 constituents. So, if the data for each pair could be pulled into memory sequentially, analyzed, and then flushed, the next pair could be analyzed without consuming additional memory.

Grant

Scott,

Regarding a scheduler, it seems like a natural fit, since data comes in chunks (daily/weekly/quarterly). If it takes awhile to analyze, it'd be nice to have the analysis done ahead of time, rather than clicking a button and waiting minutes/hours/days(?). Also, I figure that with a scheduler, the research environment could be configured to provide results to a live algorithm, to be uploaded via fetcher (e.g. nightly). Presumably, it'll be possible to output a file from the research environment that will be available to fetcher? Or no?

Grant

Thanks for the feedback Grant! Those are interesting thoughts.

One of the most exciting things about this project for me is that it's stepping into totally uncharted waters. Perhaps even moreso than with the backtester and IDE, the space of new things that we want to do is (at least in the near term) always going to vastly outstrip the resources we can dedicate to the project. That makes it even more important than ever for us to continue to get good feedback from our users so we can choose to allocate our time to the features that will be most valuable to you all.

Hi Scott,

Any update on the user written modules?

I really really want to organize my common code into a separate file and import it, that way if I find a way to improve some functions, i.e. make them faster or fix bugs, I dont have to go copy past to all of my algos.

Best,