If I have some application/software running on an EC2 cluster can my algorithm talk to it through the internet?

Great question. I'm sure there's an easy way that a standalone can be built where it interacts with the Quantopian servers and does most of the computation without giving away Quantopian's historical minute by minute data.

Does quantopian have access to GPU. EC cluster has access to GPU computing. Do quantopian run Ec2? Do be able to write algo that can use GPU from python u need an open source tool called theano which would be a good addition to quantopian

Also if I am to implement machine learning outside quantopian in ec2 and interact with quantopian the algo would need the historical data from quantopian Plus other data not available in quantopian to train the machine learning algo

Hello Saravanan,

Regarding running Quantopian on your MAC, there is a standalone version called zipline (https://github.com/quantopian/zipline). However, you will not have access to the Quantopian data set; you'll need to provide your own data. Thomas Wiecki's talk may be of interest: https://www.quantopian.com/posts/zipline-in-the-cloud-optimizing-financial-trading-algorithms.

There is a functionality to load external data into Quantopian called fetcher (described on the help page).

I suggest having a look at the list of supported Python modules (see the help page), including sklearn. Perhaps you could try an algorithm without a GPU (which, to my knowledge, is not available on Quantopian)?

Grant

Thx Grant.
I am familiar with sklearn, but it only has standard models like SVM, etc.
I design/implement my own models which is what I would love to use.

I use a python tool library call theano
http://deeplearning.net/software/theano/ and am wondering if it can be added to the list of standard quantopian libraries.
it can use the GPU if one is available to run blazing fast, if not it can run pretty fast without the GPU.

Sarvi

Saravanan,

Just put in a request to Quantopian to have the theano library added (send an e-mail directly to [email protected]). No guarantee they'll be able to add it, but it's worth a try.

Grant

It's interesting, this is the 2nd request for Theano in a week. We hadn't looked at Theano before this week, but it looks interesting. I put it in for a security review.

The hardware-specific parts of Theano aren't going to be supported in the near term. GPU is obviously very interesting, but we have some other, more basic things to work on for the next few months. We haven't had a lot of feedback yet that computation is our bottleneck.

1. If you are running on EC2, just a headsup that they do support GPU virtualization based on Tesla GPUs
   
2. When people start apply machine leaning on large nets and large data it will very quickly become a bottleneck.
   But its impact is mostly seen interms of the time it takes to train these machine learning networks. These neural networks need to be trained on historical data before they can be applied become useful. This takes a lot of time/computation and eve after it may get stuck in a local minima or may not have found the right feature set and needs to be trained again. Its a trial and error procedure. When using Trading Solutions from Neurodimensions(a neural network trading application) a few years back with basic Back Propagation Neural network and some genetic algorithms, it used to train on basic PC without GPU support for 1 or 2 hours. The worst case being about 24 hours. Kinda gives you an idea where you might be headed if people start implementing neural network and/or genetic optimizations. So trust me, if you are successful, you are gonna get there soon.
   
3. Theano is not about just GPU. It can work without it too. But allows you to represent your machine learning mathematics as formula which in then compiles for the CPU or GPU depending on what is available and runs it.
   
4. For most machine learning kind of applications, unless I have access to some sort of CPU/GPU cluster, the training and backtesting which may involve many hours of trial and error may be best done on my local machine and then have the algorithm uploaded to yall. In that sense I like that yall have the backtester open sources. So it would be helpful to have access to the data on the local host for training these neural networks.

Sarvi