Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
working with starcluster / zipline / Ipython
Pumplerod
edited

I'm not really sure which community to turn to for help, but as I started down this road from information on this forum, this is where I'll start.

I've been trying to follow Thomas Wiecki's presentation on using starcluster, zipline, and Ipython in the cloud for parameter optimization. However, when I try and instantiate the Client() object the process hangs.

Keep in mind, I'm no expert with any of this. And I apologize if this is not exactly the correct form to post in, but as I ran into this trying to optimize for quantopian, I imagine others might as well.

I was able to follow the presentation up until attempting to use the EC2 ami Thomas suggest in his talk. Figuring that I was unable to find it because I was in the wrong region or something I decided to configure my own. So I now have a new ami (ami-3c1b830c) in the us-west-2b availability zone (which is public if it should turn out to be useful to anyone).

I installed/upgraded to: Ipython (1.1.0), pyzmq (14.0.0), zipline (0.5.10)

getting the cluster up and running seems to work without issue.

>>> IPCluster has been started on SecurityGroup:@sc-zipcluster for user 'clusteruser'  
with 2 engines on 2 nodes.

To connect to cluster from your local machine use:

from IPython.parallel import Client  
client = Client('/Users/clusteruser/.starcluster/ipcluster/SecurityGroup:@sc-zipcluster-us-west-2a.json', sshkey='/Users/clusteruser/.ssh/starclusterKey.rsa')

When I look at the generated json file for configuring the ipcluster it looks like this...

{
  "control": 45326,  
  "task": 58177,  
  "notification": 35696,  
  "task_scheme": "leastload",  
  "mux": 44860,  
  "iopub": 35537,  
  "ssh": "[email protected]",  
  "key": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  
  "registration": 34502,  
  "interface": "tcp://xxx.xx.xx.xx",  
  "signature_scheme": "hmac-sha256",  
  "pack": "json",  
  "unpack": "json",  
  "location": "xxx.xx.xx.xx"  
}

HERE's the problem:

When I try and connect to the ipcluster as indicated in various on-line examples the session hangs, and never returns, when trying to instantiate the Client().

from IPython.parallel import Client  
client = Client('/Users/clusteruser/.starcluster/ipcluster/SecurityGroup:@sc-zipcluster-us-west-2a.json', sshkey='/Users/clusteruser/.ssh/starclusterKey.rsa')

When I just use IPython.parallel locally, simply calling Client() with no parameters it works without issue.

I'm not really sure where to begin to troubleshoot. Any advise from people that have gone down this road would really be of great help.

4 responses
Pumplerod

As an update, today when trying to test this process again python crashed, apparently due to some problem with libzmq

Process:         Python [22134]  
Path:            /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python  
Identifier:      Python  
Version:         2.7.5 (2.7.5)  
Code Type:       X86-64 (Native)  
Parent Process:  bash [11042]  
Responsible:     Terminal [505]  
User ID:         501

PlugIn Path:       /Library/Python/2.7/site-packages/zmq/libzmq.so  
PlugIn Identifier: libzmq.so  
PlugIn Version:    ??? (0)

Date/Time:       2013-11-05 10:42:45.910 -0800  
OS Version:      Mac OS X 10.9 (13A603)  
Report Version:  11  
Anonymous UUID:  2729F925-8BE8-7483-7DFB-61ECD670899E

Sleep/Wake UUID: 07F86273-FAD3-468D-B421-AC8BBCE22AFA

Crashed Thread:  2

Exception Type:  EXC_CRASH (SIGABRT)  
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:  
abort() called

Thread 0:: Dispatch queue: com.apple.main-thread  
0   libsystem_kernel.dylib          0x00007fff8cc4794a poll + 10  
1   libzmq.so                       0x000000010fcd0234 zmq_poll + 494  
2   _poll.so                        0x000000010fd8910d __pyx_pw_3zmq_7backend_6cython_5_poll_1zmq_poll + 3010  
3   org.python.python               0x000000010f42ab35 PyEval_EvalFrameEx + 14712  
...
Jessica Stauth

Hi Pumplerod,

You are welcome to post this to the forums, but I wanted to point out that there's also a zipline mailing list where we normally discuss zipline specific issues:
https://groups.google.com/forum/#!forum/zipline

Just in case you might find more success with that route.

Regards,
Jess

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Pumplerod

Thanks Jessica, I posted here because I didn't know where the problem was coming from, however, I was following a recipe I found through here and thought others may have had the same issue. It's good to know about the zipline specific group though.

Jessica Stauth

Sure thing - @Thomas will see it either place and he might be able to you help out!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.