Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Monolithic Code?
Fairiz Azizi
posted Oct 25, 2014

I'm a software engineer by profession, and trying to figure out the best way to structure complex algorithmic code.

The problem is, that any reasonably sophisticated algorithm would require many ideas around money, portfolio and investment management.

This invariably would necessitate dozens (if not hundreds) of helper functions.

The Quantopian IDE seems to support editing only one 'file' per algorithm, so this would mean I would have to put all these functions in the same file, i.e. a monolithic file!

What's worse is that, many of these helper functions would likely be useful in other algorithms... and this means copying the same code to multiple files/algorithms (and we all know that code duplication is bad, right? :) )

Leveraging my experience in software engineering build and release processes, I supposed I could create a build/deploy process by generating the monolithic algorithm files from templates.
Even if that would work, it would be useful to have a Quantopian API to allow batch updating of algorithms.

However, this requires considerable skills that I suspect is not readily available to the average Quantopian :)

In the mean time, does Quantopian have plans to allow the end user to create their own user defined library modules for inclusion into their algorithms (or even to share...).

Fi

3 responses
James Jack
Oct 26, 2014

Definitely +1 on this

John Fawcett
Oct 26, 2014

Hi,

We've been talking quite a lot about the developer workflow. We're all software engineers, so we feel the same pain. We worked on debugging because that was the biggest barrier to serious development. Right now, we're focusing on research via the ipython notebook, because that's where the bulk of the quant work is. When we kicked off the research notebook project, we all agreed that it would most likely make the need for code reuse more acute. While inelegant, you can generally cut'n'paste code between algorithms. To share code between research and algo development, you really need some abstraction into a library. To build and maintain a library, we think you need at least three things:

  1. git for source control, with the ability to reference a comitish in a repo for inclusion in an algo and/or research notebook lib path
  2. environment for writing and running unit tests on your library
  3. multi-file editing and debugging in the IDE

I'm a fanatic about focus in all areas of our business. For the workflow aspect of our product, we're focusing on research for now. That means we will be limited to nodding yes and agreeing with you on this need for a while :). Thanks for the ideas and I hope you'll keep sharing.

thanks,
fawce

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Fairiz Azizi
Oct 26, 2014

I certainly understand the 'nodding yes and agreeing', we do the same at the office, particularly in the face of competing high priorities (and I'm sure there are plenty of them still at Quantopian!).

Github is great (and I hope also BitBucket, since they offer free private repos), but, again, great for devs like me, but data scientists can't be bothered, unless it's well integrated (i.e. transparent) in your IDE.

Test environment (and probably CI, for that matter), would also be a plus (and I'm sure it'll go a long way towards the hedge fund profit sharing certification idea).

Keeping a close eye on your future enhancements, and hope the momentum keeps building!

  • Fi
About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.