@Tim, great idea for an algorithm, and thank you for the incredibly helpful feedback! You sparked many new ideas for me, thank you. Here are some of my thoughts:

• I love your question about transforms because it goes to the heart of our approach. We want to provide a broad and common library, while also helping you build your own. We are looking at ta-lib as a way to rapidly accelerate the expansion of our transform library. To let you build your own transforms, we are working on an extension to the execution environment that makes it easy to work with a window of trade data. Lastly, we have opensourced our entire backtest library zipline. The spirit is to help without forcing you to use our kit only.
  
• We have heard the same complaint about differing frequency between the IDE backtest and "full". We plan to update the UI so you can control bar length in both places.
  
  • Our new transform api will let you specify the refresh frequency for your transforms. This will enable you to divorce transforms and other logic from the underlying trade bar length, just as you've proposed. I've written a bit about the forthcoming batch transforms in the end of this thread: https://www.quantopian.com/posts/universe-selection
    
  • +1 for debugging and step-through. I have been dreaming of adding that kind of functionality to the IDE since the very first prototype. We haven't yet put this into our delivery plan though.
    
  • I can imagine a drag and drop UI that emits python code compatible with our API, but in my own experience, I've always found that graphical programming languages tend to be too rigid compared to just coding. However, reading our notes it occurred to me that a simple GUI could be a great onramp for a full programming interface - GUI generates python code. New programmers could use it like scaffolding to learn some python and our API, and then hand edit the generated code to get started. Just an idea, what do you think?

again, thank you so much for taking the time to share your algo and your ideas. Both are really interesting and helpful!

thanks,
fawce

As Fawce said, thanks for the thoughtful ideas here.

On your "bigger question": I think you're asking, are we going to provide a library and let people clone and edit? Or are we going to make a toolset where anyone can quickly implement and test an idea?

I'd like to provide both - I don't see a conflict between the two ideas. They are both a fair amount of work, but they work hand-in-hand. If we can provide a good library and the tools to manipulate code easily, that will unlock a lot of creativity from our members.

Did I understand the question correctly?

Thanks

Dan

For now at least

@fawce, @Dan, it was a pleasure, I enjoy coding algos and giving feedback: a perfect combo

@fawce - providing a broad common library whilst also allowing users to code their own transforms makes perfect sense. One the one hand, using the provided library would allow those who don't want to code transforms for whatever reason (time, knowledge etc.) to get as deep into building their strategies as possible. And on the other, the ability to customise, tinker, and create your own transforms is maintained.

If you're definitely going to expand the 'default' transforms library, I guess, to some extent, the question becomes one of how that library gets populated. I had a quick look at ta-lib and it looks reasonably deep in terms of technical indicators. I'm not sure how nicely Quantopian plays with R, but you might also want to look at Joshua Ulrich's TTR as a candidate suite of indicators.

(It also occurred to me that if there users out there busying themselves creating their own cool custom transforms, should they wish, you might be want to accept these into the default library - subject to relevant tests around robustness of code etc. - maybe in return for some sort of enhanced standing in the community (or more t-shirts!)… By being arbiter of the 'default' library Quantopian can help with things like argument-naming-consistency (i.e. years == yrs == y).)

I agree wholeheartedly that the drag & drop kind of functionality would be a great helping-held for those new to strategy writing, but it's completely not a method for those more advanced quants who wish to see (& write) what goes on under the hood. In the end, getting such functionality up and running might cost more dev time that it's worth.

I also, read through your post about the batch transforms - sounds like a great step in the right direction.

@Dan - you understood my question exactly, and I think both yours and fawce's reply definitely outline the best approach: provide as many good tools as possible, but also allow users to create their own as freely as they wish. (my only request - as a user - is maybe to think about some very broad optional 'best practice' guidelines (not very open-source, I know - sorry!) for those of us that might seek to develop production-level transforms for use by the community around naming conventions for function args etc, so that reading one user's code is as easy as reading the next user's code…)

Really enjoying using this platform - keep up the great work!

@Tim a small, but hopefully helpful, note, but you can use Python's collections module's deque, http://docs.python.org/library/collections.html#collections.deque, to implement the rolling tick history window that you use in your algorithm.

e.g.

    from collections import deque  
    num_ticks = 6  
    context.tick_history = deque([0] * num_ticks, num_ticks)  

To add a new value all you would need to do is:

    context.tick_history.appendleft(price)  

You might find it a nice little way to remove an extra line of juggling list positions.
It also provides the benefit of better memory performance, etc.

Hope that helps!

@Eddie - that's great, thanks very much. Think I'll be using that a lot. Really appreciate the feedback.

Hello Tim & all,

As I see it, a big advantage of Quantopian is that they have licensed a historical market dataset and are offering it free. Unfortunately, since they cannot distribute the data, they are constrained to offer limited access to it through their browser-based backtest application. To me, a better approach would be to offer users a flexible virtual/remote desktop environment with full access to view (but not download) the data. Users could then pick their analysis tools (one of which could still be the Quantopian browser-based backtest application). Quantopian folks, would this approach work technically and legally? If so, would it make business sense?

Hi @Grant,

We hear you loud and clear. My perspective is that there are three essential activities for quants:

1. Research and exploration of source data
   
2. Algorithm development, testing, and simulation
   
3. Real trading

We started with #2, and we remain focused there. I believe it is critical to reach a minimum level of competency before taking on the next challenge.

I totally agree with the spirit of your idea, and I associate it with #1 above. What I hear you saying is: Give Quantopian users a more flexible environment to do research! That resonates very deeply, and it will be our goal when we turn our focus to #1.

We believe the time for data science tools in the browser has arrived. I wouldn't claim that we have given you the total freedom of analysis you want - but I think web delivery is the best way for us to do it.

Please keep sharing your thoughts and advice, it is helping and inspiring us immensely.

thanks,
fawce

Thanks Fawce,

In my opinion, there is not such a clear distinction between "1. Research and exploration of source data" and "2. Algorithm development, testing, and simulation." I'm not sure what you mean by "web delivery" but from my perspective, flexible virtual/remote desktop access to the data would be preferred over having fixed input/output GUI's in multiple browser tabs. I poked around the web and came across http://guac-dev.org/, which claims to allow "access to multiple desktops through a web browser."

It'd be better to start a separate discussion, but I think this gets back to Lou Belle's questions regarding the business thrust of Quantopian. I imagine that you need to get to "3. Real trading" ASAP so you can make some profits off of fees/commissions, right?

Hi Grant -

I am sure @fawce agrees there is no bright line between #1 and #2. As we talk to people who trade (here, in person, at PyData, through our investors, etc.) we listen to quants talk about their creative process. They start with general ideas, do some whittling down, figure out roughly where their ideas are bearing fruit, and then drill deeply and test exhaustively. We've spent a lot more calories so far on the bottom end of that process, the exhaustive testing. We're spending more calories now on the higher part, the earlier research. We absolutely want it to be an integrated process.

Can you explain to me more about the remote desktop idea? If you remoted to our desktop, it would still be our IDE. How does that work for you? I'm missing something on that one.

Real trading is high on our list, but it's not the be-all end-all. We have enough money in the bank; we're not in a race against the clock. Like any startup, we're doing the dance of limited resources and near-unlimited ideas. We want live trading, better backtester, better IDE, better community, more data sources, open-sourced data sources, more data transforms, open-sourced data transforms, and more. We're listening to our members and our market, prioritizing as best we can, and getting it done as fast as we can.

Thanks Dan,

The remote desktop idea is to give users the experience that they have the data you have licensed at their disposal and could analyze it as they wish (including but not limited to using the Quantopian IDE). Imagine a Linux remote desktop environment, for example, with a suite of programming and analysis tools. You'd need to block the means to transfer data (at least big chunks of it) from the remote desktop, to preclude the licensing issue of enabling re-distribution of the data. Each user would have some disk space on the remote system, as well. So, to the user, the experience would be as if he had purchased the historical database and loaded it onto his hard drive. In this way, very flexible screening, statistical analysis, plotting, visualization, etc. could be performed. Hope this helps...let me know if more clarification is needed.

That definitely makes sense. I think we can deliver all of that in the browser.

For the remote desktop you described, we'd have to examine and modify and whitelist and install each app in that suite of programming tools. We can deliver the same functionality, but faster, by doing it in the browser. We can install all sorts of Python libraries and make them run in the browser, from pandas to matplotlib. So I think we've got a similar vision - the question is if we can do it in the browser like we think we can.

[jumping ahead 1.5 years or so] I'm not spotting anything wrong in the code (aside from my usual pet-peeve against use of pronouns like "we" in comments) and yet those RSI values don't appear to be in the same ballpark with charting sites out there.

RSI calculations in Quantopian do not appear correct

Ok here are a couple of ways to obtain RSI values on Quantopian pretty close to charting sites, TradingView, Google, et al...