Peter, seems like you have a lot packed in here! Could you tell me more about what you're looking for? And maybe with an example?

What precisely is the problem with using fetcher?

(I get that it's not optimal, but it seems far more convenient to me than using Zipline offline.)

It's too easy to cause look-ahead bias, it's tedious to massage CSVs from random data sources, the data is rarely available at a higher frequency than daily, you have to twist to get to to work in paper-trading at all by forward-filling the last row (thanks Alisa!), and generally it's super annoying.

Definitely would love to have proper "realtime" index data.

Yes, I agree with those points, but I don't see how they're solved by using Zipline on your own computer or using R.

The more built-in minute data the merrier of course.

Hi Alisa, Let me recap the main issues with fetcher:

Fetcher does not adhere to schedule function: the fetcher data is called every day before trading and most of my alhorithms do things near the closing. When it's called before the open I always work with yesterday's data.

Secondly it's too easy to make mistakes with shifting data in time, I bet most Algos which use fetcher have a look ahead bias in back testing and loose that in real live trading, hence the backtest and live performances are far apart.

Thirdly there are only 5 calls allowed and one call can only be bound to one Symbol. This excludes Algos where one needs more (VIX complex)

When it comes to indices: indices are tickers in Bloomberg and in my terminal I can easily create a strategy with NUGT and DUST based on the behaviour of HUI, the underlying index. Even the simplest MA crossover strategy I cannot create in Quantopian as I only can get the HUI of yesterday....

Same for the VIXs. I would love to create an algo trading changes in the VIX complex, wether it has contango or backwardation structures and how much and with which velocity. If I would be able to call the VIX futures as normal symbols in DATA that would solve it.

So either include a set of indices as callable SID in DATA, or as a patch allow algo developers to call fetcher in a scheduled function or determine the time fetcher will be called as before trading is the most unhelpful time.

Ditto on the bias. The algo I posted last night had one. The results got better and better the earlier in the day I ran my schedule_function until I got very suspicious. Sure enough, I was using the VIX daily close during the morning of.

I'll second (third?) the request for index and even better ETF NAV values.

I want to restate some of the ideas from this thread to make sure I understand them.

• Peter, Simon, and Kevin are all looking for a live (minutely) feed of a CBOE index, VIX
  
• Peter (and others?) are looking for a live (minutely) feed of a NYSE index, HUI

I call them out separately because of the different providers. We've looked at VIX before but we haven't pulled the trigger yet because of the cost of buying the historical and live data from CBOE, plus the en. I've never priced out the cost of the NYSE indices. It's good to hear the requests for them, though, because requests like this help drive the priorities.

Other things from the thread:

• Peter is looking for prices for commodity futures. This one is actually a work in progress! We've started work on supporting trading futures. The price data will come as we work on that feature.
  
• Simon and Peter point out that it's too easy to get look-ahead bias using Fetcher. I agree with that. I haven't found a solution to it that I like yet. The power of Fetcher is that it lets you bring in anything. The challenge is that when you are flexible enough to bring in anything, it's hard to manage. . .
  
• The concept of minutely-fetcher is a great one. I want to build it, but we haven't gotten to the point where it is a next-priority yet.

Kevin would like to have the NAV of ETFs as well. You might be able to kill two birds with this, as having the NAV of many of the index ETFs is a pretty good proxy for the underlying index (although sadly not for VIX).

I think almost every problem can be solved by be able to call fetch_now(fetcher_id) at the time you actually need it as we can use external apps do to the actual databuilding, or use Kimono, or Quandl or Yahoo directly. To make sure the script doesnt timeout in 50 secs, the developer could run a schedule_function at T-1 to get the data and do the work at T-zero or if you know that your fetcher is responsive one could do it it one go.

Moving that higher in the roadmap would allow the algo developers to solve for most issues. except for lookahead bias off course. As I'm planning to run most of my investments through Q at one point I'm more concerned about the fetcher on-demand then the look ahead bias as the look ahead bias will be caught out while paper trading (and comparing it with backtesting the same period).

[I develop, backtest*100, put live in paper-trading, leave for a month, then back test the papertrading period and see if the pattern is the same. If so: great, if not.... back to start.... If all good: put live]