Thanks Rich,

I think I could work up a "simple moving average crossover which checks the rsi." What is the basic outline of such an algorithm? Presumably, the RSI would use daily closing prices, right? Also, would it be important for submitted orders to be filled by the close of the day?

Grant

Hello Rich,

I think you are attributing Grant's comments to me in error!

P.

Hello Grant,

Are you going to try this with a batch_transform? I'm not sure that's possible with the (somewhat unclear) issue Thomas mentions here.

P.

Peter,

I can't click on the "here" link above. Please provide the full link.

Thanks,

Grant

https://www.quantopian.com/posts/batch-transform-in-minute-backtests

Hello Grant,

I'm getting confused. Maybe with refresh_period=0 it is possible. I'm generating this at the moment in a batch_transform:

2013-01-07PRINT 24 114 67 122 630  
2013-01-04 14:31:00+00:00 536.8700 37.9000 36.5792 43.2000 58.8000  
2013-01-04 15:01:00+00:00 535.5400 37.8899 36.6500 42.8200 58.8360  
2013-01-04 15:31:00+00:00 528.5100 37.8250 36.4600 42.6200 58.6950  
2013-01-04 16:01:00+00:00 527.4700 37.7610 36.4150 42.6199 58.6750  
2013-01-04 16:31:00+00:00 526.2900 37.7500 36.3600 42.7200 58.7700  
2013-01-04 17:01:00+00:00 529.4320 37.8350 36.4099 42.7300 58.9000  
2013-01-04 17:31:00+00:00 532.0000 37.8300 36.3600 42.6400 58.9800  
2013-01-04 18:01:00+00:00 529.8180 37.8300 36.3050 42.5650 58.9000  
2013-01-04 18:31:00+00:00 529.8400 37.8900 36.3850 42.5900 59.0200  
2013-01-04 19:01:00+00:00 528.0800 37.9700 36.4200 42.6700 59.1100  
2013-01-04 19:31:00+00:00 528.7500 38.0200 36.4700 42.6200 59.1200  
2013-01-04 20:01:00+00:00 529.8300 38.0300 36.3900 42.5300 59.2343  
2013-01-04 20:31:00+00:00 530.4800 38.1199 36.4100 42.5700 59.2300  
2013-01-07 14:31...  
2013-01-08PRINT 24 114 67 122 630  
2013-01-04 15:01:00+00:00 535.5400 37.8899 36.6500 42.8200 58.8360  
2013-01-04 15:31:00+00:00 528.5100 37.8250 36.4600 42.6200 58.6950  
2013-01-04 16:01:00+00:00 527.4700 37.7610 36.4150 42.6199 58.6750  
2013-01-04 16:31:00+00:00 526.2900 37.7500 36.3600 42.7200 58.7700  
2013-01-04 17:01:00+00:00 529.4320 37.8350 36.4099 42.7300 58.9000  
2013-01-04 17:31:00+00:00 532.0000 37.8300 36.3600 42.6400 58.9800  
2013-01-04 18:01:00+00:00 529.8180 37.8300 36.3050 42.5650 58.9000  
2013-01-04 18:31:00+00:00 529.8400 37.8900 36.3850 42.5900 59.0200  
2013-01-04 19:01:00+00:00 528.0800 37.9700 36.4200 42.6700 59.1100  
2013-01-04 19:31:00+00:00 528.7500 38.0200 36.4700 42.6200 59.1200  
2013-01-04 20:01:00+00:00 529.8300 38.0300 36.3900 42.5300 59.2343  
2013-01-04 20:31:00+00:00 530.4800 38.1199 36.4100 42.5700 59.2300  
2013-01-07 14:31:00+00:00 520.6800 37.9300 36.0500 42.0900 58.9300  
2013-01-07 15:01...  

I'm not crazy about the intervals. Should a day's 30 minute bars be 15:00 through 21:00? (Sorry, GMT here.)

P.

Hello Peter and Rich,

To do the "simple moving average crossover which checks the rsi" here's what I am thinking:

1. Use minute data.
   
2. Compute the moving averages with a custom batch_transform (refresh_period = 0), using the Quantopian data directly (e.g. a 5-day moving average is computed with a trailing window of 5*390 minutes). Alternatively, the built-in Quantopian mavg transform or TA-Lib method could be used.
   
3. For the relative strength index (RSI), accumulate daily closing prices, and feed them to the TA-Lib method (as Peter shows can be done, https://www.quantopian.com/posts/help-needed-ta-lib).
   
4. Use the moving averages and RSI to make trading decisions (i.e. as "indicators").
   
5. Timeframe constraints can be placed on the trading (e.g. wait 15 minutes between trades, close all open trades before the day's close, etc.).

Seems doable, no?

Grant

Added note:

Per my comment in the post below, it is not clear that using closing prices for the RSI is the best approach...it should be more of a rolling approach, using minute-level data, I think.

On a more general note, the whole "timeframes" discussion is confusing to me, since it seems that we talking about basically downsampling the data (e.g. grab the closing price every 15 minutes and throw away the rest of the data, i.e. information). If data are being chucked, then there is a risk that data-based decisions will not be as sound. So, rather than downsampling, it seems we should be talking about the best ways of incorporating all of the available data into the decision-making process, right?

Grant

Grant,
Yes. In my opinion, part of each trading methodology is to decide what resolution of data is important. The long term position traders says, I'm concerned about daily movements only, but not minute by minute movements. The day trader says, I want minute by minute data, even trade by trade data to so I can get in and get out. The swing traders and other momentum traders want something in between. They are mainly concerned about daily timeframes, but thier trading is short enough that they must get in and out as soon as trend changes are spotted, which means in the middle of the day, definitely prior to the close. But they generally don't take into account very short time frames.
Some people on the list have asked for multiple time frames, because they pay close attention sometimes, and other times they just sit. Perhaps they really want to get the best entry or exit price, or perhaps they want to trade actively at the open, and then just hang out all day till the close.
So traders select the resolution of the data for their purpose, but they shouldn't lose data. A fifteen minute period should have a proper high,low and close for the entire 15 minutes, not just the last minute.
A comment on your framework. It looks good. I think it's important to be able to use ta-lib. I asked for and RSI as an example. The purpose was to find out how to use ta-lib to get any indicator. It's easy to get a moving average, so we don't really need ta-lib.
Thanks for jumping in and helping figure this out.
Rich

Hello Rich,

In the scenario where we have 20 days of minute data cut into 260 x 30 minute slices what do we use as the closing price? The close of the last one minute of the 30 minutes?

P.

Hi guys - the Quantopian development team is working on a feature review that is closely related to these questions. This thread has been really helpful to us - so thanks for the dialogue.

I wanted to share a high level overview of how we are thinking about this issue and invite any additional feedback and use cases here.

A few use case examples we currently have in mind (note: we expect these all to work in both daily and minutely backtest mode) are:

1. I want to compute a 90-day moving average of daily close prices. duration = 90 trading days, frequency = daily
   
2. I want to compute a 30-minute trailing aggregate of trade volume. duration = 30 minutes, frequency = minutely
   
3. I want to compute a moving average crossover using trailing 30 and 60 minute averages. duration = 60 minutes, frequency = minutes.

The goal is to provide access to a trailing window of historical data inside the handle_data method in a flexible, but intuitive way.

We are currently leaning towards allowing the user to independently define the duration and frequency of the historical data they want. The duration would be the longest duration you would need in the algo (e.g. the large window size from a moving average crossover signal with a large and small window).

Once the user has defined these parameters they will then have access to a data structure (details TBD) inside handle_data that they can slice, dice, down-sample and manipulate in any way they choose.

Initially we expect that daily and minutely frequencies are the most common - so we would make those the defaults, but we would offer good documentation with clear examples on how to re-sample to get arbitrary time frames. Additionally, we would make it easier to reference trailing daily data from inside of a minutely backtest, solving the issue of asking the user to calculate 5 days * 390 minutes/day (and keep track of holiday half day gotchas and so on).

Thanks again for all the discussion on this issue - we consider this a critically important feature and value your input as active members of the community.

Jess

Peter,
That is the way I see it. Open is open of the first minute, close is close of the last minute, and high and low are max and min across the highs and lows of all minutes.

Rich

Hello Rich,

Thanks - that's what I assumed.

P.

Hello Jessica,

Aren't the data already fully exposed by the batch transform? It is confusing that you are working on another approach, since I thought that with the batch transform refresh_period = 0, there is just a stream of OHLCV data available, down to the minute level? Is there something that couldn't be done already with the batch transform (and Python/Pandas/etc.), if you provided some examples? Looking at your use cases 1-3 above, couldn't they be handled already?

I gather that one sticking point with folks is that a lot of the established analysis methods are geared around OHLCV data at a certain frequency (other than daily/minutely). But if I understand the Quantopian underlying "event-driven" architecture, the backtester can only run on one base underlying frequency, either daily or minutely bars, correct? So, if you re-sample, handle_data will still be called every day or every minute, depending on the base frequency of the backtest, right? Or will you actually support running the backtest at alternate base frequencies (i.e. do the re-sampling prior to the backtest start, and run the backtest on the re-sampled data)? Or maybe set up two different re-sampled data sets (prior to the backtest start), at different frequencies, and run on both simultaneously (e.g. daily, and 15-minute bars)? It seems that the architecture could have been different from the get-go, or maybe I'm misinterpreting?

Might you also be fixing the warm-up issue for backtesting?

Grant

Hello Jess,

That all sounds extremely promising. Some thoughts:

(i) I would suggest you offer all the common timeframes from 1 minute to 1 year - don't make yourselves different to the mainstream if you want to attract users. Obviously make duration and frequency customisable as well.

(ii) 'Warm up' needs to be addressed, as Grant says. Only the small number of built-in indicators (mavg, stddev, vwap) come warmed. The TA-Lib functions are not although they do handle NaNs during the warm up period.

(iii) The TA-Lib library needs to be available for use on the data structure you are proposing in 'handle_data'.

(iv) Maybe not related but since you are now here: do you see 'Live' algos - not 'Paper' - only been fed minutely price events? That feels very wrong and very different to the rest of the algo trading world. (I can't use 'space' - it's an age thing.)

P.

Hello Peter,

The built-in indicators aren't actually warmed up, but rather build up from the start of the backtest, eventually reaching steady-state after the backtest has progressed beyond the trailing window length. From the help page:

Note that simple transforms don't have a warmup period in backtesting. As an exmple, if your algorithm uses mavg(3), the window for the moving average won't be filled until after the third day. The first day mavg(3) will report the first day's price, the second day will report the average of the first two days, etc. However, in paper trading and live trading the simple transforms are warmed up.

This can be a "gotcha" if you don't realize it.

Regarding you (iv) above, this is a good point. Ideally, the backtest/paper/live progression should be seemless. Presumably, Interactive Brokers (IB) will be providing a low-latency minute-level OHLCV data feed (and it would seem that the data would need to come from IB, since they will be on the hook for filling orders at prices that relate directly to the trailing data). Or perhaps feeds at multiple frequencies will be available, for which an "event driven" algo would capable of triggering off of each independently (sychronously? asynchronously?)? I'll post a separate thread with this question.

Grant

Jess,

Not sure what's in the works, but I recommend maintaining the native pandas functionality that is presently available with the batch transform.

Grant

Hello Grant,

Thanks, I did not realise that. Maybe it's because I speak English English rather than American English but to me:

Note that simple transforms don't have a warmup period in backtesting.
As an exmple, if your algorithm uses mavg(3), the window for the
moving average won't be filled until after the third day.

should say

Note that simple transforms have a warmup period in backtesting. As an
exmple, if your algorithm uses mavg(3), the window for the moving
average won't be filled until after the third day.

as to me those first two days are the warn-up period. Unless the warm-up period is meaning a period of days before the backtest start.

If we use TA-Lib ta.MA(timeperiod=3) in an algo then that doesnt' have a result until day 3. So that has a warm-up period? Or not?

P.

Hi Guys,

@Peter - thanks for the comments, I get #1-3, actually afraid I was unclear on #4. It might be answered below, but if not please let me know.

@Grant - I think some of your questions above relate to the frequency of how often the handle_data event is called - my post was intended to address just the historical data snapshot that you have access to within handle_data for computing signal values. One example that batch_transform doesn't satisfy currently is to let me compute a trailing 90 day moving average of close price (based on daily close) when I'm in running my backtest in minute mode. In theory I could get that by asking for 90 days of data - but it will be 90 days of minutes which will just give me a runtime error, along with being awkward to handle. Another way of saying that is, that if memory were infinite and free - then we could just always return minute-level historical data and provide convenience wrappers to down-sample - but since it isn't you can make a tradeoff by specifying a less granular frequency and getting a longer historical data window at the same 'cost'.

We want to make sure that the same code called the same way will function regardless of whether you choose to backtest in 'minute mode' - where handle_data is called each minute, or in 'daily mode' where handle data is called 1ce per day.

The live feed we get from IB will indeed be minute level, and your live algo will be able to run at that frequency - should be seamless from backtest to paper trading to live trading. The question is just, each new minute, if your signal needs to be recomputed based on trailing historical data, how do we make that data accessible to your algo in the fastest and most intuitive way.

I'm also happy to discuss offline in email if this still doesn't answer all of your questions!

Jess

@Grant - Also there's nothing in the works that would break the ability to use pandas - we continue to see pandas as an integral part of the toolset for quant finance work!

Hello Jess,

Maybe I have some conceptual difficulty here. If I was trading off a 1-minute candle chart then the chart will print a new bar each minute. But in platforms I've seen - mostly Forex admittedly - the current bar is being continually updated with new prices and any algo reacts to that price. In the Quantopian/IB/minutely event scenario if I'm short a stock that suddenly takes off I have upto 59 seconds to realise an unlimited loss until the algo trades out of my losing position (or I get a margin call in the meantime).

P.

@Peter - good question - but totally unrelated to the historical data timeframes question I was trying to answer here :)

Quantopian's backtesting and live trading are all built around minute-level pricing data and that is the fastest access that Quantopian-IB link will give you to the market. For the equity markets this is pretty standard for any type of trading that falls outside of market making or what is often referred to as high frequency trading' or HFT. As you described, you are exposed to intra-minute price risk.

Peter,

The built-in transforms are not warmed up in the sense that they return numerical (rather than None/NaN) values at the backtest start, but as you point out, the computations are done for partially full windows. When I first started using Quantopian, I didn't realize this fact. At one point, I wrote an algorithm with some whacky start-up behavior, and traced it down to the built-in mavg.

Grant

Hello Grant/Rich,

This is my attempt at getting 30-minute closing prices in a minutely algo. It definately has one issue - I get 261 30-minute intervals out of 20 days!

bars = data.resample('30Min', closed='right', label='left').dropna()

<class 'pandas.core.frame.DataFrame'> DatetimeIndex:  
261 entries, 2013-01-02 14:30:00+00:00 to 2013-01-31 14:30:00+00:00  
Data columns (total 1 columns): 24  
261 non-null values dtypes: float64(1)  

P.

Richard and Peter,

I started a new post:

https://www.quantopian.com/posts/ma-cross-over-w-slash-rsi

So far, I just have the custom moving average, but will add the TA-Lib RSI (run on daily closing prices) as time allows. Feel free to comment/contribute.

Grant