First, just as sort of a NB/FYI for the other coders on the site: as demonstrated by this backtest, if you're using data from quandl, etc, with the fetcher and backtesting in minute mode, you do still have some lookahead bias! Right at the beginning of the day you'll have access to all the information from that day's line in the CSV, which if you're using quandl, includes the close.

Yup, I have made this mistake several times already.

yep happened a lot to me and now I'm very very vigilant (as I put an algo live that had it and did not do what I predicted it would do. These are 2 examples of fixes I use:

#fix fetcher cols  
def fixVIX(df):  
    df  = df.rename(columns={'VIX Close': 'price'})  
    if get_environment('arena') == 'backtest': df = df.tshift(1, freq='D')  
    df.fillna(method='ffill')  
    df['symbol'] = 'VIX'  
    df = df[['price','symbol', 'sid']]  
    #view(df)  
    return df

def fixip(df):  
    df  = df.rename(columns={'Value': 'production'})  
    if get_environment('arena') == 'backtest': df = df.tshift(3, freq='M')  
    df.fillna(method='ffill')  
    df['symbol'] = 'IP'  
    df = df[['production','symbol', 'sid']]  

sometimes you need to know how your data is sampled. The INdustrial production is sampled such that you have to push it back 3 months to be free of look-ahead bias. To make sure I always work with the latest data in live of forward testing situations I use the arena parameter.

PB

so in this case it would be something like this

ow and I gave up using the daily data: just is rubbish and you spend an enormous amount of time developing only to find out it doesn't work in minute mode..... maybe coz I'm a rubbish developer but I'm just using minute mode

Agreed, though I haven't given up on daily mode. Nowadays my handle_data is empty, and I schedule_function everything, which winds up working okay in both daily and minute, though clearly in daily mode you lose out on any intraday edges.

Data poisoning is almost inevitable here if you run a back-test look at the chart and do some adjustment your "Theory" is no longer valid.
I think it would be useful if Q would provide something like Wekka the machine learning software where you can define cross-validation split time into train-test. If we take time series one could offer validate on bull/bear/volatile market.

I've been thinking some additional way to try 'new' data would be useful: even if it were only the injection of brownian geometric walks into the price stream (in way so you couldn't tell) would be useful to run random case scenarios.
WEKA is very useful (I use it for my other research) but unfortunately its Java based (not that makes it impossible...just saying)

We have (and continue to) work really hard to make the backtester just automatically do the right thing. Slippage, commission, etc., if you do nothing, you end up with a decent result. fetch_csv is harder because we don't control it in the same way. The classic example is earnings. If you fetch corporate earnings data, you'll start seeing trades on the last day of the quarter - that's obviously wrong. Earnings data comes out weeks after the last day of the quarter. It's much more pernicious when the data is off by only a few hours, like backtesting on a day's data that is only available at the end of the day.

We have a few projects in the works that will help manage that. We're looking at ways to load data that result in better point-in-time data sets so that not only do you have the data, but you know when you knew the data. Until those are done, you'll have to be smart about how you use fetched data.

For live trading, the algo will run fetch_csv in the early morning, and then trade according to whatever data is has.

For minutely v. daily data fill prices, the FAQ covers that I think.

This is probably not news to many folks here, but lookahead bias can happen in paper trading as well as backtesting, so the admonition to paper trade before trading with real money [looking for confirmation of favorable backtesting results, say] will not protect the unwary from code that has a lookahead bias.

I wrote a blatantly future-biased code that fetches SPY data from Yahoo. In backtesting, each morning it looks at Yahoo's close price for the day (thus the future bias), and buys SPY if SPY will close higher and SH if SPY will close lower (see attached backtest).

One way I know of to fix the lookahead bias is to use tshift to shift the time index, like so,

def rename_col(df):  
    df = df.rename(columns={'Close': 'price'})  
    df = df.fillna(method='ffill')  
# deadly future bias if following line is commented out  
    df = df.tshift(1, freq='b')  
    return df  

With that change, this algorithm does poorly, with a return of -28% year-to-date, as it is subtracting the morning's SPY from the previous day's close to determine whether SPY is going to go up or not.

A few days ago, I started paper trading these two codes, the future peeker (without tshift), and the future non-peeker (with tshift). They gave the same poor result until today, when the future peeker surged ahead (see screenshots, below).

Perhaps someone from Quantopian can fill in here, but I believe that the paper trading algorithms are being re-run from time to time (when? why?), so a code with lookahead bias can, as here, suddenly show dramatic improvements. I think the gotcha is that if you weren't paying close attention (went on vacation, say), you might look at superior returns from a paper trading algo with a lookahead bias you weren't aware of and conclude that it was ready to go live.

Is this the reason that fetch_csv is no longer allowed in the contest? (Codes with lookahead bias could have superior paper trading results but fail when live traded.)

Robert

P.S. In case it wasn't obvious from the above, I think Quantopian is a fantastic resource/IDE/community. Just hoping that in the future it isn't as easy to shoot oneself in the foot in this particular way (since I already have plenty of ways to go astray [overfitting, data snooping, etc.] :)).

Future non-peeker screenshot:

Future peeker screenshot:

I don't see how you could actually have data-snooping in paper trading, even with fetch_csv (though data-snooping with fetch_csv is very easy in backtesting).

More likely you just hit some random difference between close yesterday vs close two days ago?

Ooooh you know what, I think Quantopian paper trading some something silly if you change the values for the same date over time. Yeah, perhaps there is a bug...

Or a feature...:)