In answer to the question: I don't have any HFT examples available. Quantopian uses minute-bar data, so the highest frequency you can meaningfully trade with our software is minutely.

And the meta-question. Why do we have someone named "guest" asking a question in the forums? The answer, my friends, is a bug in the software! We intended to require registration before posting to the forums, but didn't nail it down. Sorry about that, will fix it.

Dan

It will be good if you have tick level support. Your site may have 1 minute data but we should be able to get ticks when connected to other data sources.

Hi there Dan, Suminda,

Noticed that you are looking to test tick level data - so am I - else we could not really replicate HFT - any ideas re getting some tick level FX data into the system?

Gary

Hi Suminda and Gary,

Quantopian isn't trying to be an HFT trading platform. When we roll out support for live trading, we will be feeding only minute bar data, not tick data, to live algorithms. We believe there is a great deal of opportunity for talented quants in the area of algorithmic investing, as opposed to high-frequency trading.

Hi Jonathan,

Thanks for making this clear - however, even if Quantopian is to be a medium frequency platform you will still be dealing with prices arriving tick by tick in a live environment - surely the results you get from back testing in a minute environment based on the far greater volumes of ticks that today's markets generate have to be taken into account for any testing to be remotely meaningful I would guess ?

Gary

Jonathan,

I have to agree with Gary that a sub-minute feed to algorithms is important to consider. Think about the problem from the point of view of a stop-loss, which users will certainly want to put into their algorithms. The more real time the data is, the better the performance of the stop is going to be. Your target users may not need microsecond execution, but it's not inconceivable to believe that prices can swing one or two percent in under a minute. If Quantopian users are taking that extra hit on each of their stops then they will run out of trading capital and confidence in their algorithms more quickly than they would otherwise.

Yes, on average the stop will perform approximately the same as in the backtest, but in live trading users are going to wonder why they lost an extra $100, $200, or more on a trade then they had intended. Whether it's correct or not, I think people will make decisions about whether to continue trading a particular algorithm or not based on short term results. So, the extra slippage might, in the end, cause Quantopians to trade less. That's just my two cents. I'd really like to hear your thoughts.

-David

We may get to faster than one minute frequencies at some point, but it's not on the short- or medium-term plan. One of the neat things we're doing is using the same code for backtesting and for live trade execution. We believe that is going to enable good fidelity between backtesting and live performance, at least as far as order filling goes. One minute bars are going to drive everything.

In the end, it's all about feature prioritization. Live trading, data exploration, forex, futures, are all higher on the list than tick data. Of course, your feedback shapes that list, so we're all ears.

Dan,

My thoughts on back testing are primarily shaped by what the market leaders are doing as ultimately the bulk of trades execute via their platforms. To me back testing needs to "try" to replicate a real time environment insofar as its possible - this in itself is sometimes only achieved by market "playback" features in certain platforms. Nonetheless, I came across an interesting article which I am including a link to so as to share these insights with those interested.

NEW YORK | Thu May 10, 2007 9:12pm EDT
(Reuters) - Algorithmic trading, in which computers execute orders at lightning speeds, has financial exchanges relocating computer systems to shave milliseconds off execution times, executives said this week at the Reuters Exchanges and Trading Summit.

Transaction decisions are measured in fractions of a second in algorithmic trading in which computers are modeled to execute orders based on criteria like price or timing. One-fifth of global equity trading last year was algorithmically driven, up from 14 percent in 2003, according to financial consultant Celent.

Read the full article here:
http://www.reuters.com/article/2007/05/11/us-exchanges-summit-algorithm-idUSN1046529820070511

Gary

Dan / Gary,

I think there are a few issues here:

(i) true HFT with its need for low-latency and hence co-location is not something the retail trader is ever going to be able to do effectively.

(ii) tick-level data for backtesting may reside in the area between nice to have and must have depending on your strategy. But tick level data could be modelled either by the back test engine as in Metatrader or in the algo although I've seen the latter yet..

(iii) live trading without tick-data resides in the area between must have and using another platform, I think. Imagine you are using soft stops because of paranoia about the beahaviour of brokers to their retail clients. How could you be watching your positions going disasterously wrong in your brokers platform knowing that you have between 1 and 59 seconds befoe your algo actions your soft stop on the next one minute bar over the brokers API? (Two caveats: 1. I don't trade live at present, 2. use hard stops.)

But I think I would feel very uncomfortable at a volatile time of day knowing that my algo was reacting once a minute.

Regards,

Peter

Hi Dan,

Are the minute tics in the Quantopian data base real, in the sense that they are the actual OHLCV data for the actual historical minute, capturing all of the high-frequency tics from the beginning of the minute to the end? And will the tics be real, in the same sense, when you go live with Interactive Brokers?

Or will there be some sort of synthesis/smoothing/sampling going on?

Also, will there be any latency when you go live, or will the algorithm be running in synch with the high-frequency trading? All a bit confusing...presumably you'll be using a feed from Interactive Brokers...do they have an explanation of how it works under the hood?

Grant

Gary, the info from the article is, of course, a little out of date. Currently, software vendors offer out of the box solutions that promise to deliver micro-second scale execution speeds from a collocated server. We can safely assume that the HFT Market Makers are definitely below this scale. so the market these days is at least 1000x faster. If anyone is interested in learning more about HFT they should check out NANEX research: http://www.nanex.net/FlashCrash/OngoingResearch.html

However, I think the user base should put to bed the idea of wanting or needing Quantopian to be an HFT execution platform. Most of the quotes sent to the market by HFT algos never get executed on, and they're involved in a game that our algos are never going to have anything to do with. There's a great TED talk about this: http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world.html. The OHLCV are based on actual transactions, and is therefor far more relevant to the style of trading that Quantopian was made for.

Dan, if Quantopian will be able to send "stop" orders to the broker exectuing the trades, then, since stops should be entered at the time the position is opened, I think you can get around the need to update the alogs for every tick. I guess that also begs the question, will Quantopian be supporting any order types other than market orders? Based on the API docs for the "order" function it doesn't look like that is the case.

Hi, this is my first contribution to discussions on this board. Greetings to all. :)

I hope that some of you will be able to attend the meetup in Boston on April 24, because this conversation is extremely relevant to what I'm covering in my presentation on that night. My talk is about incorporating transaction costs into backtesting, and I will be discussing some of the differences between high-, medium-, and low-frequency strategies in terms of data needs (including the potential need for actual tick data) and the degree of realism required when simulating trading for different types of strategies.

A couple of points to contribute to this thread:

• The cost of building out a realistic market simulation capability increases exponentially with the degree of realism required. Tick data is expensive to buy, annoying to scrub, and it dramatically increases the computational requirements for backtesting. And once your tick data is online, you still need to write a realistic market simulator, and then develop the ability to bring execution algos into the backtest so that you can simulate fills on child orders, not just parent orders. This all takes a lot of money to build and maintain. The hedge funds, banks, and HFT shops who are running cost-sensitive algorithms that require this level of realism have made considerable investments in their backtesting infrastructure. I don't think it's realistic to expect Quantopian to march down this path anytime soon. And if they do, it will require much more than just adding tick data support.

• HFT strategies require additional costly infrastructure to minimize every operational latency. And once again, the cost of such infrastructure increases exponentially with the degree of latency reduction required. The people who win at the ultra-low-latency game are the ones who trade in sufficient scale as to absorb the massive infrastructure investment required. HFT is really a totally different animal than the types of algos supported by Quantopian, and I agree with David that it is unrealistic to expect Quantopian to evolve into a HFT platform.

• The term "algorithmic trading" covers a vast territory. Within this landscape, Quantopian's sweet spot is not HFT or intraday, but really mid-to-low-frequency: strategies with holding periods of days to months. These strategies tend to be less cost sensitive, which means they are backtestable with lower infrastructure requirements (i.e. without requiring tick data). You can backtest with conservative cost estimates and then be pleasantly surprised when you go live.

• In contrast, a really good intraday stat arb strategy might have unlevered round-trip paper profits of 5 cents on a universe with 5 cents typical spread. That makes it a very cost sensitive strategy: whether you ultimately make or lose money is going to come down to how good your execution vehicle is at capturing your alpha with minimal slippage. There is no practical way to model this with any accuracy in Quantopian's existing framework, nor would it be reasonable to expect there to be. You really need to build out your own backtesting capability in order to do this kind of stuff.

• The point that backtested results won't be "remotely meaningful" without tick data really only applies to HFT or intraday. For strategies with daily-to-monthly holding periods, you can generate reasonable cost estimates using only 1-minute bars. The point is that you don't have to be hyper-accurate with your simulated costs when you're backtesting a strategy that isn't as sensitive to cost.

• None of this is intended as a slight to Quantopian; on the contrary, I think it's amazing how quickly the platform allows users to go from "0 to 60" with only minimal investment of time and money. This is a huge accomplishment, in my opinion. There is plenty of fertile ground for users to till in the lower frequency end of the farm without jumping over the fence into the intraday and HFT areas.

Look we do not want you to spend on tick data. If the broker provides it we want to have it. We are not asking for HFT this are time sensitive and need latency guarantees.

What you guys are saying about only 1 minute data is like blind folding us in driving to slow us down and asking to have a flash peep every minute to see that is happening on the road.

To be conferrable to trade we need see the whole market, i.e., see the order book and ticks what ever frequency one trades. At least for intra-day strategies. Otherwise Quantopian will be for ever a toy to those how who might be afraid to trade on just seeing flashers through a blind fold.

David,

You say "since stops should be entered at the time the position is opened". That's an opinion many hold but some people do not want the broker to have acess to information on their stops as I stated above.

All,

I think the important discussion of tick data in live trading should be taken to a new thread. Here it is muddled with an inconsequebtial discussion of HFT.

I'm looking at a demo of IB's Trader Work Station just out of interest. It lets me view candle charts from monthly down to 1 minute but also 30 second, 15 second,10 second and 5 second. But that is solely for 'human' preference. Price changes at all bar frequencies are show tick by tick. I can't imagine a trading platform working any other way.

Regards,

Peter

Tomas,

Will your talk be on video later?

It's the first time I've read that "Quantopian's sweet spot is not HFT or intraday, but really mid-to-low-frequency: strategies with holding periods of days to months". I have no interest in HFT but a great interest in intraday. Interesting.

Regards,

Peter

Welcome Tomas,

Some feedback from someone not in the field:

• I'm confused about your discussion regarding "tic data support"...I thought Quantopian was providing actual tic data? In my mind, there's been an issue from the get-go in that Quantopian has used the quality of their data as a selling point, but they have not yet provided details on how it is obtained and scrubbed. If the HFT trade data are part of the real historical market, then do you think that they are sampling the market at a sufficiently high rate to capture all of the trades? Keep in mind that I have no idea how electronic markets work, but I figure that the "market" must have some centralized "ticker" that synchronizes and keeps track of all market activity (perhaps a flawed concept).
  
• Quantopian is funded by GETCO, which according to http://en.wikipedia.org/wiki/High-frequency_trading is one of the largest high-frequency trading outfits. So, infrastructure and expertise in HFT shouldn't be a problem for Quantopian.
  
• Your comment regarding Quantopian's sweet spot is revealing. For weekly/monthly/quarterly trading, what's the point of automation? Why would I want to set up an account with Interactive Brokers, etc.? Couldn't I do everything manually and more cost-effectively without automation (for example I see that for up to 30 trades a month, trades are free at http://www.merrilledge.com/pricing, with some minimum balance requirements)?
  
• I think you said that Quantopian is basically useless for intraday strategies, since it does not capture real-world factors important to assessing the profitability of a strategy. Did I read you correctly?
  
• Is minute-bar data useful for a low-frequency algorithm (multi-day/weekly/monthly trading frequency)? I'm trying to understand your statement "...you can generate reasonable cost estimates using only 1-minute bars."
  
• Regarding your statement "There is plenty of fertile ground for users to till in the lower frequency end of the farm..." would you be willing to publish specific recommendations for algorithms that we could backtest on Quantopian? Also, if there are lots of profits to be had at low trading frequencies, why would commercial firms bother with all of infrastructure and associated costs associated with HFT.
  
• Generally, I feel like there is a dissonance between the fact that "73% of all equity orders volume" is attributed to HFT (http://en.wikipedia.org/wiki/High-frequency_trading), and the idea that Quantopian's long-term mission is to make finance more open (to the retail masses). It would seem that individuals, without access to the kind of trading that really defines the market, will just be picking up scraps. Why can't the costs come down for retail?
  
• I agree with your compliments to the Quantopian team...very open, responsive, geeky, and entrepreneurial.

Grant

Hello Grant,

I'm going to contradict myself now: I find HFT discussion very interesting. But the discussion of tick level data is much more important. When the Wiki says "73% of all equity orders volume" is from HFT algos do you think that means 'HFT algo orders entered into the market' - most of which are spurious and facilitate some kind of price discovery - or does it mean HFT algo orders filled?

Regards,

Peter

Hello Peter,

I suggest that you start another thread, if you want, on tic-level data. I'm clueless about what you are talking about, so perhaps you could kick-start the discussion with some background and a problem statement. I'm also clueless about how retail electronic markets work (e.g. what goes on "under the hood" at Interactive Brokers and beyond). Lots of missing details, in my mind... In order to beat the system, first, one needs to understand it.

Grant

Gentlemen,

I think the mere fact we are all able to converse, share ideas and work with Quantopian at no cost whatsoever is really a win win for all. There are definitely 2 distinct discussions - Data Frequency and its impact on back testing & HFT. It will hopefully be a good place to learn and share - last - when will appropriate privacy measures be installed ?

Gary

I'll need a little time to respond fully to the last few posts, but I wanted to quickly dash off a couple of points:

• First, my statements about what kinds of algorithms are inside/outside Quantopian's sweet spot are just my personal opinions.

• When I talked about "intraday" strategies, I was talking about intraday holding periods. In other words, strategies where you are trying to profit on intraday price moves (relative or absolute) and close out positions within minutes / a few hours of opening them. If your strategy makes trading decisions every minute but has typical holding periods of days, weeks, or months then I don't consider it an intraday strategy. Yes, you are trading continuously (once per minute), but you are attempting to profit off of longer-term price moves. In which case I would consider your strategy to be ideal for Quantopian.

• Defined this way, the reason that intraday strategies are challenging to develop within Quantopian (at least at present) is that these strategies tend to only be profitable under fairly optimistic transaction cost assumptions. Just think about it: if your round trip paper profits are something like 1 x typical spread, then you need to be very careful with how you execute ... and also careful with how you model the transaction costs of whatever execution vehicle you select. Alpha decay is also typically much more of an issue for intraday strategies. It may be dangerous to develop such a strategy in a world driven off of 1-minute trade data (no quotes / no ticks). And addressing this issue is not merely a matter of providing support for tick data, regardless of whether Quantopian supplies the ticks or the broker does. It's a very expensive and time-consuming proposition, for reasons I have previously described.

• If you have written an intraday strategy that generates much higher paper profits, then you're less cost-sensitive and the previous paragraph may not apply to you. And again, if your strategy has longer holding periods, then you should be all set.

• @Peter: when people talk about HFT being 70% of US trading, they are talking about shares traded. If you measured in terms of messages, then HFT would probably be a much higher percent share.

• @Grant: in response to your "just picking up scraps" comment: you're confusing share of trading volume with profit share. HFT guys trade a ton of volume but make razor-thin profit margins (and by all accounts, these profit margins are only getting more scant over recent years). Successful low-frequency strategies make far fewer trades but with much higher average profit margin. The Quantopian platform is positioned at the market segment where they can make the most impact without building out the considerable infrastructure required to support investing at shorter timescales.

Hello Gary,

Regarding privacy/security, I've been opinionated (although I've reached the limit of my technical expertise). Here are some links to my comments:

http://blog.quantopian.com/post-mortem-of-121212-attack-on-quantopian/ https://www.quantopian.com/posts/quantopian-in-the-real-world

Grant

Hi Grant,

I believe a simple answer to your question, "why would commercial firms bother with all of infrastructure and associated costs associated with HFT?", is the level of risk. Generally, if you can buy a stuck from a seller and sell it to a buyer for a higher price in well under a second, then there is very little market risk involved because the holding period is so short. If, on average, you can make a small positive profit and can trade many, many times, then you can make some real money. This is the basic outline of a HF market making strategy, something that Kinght Capital is famously involved in. A HF strategy like this can work for someon like KC because they're per transaction costs are very low, whereas, because of brokerage fees, a retail trader's fees will be much higher. If you can't cover your transaction fees, then you can't amke money. This is a basic intuition as to why HF is feasible for institutions, market makers, and professional traders, but not for retail investors.

Instead, by holding a position for much longer, a retail investor is inherently taking on more risk, but has the opportunity for greater returns on a per trade basis. That's why Quantopians sweet-spot is on a longer time scale than HFT.

Let me know if that clarified things a bit.

Best regards,
David

Hi there David,

Great comment - agreed for the most part - however still am of the opinion that back testing on a per minute basis will only give a limited assessment of forward execution based profitability - however, to be able to do so with little more than effort is a great win for all involved.

Best regards
Gary