Hello Pedro,

Additionally, I think that new users should be encouraged to write code that will run on minute-level data, since Quantopian will only support paper and real money (live) trading at this frequency ("Before you can paper trade your algorithm, you must run a minute backtest."). There are some additional complications, but the payoff is that with the click of a button, the algorithm can be launched into paper trading for a reality check. The new history API should help in writing minute-level algorithms.

Grant

That's some great feedback, thank you.

It is getting to be time to revise the sample algorithm - as Grant says, getting more people thinking about minute-level algorithms is an important goal. We need to think through the other goals of the sample too. How much of it is intro to Python, how much of it is intro to finance, how much is intro to Quantopian? I like the idea of having different samples for different starting points, too.

Dan,
I'm backtesting a strategy that runs on the close...not 1 min frequency data. If i want to papertrade, is there a mechanism to do so in Quantopian? If not, it seems like a limitation. If i can backtest on daily data, should i not be able to papertrade it as well?

@Adam that is exactly what I have been thinking. More than likely the average user on this site is not prepared to run such high frequency strategies successfully with consistency in the live market, especially if they can’t use tick data, volume bar data, range bars, renko bars, etc. I believe most would be much better suited paper testing daily strategies which are much easier to conceptualize, develop and implement using simple tools like excel. Trading at 1 minute frequencies in real time requires a lot of market structure, micro market structure knowledge, as well as having software (redi, sterling trader pro) to route your orders directly to exchanges and collect rebates for providing liquidity which helps to offset transaction cost that are inherent in higher frequency strategies. If users can only test higher frequencies then I feel they would be better served having many more time frame options to backtest on. Minute data only updates every minute while tick data is continuous as it updates every trade. Finally even if those higher frequency options become available I still believe the focus should be on the algorithmic portion of automating strategies instead of encouraging short term hft which puts you in a pool of capitalized sophisticated market participants with whom you cannot compete with and for most will be a losing proposition without the knowledge of ecn fees, dark pools, how brokers re-route and trade opposite customers, etc.

Thank you for this, this is helpful to someone just starting out.

@Pamo couldn't agree more. This is precisely how i feel about this platform. The target for this platform ..at least to me..are not large funds/banks. Rather individuals with ideas. There's no way individuals can afford the infrastructure to implement a HFT/1 min frequency strategy. To be more realistic, the focus should probably be on daily strategies.

Much needed post. I remember starting out and having literally no idea what was going on. This is much more explicit and still provides a lot of room to grow. Thanks for posting.

Hi folks,

There is a distinction between executing trades at a minutely frequency and running an algorithm on minute-level bars. It is straightforward to constrain trading to a fixed time. For example, you can add a function like this to trade only at noon:

from pytz import timezone

def intradingwindow_check(context):  
    # Converts all time-zones into US EST to avoid confusion  
    loc_dt = get_datetime().astimezone(timezone('US/Eastern'))  
    if loc_dt.hour == 12 and loc_dt.minute == 0:  
        return True  
    else:  
        return False  

The thing to keep in mind is that if you submit an order at noon, it will be filled the next minute (subject to liquidity and your slippage setting).

Grant

Hi All,
Grant - thanks for the post, you captured exactly what I was planning to suggest here.

To elaborate just a bit - our view is that Quantopian users will (at least eventually) want minute-level granularity in sending their orders to the market - some strategies do best to avoid the first and last parts of the trading day for example. As Grant points out, the control over when you trade is pretty separate from the decision on how frequently you trade. From my experience so far with our pilot live trading program I would concur with @Adam and @Pamo that the common starting use case for Quantopian traders will be daily, weekly or even monthly rebalancing.

I'm attaching an example backtest from a pairs trading algo that I shared previously which shows two techniques that are useful in migrating a strategy you build with daily backtest mode over to minute mode.

1) There is a time based IF statement that constrains the algo up updating the OLS regression once daily at 10:30 AM:

exchange_time = pd.Timestamp(get_datetime()).tz_convert('US/Eastern')  
if exchange_time.hour != 10 or exchange_time.minute != 30:  
    return

2) There is a second set of IF statements that check when the strategy last rebalanced (traded). I created a context variable called rebalance_date which is initialized to None. The logic below will execute the rebalance if that variable is None (i.e. first day of trading) or if it has been the specified number of days since the last rebalance. In the example attached I used a 20 trading day trigger - so this strategy will trade once during the day, every 20 trading days.

if context.rebalance_date == None:  
    place_orders(context, data, zscore, exchange_time)  

elif context.rebalance_date and exchange_time > context.rebalance_date + /  
                                                datetime.timedelta(days=context.rebalance_trigger):  
    place_orders(context, data, zscore, exchange_time)

My view on the daily backtest mode is that it has two main benefits, (1) it let's you set up a strategy quickly without having to worry about the execution details to do back of the envelope testing, (2) given the current constraints of our backtesting system a daily mode test runs much more quickly than a minute mode test, so iteration is easier. We have pretty aggressive performance goals for the platform - so I see this second concern as being something that becomes less of an issue over time (not to mention if and when we get to better directly addressing parameter optimization and other research workflow tasks).

Last point I'll make - if you have any specific questions on how to get the same behavior out of your 'daily' algo in minute mode so you can start paper trading please either post back to the forums or reach out to us directly via [email protected] - we'd love to help!

Best,
Jess

Hi guys,
Good ideas here. I agree that the sample algorithm needs an update.

One thing I was confused about when I first started using Quantopian was when and how trades were executed. If I bought a stock in a certain day frame, was the order being queued up only to be bought at the start of the next trading day? Or was the stock being bought in the same day? In daily mode, what minute am I buying at (if not an average of minute's prices), and is this the closing price for that minute or what? How does this work in minute-mode? I think the answer to all of these questions is important for new users, because without these answers things are very uncertain and programmers don't like uncertainty. Of course it's possible to get answers by looking at the documentation, but maybe there is a way all of this can be described in a sentence or two in the example algo.

Another thing to consider is whether using Apple is in a way dishonest and will drive people away. It would be much more interesting to me if the example algo used a stock that's price stayed fairly constant over the past couple of years but the algo made a decent profit. Also now that Apple's big boom is getting farther into the past, the example algo takes longer to run if we want to keep it updated. So cutting the algo's run time to 2 years or so could be a good idea. Also I think the variable names should be general and not specific to a certain stock (not context.apple in the current case). If I change the sid used, then the variable name needs to be changed too for the program to remain non-sloppy, which is probably not good for new users.

Also maybe adding an example of record is a good idea? Would just show how the platform is dynamic. These are just some ideas, obviously stuff that is not needed should be taken out to make the algo as simple as possible. In the backtest below I adjusted the variable names and added a possible thing to record.

Hi Jess,

Perhaps you could outline (for Gus and the rest of us) how the order filling process works for the backtester (and presumably for paper trading, as well). I know that typically, orders will be filled the next call to handle_data. However, slippage can affect this, right? Also, what if the stock did not trade historically in the next tic (day/minute), but the tic data was filled (see https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps)?

Grant

Hi Grant,

Absolutely - let me try to address each of those.

Order filling: As you said orders placed in one 'bar' are filled in the next 'bar' of data, in daily mode an order placed on day 0 is filled on day 1, in minute mode orders placed in minute 0 are filled in minute 1. There are some technical reasons for doing it this way in our system, and while it's arguably overly conservative for daily mode it is the 'best case' scenario in minute mode, this is how orders are filled in paper and live trading. More detail on ordering: https://www.quantopian.com/help#ide-ordering

Slippage models: Our backtester defaults to a volume share slippage model (we think this is better quant hygiene than defaulting to no slippage - but I'd say there is some confusion about this since people often assume that the default would be no slippage and no t-costs) The default settings limit your algo to trading no more than one quarter of the trading volume in a given minute. If you order 1000 shares of Stock X, but Stock X's volume is only 2000 shares that minute, then in minute 1 you will get a partial fill for 500 shares, on the next bar you will get another partial fill for 1/4 * that minute's volume and so on until you get the full 1,000 share order filled. The default volume share slippage model will also simulate the market impact of your trade by moving the price away from you in proportion to the size of your order. For more detail on this check out our FAQ: https://www.quantopian.com/faq#backtester and the slippage section of our help docs: https://www.quantopian.com/help#ide-slippage

In order to remove the slippage model, or set to a fixed per share slippage model, in which your order volume does not impact your simulated execution price - you can set the slippage model in the context() method as follows:

set_slippage(slippage.FixedSlippage(spread = 0.00))

Also NB: when you paper trade via Interactive Brokers the Quantopian defined slippage and t-costs are replaced with IB's simulation settings.

Bars with no trade data: This one is the same answer that Fawce gave on the other thread earlier. Orders are only filled in simulation when there is real trade data for that bar.

Hi guys, trying to grasp all the arguments made above. in the meantime...had some rudimentary questions:
1. For backtest: Should i use price_history functionality to specify the timeframe for each security, or rather a loop to run thru each to see if price_history exists for a explicit timeframe?
2. For backtest: If I"m looking at daily data, and my algo triggers an entry, is the order placed on the next day open? Same for exit?
3. For papertrade: .Can you expand upon/provide more resolution on how this is done on a daily strategy? I'm confused on how to get the "same behavior out of your 'daily' algo in minute mode" as mentioned by Jess above...

Thanks Jess,

So, orders are filled (subject to the slippage model) as soon as the next real trade data becomes available; no filling is done if trade data are absent or the trade data are forward-filled (per the aggregation described by Fawce in https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps). Correct?

Grant

Hello Adam,

The basic idea is that when you run a minutely backtest (or paper trade via Quantopian, not Interactive Brokers), you need to provide a constraint to limit the trading to once per day (if that is what you want). One simple approach is to add a statement in your code so that it only executes once per day (e.g. if it is 10:30 am, then execute code, otherwise do nothing). Another approach is to add a flag that gets set if a trade is triggered (e.g. if orders are submitted, then set the flag to True). The flag is used to block additional trading for the remainder of the trading day.

Grant

Hi Grant - yes that is correct.

Hi all,
Please bear with me since I am new here.
When I first checked the sample algorithm, I thought that it had to have a "look ahead bias" because a buy order is issued at the same time a check on the moving average of the close price. From the discussion above, I understand that the order will be issued on the next bar. However, the discussion confused me even more about the meaning of the "close price". Is it a day close price or is it a minute close price? To clarify the question: For the sample algorithm above, is it a day close price and the order is issued the next day when the price any time of the day (next day) fulfills the conditions? Or is the average price calculated minute-wise and the order is made the next minute?

Grant,
I'm assuming i can specify to run my algo at the end of day / after the market close? Such that:
from pytz import timezone

def intradingwindow_check(context):
# Converts all time-zones into US EST to avoid confusion
loc_dt = get_datetime().astimezone(timezone('US/Eastern'))
if loc_dt.hour == 17 and loc_dt.minute == 0:
return True
else:
return False

Hello Adam,

Yes, if you want to execute a block of code only once per day (running on minute bars), you can use the code you posted above. However, the backtester will not execute code outside of the trading day, since handle_data only gets called when trade data exists for at least one of the securities in your portfolio (so, it's basically within NYSE normal trading hours).

Within handle_data, you can just insert:

if not intradingwindow_check(context):  
        return  

Grant

Hello Jess,

I wanted to pick up on your comment that from your "experience so far with our pilot live trading program...the common starting use case for Quantopian traders will be daily, weekly or even monthly rebalancing." Is there evidence that more frequent trading in U.S. equities will be fruitless even for experienced retail traders? Have there been an existence proofs via Quantopian algorithms that show that trading every 5-30 minutes is feasible? Or is it pointless to be thinking at such short time frames?

Note that I'm looking for quantitative evidence, not opinions here (e.g. academic/industry studies, example algorithms, back-of-the-envelope estimates, etc.).

Thanks,

Grant

https://faculty.fuqua.duke.edu/~charvey/Teaching/BA453_2004/Muller_Proprietary_trading.pdf

Here's a thought on the simple example algos. It can be daunting for new users to think that they have to come up with an algorithm that buys and sells on their behalf. There should be strategies available to the less technically inclined user to implement traditional, simple strategies like buy&hold.

The idea is that most people trading don't have some wonderful strategy and prefer to go with the tried and true methods. Traditional, easy to implement strategies, could help get new users in the door and on the platform sooner. Once people are invested, they're more likely to begin tinkering with other ideas. My guess is that a lot of people that want to be quants don't know it yet, a couple quick wins could help to spark their imagination.

This backtest algo is the simplest one I could think of that works on minute or daily data. It buys and holds, not a cool one but everybody gets the concept and you have to get the concept to follow the algorithm.

Basically, for sample algos, simple is better.

trading on "a couple of big wins" on top of huge emotions, absolutely no background on finances (like most people here) and/or poor software engineering (going with "tried and true") on their behalf will only put them in a worse situation in the long run. This is all sounding a little crueler as time goes by.

Leo, I don't think we are looking at this the same way. I think those with little background in finance and rusty programming skills would learn a lot by using quantopian. Traditional websites use GUIs, click a button to buy some stocks, quantopian forces traders to think about everything else around the button click, why to click it, when to click it, how does the click affect my portfolio, etc.

Quantopian is a great place for people to gain experience, they have access to quality industry tools, and nobody said we have to use real money. That's why there's data to test on. I don't think too many people will go live with a broken algorithm or 'trade on emotions' because they've seen it so many times in backtests. Anybody can learn anything if they mess with it enough, but it's easy to not bother trying if it's too complicated right away. Some basic example that don't assume knowlege beyond syntax could be beneficial.

sure

I'm having trouble getting the talib function stochastic (STOCH) to work using history. Has this been ported?

Hello Richard,

Unless you specify otherwise STOCH needs a 30-day history (see attached). You can specify otherwise with:

stoch = talib.EMA(prices[context.sid], timeperiod=14)  

P.

All,

Regarding a new sample algorithm (or set of sample algorithms), it'd be worth considering an illustration of simple periodic re-balancing of a long-only portfolio with no margin. I've attached an example, which maintains a 50/50 mix of SPY/BND (ideas for improvement welcomed). This is an investing strategy that should be familiar to most new Quantopian users (and I recall reading in a John Bogle book that for many investors, it would be optimum, versus constantly fiddling with the asset allocations and portfolio mix). Although it is basically passive investing (and as such, probably not a good match for Quantopian), conceptually, there is a much lower barrier to entry compared to an active trading algorithm. Additionally, if users have long-term portfolios (e.g. for retirement), they can straightaway model them with ETFs (and individual stocks, if applicable). The other advantage is that it provides a starting point for understanding if active trading could improve the risk/return profile of their overall portfolio (e.g. assessing how much, if any, of a passively managed portfolio should be allocated to active management via a Quantopian algorithm).

Grant

Hi Grant,

First, let me warn you: I'm a total newbie to all of this. I am not a quant and I don't have any finance background. I'm just a programmer (with some math background) who suddenly got interested in the stock market. I started reading a little bit about how to trade and be successful (in my spare time, which is pretty rare unfortunately) and I started trading a little. Then I had this idea that I could possibly build an algorithm that would trade for me. I did a quick search on Google and I stumbled uppon Mr Peterffy's story (which you probably know). Well that was it, I am now officially hooked and very excited to start building my own algorithm!

I know, I'm probably very naive... forgive me

So after quite a bit of research on open source/free libraries or software to build a trading algorithm, I finally found zipline and Quantopian. This is amazing. Well done Quantopian!

OK, so enough context, here's my questions:

First, I looked at your algo and I don't understand why you execute this in a loop:

for stock, percent in zip(context.stocks,context.percent):  
            order_target_percent(stock, percent)  

What I don't understand is that, unless I'm wrong (which is probably the case), on the first iteration you will get your 50/50 mix of SPY/BND but then every other time that your modulo condition will be satisfied, nothing will happen because you will already have your 50/50 ratio. Right? What am I missing?

Also, how much money does your algo invest? Why don't I see anything about that in your algo? Is it normal that you don't define how much money you want to invest? I see $25000 at the top of the right pane. Does that mean that your algo buys for $25000 of SPY and BND (50/50) every single time it executes "order_target_percent"? If this is the case then I see how your algo would do something every time your modulo condition is satisfied, but that doesn't seem very realistic to me, at least not for a guy like me who has a very limited amount of money to play with (once it is all invested, there's nothing else I can do but look at it and hope that my profit goes up. I have to sell some stocks to be able to buy other stocks).

OK, so I just went and played a little more with your algo (and I added a couple 'print' statements) and I think I understand better. Please confirm that I get it right this time.

So you start off with $25000 and you will not invest more money than that. On the first day you will invest all this money to buy a 50/50 mix of SPY/BND. That doesn't mean the same amount of shares for both. It means that you more or less invest $12500 in SPY and $12500 in BND. You won't spend a total of exactly $25000 so you will have some money left in cash. Not much though. Not enough to buy 1 more share of the most expensive stock between SPY and BND and x shares of the other stock (where x is the number of shares needed to keep your 50/50 ratio).

Then 63 trading days later, the value of your stocks will have obviously changed and you will most probably not have a 50/50 ratio any more so you will readjust your ratio by selling and buying shares of SPY and BND to bring your ratio back to 50/50. And you will not invest any more money than what you had left in cash from the previous iteration (plus the money you would get from selling shares of SPY or BND).

And then you just repeat this over and over. Is that right?

Sorry for the long post, I guess you can see that I am very new to all this.

Thanks a lot!

Cheers!

Christian

Hello Christian,

You've moved up the learning curve a bit--congrats! Your description of the algo I posted above sounds correct.

By the way, I'm not an active trader. So far, I just tinker around on Quantopian as a challenging hobby and to learn Python.

Grant

Great! Thanks for your quick reply!

Glad to see I'm not alone in this boat! I have to learn EVERYTHING, including Python. Any advice on where to start and what to read/watch? I already have a long list of books that I'd like to read when time permits but they all are about the stock market in general and not about how to build a successful trading algorithm. I'm currently reading Benjamin Graham's "The Intelligent Investor" but from what I've read so far it seems like this guy doesn't really believe in day/swing trading...

Anyhow, I have a whole world to discover and I'm pretty excited! But to be honest, sometimes it seems like there's so much out there that I don't even know where to start.

Any luck so far with your algorithms?

Many thanks!

Christian

PS: Should we start a new thread? Something like "Where to start for absolute beginners". I don't think I could really be of any help unfortunately, but I'd be more than happy to read every posts and comment when I can.

Christian,

This thread is pretty extensive:

https://www.quantopian.com/posts/trading-strategy-ideas-thread

Regarding a new thread, feel free to start one.

I have had some success with algorithms, but nothing that "prints money" yet! But, as I said, it is just a hobby at this point. Currently, I'm working on code based on:

Bozdog, Dragos and Florescu, Ionut and Khashanah, Khaldoun and Wang, Jim, Rare Events Analysis
of High-Frequency Equity Data (July 1, 2011). Wilmott Journal, pp. 74-81, 2011. Available at
SSRN: http://ssrn.com/abstract=2013355

Grant

Thanks Grant,

I'll read all that and maybe start a new thread if I still feel the need afterwards.

Christian

I'm a civil engineer working on a traffic control system. I want to develop an algorithm that is adaptive and the format will look as follows: If A1, A2, A3: G
then B1: G until A1, A2, A3: Y
If only A3 not detected at B(z, y, x), then A3: Y
then compare queue at C(x, y, z) and B(x, y, z)
{ if C(x, y, z) > B(x, y, z)
then C1, C2: G until A1and A2: Y
else C1, B1: G until A1 and A2: Y
} then B1, B2, B3 start G
If only A2 not detected at C(z, y, x), then A2: Y
{ then D1, C1: G until A1and A2: Y
} then B1, B2, B3 start G
If A1 not detected at D(z, y, x) AND
If A2 not detected at C(z, y, x), then A1, A2: Y
{ then D1, C3: G until A3: Y
} then B1, B2, B3 start G
If A2 not detected at C(z, y, x) AND
If A3 not detected at B(z, y, x) then A2, A3: Y
then compare queue at C(x, y, z), B(x, y, z) and
D(x, y, z)
{ if B(x, y, z) > C(x, y, z) and D(x, y, z)
then B3, B1:G until A1:Y
if C(x, y, z) > D(x, y, z) and B(x, y, z)
then C1, C2:G until A1:Y
if D(x, y, z) > B(x, y, z) and C(x, y, z)
then D1, D2, D3:G until A1:Y
} then B1,B2,B3 start G
If A1, A2, A3 not detected at A(x, y, z)
then B1, B2, B3 start G
Please how do i get it right such that it can be stimulated