In daily mode, your order is executed at the next day's closing price. Your code doesn't include the logging that generated the output in your post, but I suspect that the prices that look wrong to you are the prices on the order date and (one day later) the execution date. In other words, you're probably looking at returns on the day after you place your order, whereas you made your buy/sell decision based on the returns on the day of your order.

Thanks Michael. Is there any reliable way to solve this and actually compare current price to yesterday's at the same time of day?

And, just to understand better, is returns() buffering by a day due to system restrictions or purposely (for a purpose I'm simply missing)? Thanks!

If you use

prices = history(2, '1d', 'close_price')  

prices[0] = yesterday close
prices[1] = In minute mode, this will change to the most recent price for a given security per bar (aka per each handle_data() execution).

Could you elaborate on "actually compare current price to yesterday's at the same time of day?" ?

I may be able to help, just need some more clarification from you.

Greg

Thanks Greg,

Yes, I originally tried using history but then switched to returns() because I thought it was providing a simpler way to get the same thing (I was incorrect). I can't seem to find how to apply history() to a specific security, but I may just be missing it in the API docs. I'll look further.

It seems like your solution will work for me...if it will compare yesterday's close to the most recent price available. That functionality will negate my need for any sort of "same time of day" comparison.

I'll give it a shot. Thanks!

To apply history for a specific stock, you can use panda's indexing: http://pandas.pydata.org/pandas-docs/stable/indexing.html

Using Greg's example to get AAPL's prices,

prices_aapl = history(2, '1d', 'close_price')[symbol('AAPL')]  

Thanks Alisa! Perfect solution.

To compare yesterday's close to the most recent price available, you can use either returns() or the prices from history(). The attached backtest uses both methods, and they give identical results (the first % in each line is from returns(), the second uses prices from history()):

2015-01-13handle_data:28INFO2015-01-13 14:00 AAPL 110.021,  0.7% [ 109.26 -> 110.021:  0.7%]  
2015-01-13handle_data:28INFO2015-01-13 15:00 AAPL  109.74,  0.4% [ 109.26 ->  109.74:  0.4%]  
2015-01-13handle_data:28INFO2015-01-13 16:00 AAPL 110.225,  0.9% [ 109.26 -> 110.225:  0.9%]  
2015-01-14handle_data:28INFO2015-01-14 14:00 AAPL 108.927, -1.2% [110.225 -> 108.927: -1.2%]  
2015-01-14handle_data:28INFO2015-01-14 15:00 AAPL 109.091, -1.0% [110.225 -> 109.091: -1.0%]  
2015-01-14handle_data:28INFO2015-01-14 16:00 AAPL  109.79, -0.4% [110.225 ->  109.79: -0.4%]  
2015-01-15handle_data:28INFO2015-01-15 14:00 AAPL   107.1, -2.5% [ 109.79 ->   107.1: -2.5%]  
2015-01-15handle_data:28INFO2015-01-15 15:00 AAPL  107.75, -1.9% [ 109.79 ->  107.75: -1.9%]  
2015-01-15handle_data:28INFO2015-01-15 16:00 AAPL  106.83, -2.7% [ 109.79 ->  106.83: -2.7%]  

Whichever method you use, remember that if you run in daily mode, your order won't execute until the next day -- and the returns for that day will be different from the returns on the day you decided to place your order. So if I understand your intent correctly, you'll probably want to run in minute mode. Then if you examine returns and place an order before market close, the order will usually execute one minute later. Instead of a full day's delay, you'll only have one minute delay and not much change in returns.

Thanks Michael. That makes sense.

Matthew I to am new to Quantopian and I found reading through you code very helpful to my understanding, and after looking at you initial_portfolio_population function I did one change to how it buys stocks initially, and then ran the same Backtest you did which had an interesting result, all I did was have the program only buy stocks if they are lower than usual a.k.a buy low sell high. and had a large difference in performance. As a *new user I would like to know why that happened.

*Quantopian user for two-three days

Hi Tim. I haven't messed with mavg() yet but I see your point in making sure to avoid buying in a peak by pure chance. I'll try it out. Thanks!

No Problem actually I should thank you, I learned a great amount about Quantopian because of just reading through this and testing new things out.

Glad I could help, though I'm brand new myself. Hey, a thought--the way you had it avoid high priced stocks happens after establishing the amount of money to dedicate to each stock. If you set that logic earlier, you'll be in a better place to distribute properly. Something like this:

        # Figure out how much money you're dedicating to each stock  
        buy_count = 0  
        for this_stock in context.stocks_allowed:  
            this_stocks_data = data[this_stock]  
            # Make sure not to buy at the stock's peak  
            average_price = this_stocks_data.mavg(2)  
            if average_price > this_stocks_data.price:  
                buy_count = buy_count + 1

        # Buy an equal amount of each stock  
        amount_per_stock = (context.portfolio.cash * (float(context.initial_buy_pct) * 0.01)) / buy_count  
        for this_stock in context.stocks_allowed:  
            this_stocks_data = data[this_stock]  
            average_price = this_stocks_data.mavg(2)  
            if average_price > this_stocks_data.price:  
                num_of_shares = math.floor(amount_per_stock / this_stocks_data.price)  
                order(this_stock, +num_of_shares)  

that is very interesting, I am working on a version now. were would this go? Because I wiped the initial_portfolio_population function and put this in but when I ran a back test it tanked, something like -3,000 percent so I get the concept and am working on a version but how do I incorporate this specifically, or is this just and example of a possibility

I edited the above to correct a mistake. I'm just taking what you did and figuring out the $ amount to dedicate to each stock beforehand. See below for it in context.

import math

##################################################################################  
##################################################################################

def initialize(context):

    # Define all stocks involved in this algorithm (by sid)  
    context.stocks_allowed = [sid(22418), sid(45620), sid(39227), sid(21481), sid(29686), sid(14596), sid(21413), sid(15306), sid(2214), sid(32331), sid(32781), sid(21203), sid(21724), sid(677), sid(33989), sid(34443), sid(18184), sid(41522)]

    # This determines whether this is the first time these stocks have been purchased  
    context.initial_purchase_flag = 0

    # This array stores the returns of each stock to purchase as well as the symbol of each. It is used to determine the volume of each stock to buy  
    context.curve_index = []

    # The min amount of settled cash available to allow stock purchases to take place  
    context.min_account_balance_to_buy = 1000

    # Of all the stocks purchased, what pct (total) do you want to decrease the volume of?  
    context.exterior_pct_to_deprioritize = 40

    # How drastic is the difference between the prioritized and deprioritized stocks?  
    context.curve_delta_pct = 80

    # How much of total cash should be used to buy stocks upon first load  
    context.initial_buy_pct = 50

    # No point in applying a curve to a few stocks. Define a min number to apply a curve to.  
    context.min_num_of_stocks_to_curve = 5  
    # Define notional limits  
    context.max_notional = 1000000.1  
    context.min_notional = -1000000.0

    # Not yet implemented: num of days after which to sell a stock that's been sitting in your portfolio, regardless of performance  
    context.age_out_days = 5 

    # Define the criteria by which the algorithm will purchase or sell a given security (% of change up/down)  
    context.pct_change_to_sell = 0.5  
    context.pct_change_to_buy = 0.5

    # Define the % by which a stock would have to decrease in order to "cut losses" and sell it (to avoid total loss from a diving stock)  
    context.max_decrease_pct = 4

    # Don't fully understand this, but was told to do this in order to get "yesterday's closing price" for a given security  
    set_universe(universe.DollarVolumeUniverse(90.0, 90.1))

##################################################################################  
##################################################################################

# handle_data broken into a series of synchronous functions  
def handle_data(context, data):  
    # Only execute trades if there are currently no open orders  
    if not get_open_orders():  
        initial_portfolio_population(context, data)  
        sell_stocks(context, data)  
        buy_stocks(context, data)

##################################################################################  
##################################################################################

def initial_portfolio_population(context, data):  
    if len(context.portfolio.positions) == 0:  
        # Make sure you don't immediately sell these by accident  
        context.initial_purchase_flag = 1

        # Figure out how much money you're dedicating to each stock  
        buy_count = 0  
        for this_stock in context.stocks_allowed:  
            this_stocks_data = data[this_stock]  
            # Make sure not to buy at the stock's peak  
            average_price = this_stocks_data.mavg(2)  
            if average_price > this_stocks_data.price:  
                buy_count = buy_count + 1

        # Buy an equal amount of each stock  
        amount_per_stock = (context.portfolio.cash * (float(context.initial_buy_pct) * 0.01)) / buy_count  
        for this_stock in context.stocks_allowed:  
            this_stocks_data = data[this_stock]  
            average_price = this_stocks_data.mavg(2)  
            if average_price > this_stocks_data.price:  
                num_of_shares = math.floor(amount_per_stock / this_stocks_data.price)  
                order(this_stock, +num_of_shares)

##################################################################################  
##################################################################################

def sell_stocks(context, data):

    # First, ensure that the portfolio contains one or more securities (otherwise, there's nothing to sell and this function becomes pointless)  
    if (len(context.portfolio.positions) != 0) and (context.initial_purchase_flag == 0) and not get_open_orders():

        stocks_to_sell = []

        for this_stock in context.portfolio.positions:  
            # prices[0] is yesterday's close; prices[1] the most recent price for a given security per bar (aka per each handle_data() execution).  
            prices = history(2, '1d', 'close_price')[this_stock]  
            price_diff = prices[1] - prices[0]  
            price_diff_perc = float(price_diff / prices[0])

            # If it's increased in value enough, sell all shares (haven't yet added "age out" logic (context.age_out_days))  
            if (price_diff_perc > 0) and (price_diff_perc > (float(context.pct_change_to_sell) * 0.01)):  
                stocks_to_sell.append(this_stock)

        # Then actually sell them. You can't sell them inside the above loop because the dictionary size will change and error out.  
        for this_stock in stocks_to_sell:  
            order_target(this_stock, 0)

##################################################################################  
##################################################################################

def buy_stocks(context, data):

    if (context.initial_purchase_flag == 0) and (context.portfolio.cash > context.min_account_balance_to_buy) and not get_open_orders():

        # Establish list of stocks to purchase  
        stocks_to_purchase = []

        # Go through the allowed stocks list and determine which meet the criteria (which ones have decreased in price today)  
        for this_stock in context.stocks_allowed:  
            this_stocks_data = data[this_stock]  
            # prices[0] is yesterday's close; prices[1] the most recent price for a given security per bar (aka per each handle_data() execution).  
            prices = history(2, '1d', 'close_price')[this_stock]  
            price_diff = prices[1] - prices[0]  
            price_diff_perc = float(price_diff / prices[0])  
            if (price_diff_perc < 0) and (price_diff_perc < (float(context.pct_change_to_buy) * 0.01)) and (this_stock not in context.portfolio.positions):  
                # Don't buy anything that has dropped more than cmax_decrease_pct (avoiding diving stocks)  
                if (price_diff_perc > (0 - (float(context.max_decrease_pct) * 0.01))):  
                    stocks_to_purchase.append(this_stock)  
                    context.curve_index.append([price_diff_perc, this_stock.symbol])

        # Proceed with purchase logic if there's anything to purchase  
        if (len(stocks_to_purchase) != 0):

            # TO DO: Add logic here to randomly strip away number of stocks to purchase if there are too many  
            # Determine curve:  
            curve_nums = create_buy_curve(stocks_to_purchase, context)

            # Actually purchase the stocks  
            wallet_snapshot = context.portfolio.cash  
            i = 0  
            for this_stock in stocks_to_purchase:  
                this_stocks_data = data[this_stock]  
                num_of_shares = math.floor((wallet_snapshot * (curve_nums[i] * 0.01)) / this_stocks_data.price)  
                order(this_stock, +num_of_shares)  
                i = i + 1

    else:  
        context.initial_purchase_flag = 0

##################################################################################  
##################################################################################

def create_buy_curve(stocks_to_purchase, context):  
    context.stocks_to_purchase_sorted = []  
    context.curve_index_sorted = []

    # Sorting curve_index based on returns  
    context.curve_index_sorted = sorted(context.curve_index, key=lambda x: x[0])

    # Then go through each of the sorted indexes  
    for index in context.curve_index_sorted:  
        for this_stock in stocks_to_purchase:  
          if this_stock.symbol == index[1]:  
               context.stocks_to_purchase_sorted.append(this_stock)

    # You now have context.stocks_to_purchase_sorted. Time to figure out the num of stocks and develop a curve based on that  
    curve_nums = []

    total_num_of_stocks_to_purchase = len(context.stocks_to_purchase_sorted)  
    if (total_num_of_stocks_to_purchase != 0) and (total_num_of_stocks_to_purchase > context.min_num_of_stocks_to_curve):  
        # Determine respective %s  
        num_of_stocks_at_beginning_and_end_to_deprioritize = int(math.floor(((context.exterior_pct_to_deprioritize * 0.01) / 2) * total_num_of_stocks_to_purchase))  
        num_of_stocks_in_middle_to_prioritize = int(total_num_of_stocks_to_purchase - (num_of_stocks_at_beginning_and_end_to_deprioritize * 2))  
        smaller_pct = int(((100 - context.curve_delta_pct) * 0.01) * (100 / (num_of_stocks_at_beginning_and_end_to_deprioritize * 2)))  
        larger_pct = int((context.curve_delta_pct * 0.01) * (100 / num_of_stocks_in_middle_to_prioritize))

        for i in range(num_of_stocks_at_beginning_and_end_to_deprioritize):  
            curve_nums.append(smaller_pct)  
        for i in range(num_of_stocks_in_middle_to_prioritize):  
            curve_nums.append(larger_pct)  
        for i in range(num_of_stocks_at_beginning_and_end_to_deprioritize):  
            curve_nums.append(smaller_pct)

    else:  
        for this_stock in context.stocks_to_purchase_sorted:  
            simple_pct = math.floor(len(context.stocks_to_purchase_sorted) / total_num_of_stocks_to_purchase)  
            curve_nums.append(simple_pct)

    return curve_nums

##################################################################################  
##################################################################################

okay how does this run? would be interesting to see, you gave me an Idea I am working on a version that the lower the stock the more money it will invest into that stock aka, one stock is 4$ below average and one is 2 dollars below average invest more proportionally to the one 4 under average

It performs poorly, to say the least haha. I'm going to work to refine it into something potentially usable once I fix all the problems I'm facing.

I'm using history to compare previous close to current and it's still buying after prices dropped, so I need to spend some time stepping through the breakpoints to determine how that's happening.