Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Porting an algorithm to research plus issue extending the position object using a dataframe (new to Python/Quantopian)
Steve Salemy
posted

Not only am I new to Python on Quantopian, this is a first post.

Below is a simple algorithm / trading system based on RSI2. It runs on a series of symbols / sids.

My questions are as follows:
1. I'd like to port this algorithm to run in research but have struggled doing so for several days.
2. In research, the list of symbols hard-coded in the algorithm are sitting in a flat-file in data (though it keeps disappearing :) ). Is there a simple way to read a universe of symbols / sids from a flat file maintained outside of quantopian?
3. For my uses, the Quantopian Position object is missing some essential data (eg: Date position was established for example, Tags to log other data to store when the position is entered, among others). I've tried extending the Position object by adding my own position object in context, Context.Positions -- which I tried to create as a pandas dataframe. The code is commented out in the attached code; it compiles but generates a runtime error (TypeError: 'DataFrame' objects are mutable, thus they cannot be hashed).

Any help with the above items would be greatly appreciated.

Thanks

# This is meant to be a toy algorithm to learn quantopian.  Changes I'm trying to learn to make are #  1) not hard-coding the sids but instead reading them from a flat-file.  This sid list was produced by running a program in RESEARCH that reads a flat file of symbols, converts them to sids and then outputs this list to be pasted into the code.  
#  2) I want to convert this simple algorithm into a backtest that I can run in RESEARCH;  I spent several days trying to understand how to do that directly and could not figure out how.  

import pandas as pd  
import numpy as np  
#import matplotlib.pyplot as plt  
import talib  
from zipline.api import get_environment

def initialize(context):  
    schedule_function(entry_exit_rules, date_rules.every_day(), time_rules.market_close())  
    set_commission(commission.PerShare(cost=0.005, min_trade_cost=1.00)) #IB commissions  
    set_slippage(slippage.FixedSlippage(spread=0.001))  #NOT USING SLIPPAGE RIGHT NOW  
    set_benchmark(symbol('SPY'))  
    context.Mode = "SSB"   #  Allowed == SSB: Single Symbol Backtest or MSB: Multi Symbol Backtest  
                           #  IF SSB, record single name dataseries if MSB only portfolio stats  
    context.ssid = [sid(19920)]  
    context.sids = [sid(25485), sid(41013), sid(25801), sid(32406), sid(35793), sid(23881)] #sid(24744), sid(32888), sid(21518), sid(42041), sid(21517), sid(45555), sid(21519), sid(41382), sid(44849), sid(26981), sid(39941), sid(33652), sid(32505), sid(33127), sid(35071), sid(28054), sid(34385), sid(19920), sid(46302), sid(35175), sid(34831), sid(35323), sid(42039), sid(24705), sid(27100), sid(41603), sid(14520), sid(25098), sid(14519), sid(8554), sid(14517), sid(23911), sid(14516), sid(14518), sid(27536), sid(21757), sid(21491), sid(22972), sid(14526), sid(14530), sid(14529), sid(21619), sid(41178), sid(40107), sid(21513), sid(27102), sid(21785), sid(23134), sid(25910), sid(34022), sid(26807), sid(34648), sid(21512), sid(33651), sid(38901), sid(12915), sid(38985), sid(33486), sid(23921), sid(35970), sid(41553), sid(43548), sid(38987), sid(33697), sid(35971), sid(26432), sid(23870), sid(43549), sid(33655), sid(28320), sid(42633), sid(2174), sid(46300), sid(33541), sid(22739), sid(33650), sid(42413), sid(35072), sid(28368), sid(39116), sid(25909), sid(26703), sid(42524), sid(38297), sid(48049), sid(21508), sid(32620), sid(21758), sid(33748), sid(34164), sid(21507), sid(39080)]

    if context.Mode == "SSB":  
        context.sids = context.ssid

    context.RsiPeriod = 2  
    context.OB = 100 #Overbought threshold  
    context.OS = 5 #Oversold threshold  
    context.ExitRSI = 50 #RSI Level to exit positions  
    context.MaxHoldDays = 5  
    context.TradeAmount = 100000.0  
    context.longs = 0  
    context.shorts = 0  
    context.netposn = 0  
    context.WinningTrades = 0  
    context.LosingTrades = 0  
    context.WinningLongTrades = 0  
    context.LosingLongTrades = 0  
    context.WinningShortTrades = 0  
    context.LosingShortTrades = 0  
    context.EndDate = get_environment('end').date()

    ids = []  
    for s in context.sids:  
        ids.append(s.sid)  
    context.Positions = pd.DataFrame(index=ids, columns=['ActivePosition','EnterDate','EnterPrice', 'Amount', 'Shares', 'ExitDate','ExitPrice', 'PL'])

    log.info(context.Positions.to_string())  
    #def handle_data(context, data):  
    #not implemented since nothing happens on a minute basis  
#def before_trading_start(context, data):  
    #not implemented since nothing needs to happen before the market opens each day

def calculate_trade_stats(context, cur_sid):  
    cur_posn = context.portfolio.positions[cur_sid]  
    if cur_posn.amount > 0:  # Long Posn  
        if cur_posn.last_sale_price >= cur_posn.cost_basis:  
            context.WinningTrades += 1  
            context.WinningLongTrades += 1  
        else:  
            context.LosingTrades += 1  
            context.LosingLongTrades += 1  
    else:   # Short Position  
        if cur_posn.last_sale_price <= cur_posn.cost_basis:  
            context.WinningTrades += 1  
            context.WinningShortTrades += 1  
        else:  
            context.LosingTrades += 1  
            context.LosingShortTrades += 1

def execute_order(cur_sid, OrderType, Amount, CalcTradeStats, context):  
    cur_posn = context.portfolio.positions[cur_sid].amount  
    s = cur_sid.sid  
    if cur_posn <> 0:  
        ExistingPosition = True  
        cur_price = context.portfolio.positions[cur_sid].last_sale_price  
        cost_basis = context.portfolio.positions[cur_sid].cost_basis  
    else:  
        ExistingPosition = False  
    if OrderType == "order_value":  
        order_value(cur_sid, Amount)  
    elif OrderType == "order_target_value":  
        order_target_value(cur_sid, Amount)  
    new_posn =  context.portfolio.positions[cur_sid].amount  
#    this code does not work;  i get a runtime error:  
#    "TypeError: 'DataFrame' objects are mutable, thus they cannot be hashed"  
#    I also tried using context.Positions.set_value(sid, 'ActivePosition', False)  
#    which also raised the same TypeError above  
#  
#    --------------------------------------------------------  
#    I want to create my own position object to track more data about a position than  
#    Quantopian tracks.  For example, Data position was initiated so that I can easily determine  
#    How many days I've held a position.  Also, I want to make it easier to sum up P&L  
#    --------------------------------------------------------  
#  
#    if new_posn == 0:  
#        context.Positions.loc[sid]['ActivePosition']=False  
#        context.Positions.loc[sid]['ExitDate']=get_datetime().date()  
#        context.Positions.loc[sid]['ExitPrice']=cur_price  
#        context.Positions.loc[sid]['PL']=(cur_price-cost_basis)*cur_posn  
#    else:  
#        context.Positions.loc[sid]['ActivePosition']=True  
#        context.Positions.loc[sid]['EnterDate']=get_datetime().date()  
#        context.Positions.loc[sid]['EnterPrice']=context.portfolio.positions[cur_sid].cost_basis  
#        context.Positions.loc[sid]['Shares']=new_posn  
#        context.Positions.loc[sid]['Amount']=new_posn * context.portfolio.positions[cur_sid].cost_basis  
#    log.info(str(context.Positions.ix['ActivePosition', sid]))  
    if CalcTradeStats == True:  
        if Amount > 0:  
            context.longs +=1  
            context.netposn +=1  
        elif Amount < 0:  
            context.shorts += 1  
            context.netposn -=1  
        else:  #Amount == 0 means to exit the position  
            if ExistingPosition and cur_posn > 0: #we're selling out of a long  
                context.longs -=1  
                context.netposn -=1  
                calculate_trade_stats(context, sid)  
            if ExistingPosition and cur_posn < 0: #we're selling out of a long  
                context.shorts -=1  
                context.netposn +=1  
                calculate_trade_stats(context, sid)  
def entry_exit_rules(context, data):    

    hist = data.history(context.sids, 'price', 3, '1d') # get 3 days of px history for universee  
#    rsis = talib.RSI(hist, context.rsi_period) <-- this doesn't work would be great if you could calculate technical indicators for your entire universe at once instead of looping  
    for cur_sid in context.sids:  # for each instrument in my universe  
        rsi_c = talib.RSI(hist[cur_sid], context.RsiPeriod)  
#  
#  DO WE HAVE A POSITION?  IF SO, APPLY EXIT RULES FIRST  
#  
        if context.portfolio.positions[cur_sid].amount > 0:  # EXISTING LONG POSITION  
#            DaysInPosition = context.portfolio.positions[cur_sid].  
            if rsi_c[-1] > context.ExitRSI:  
                execute_order(cur_sid, 'order_target_value', 0, True, context)

        if context.portfolio.positions[cur_sid].amount < 0:  # EXISTING SHORT POSITION  
            if rsi_c[-1] < context.ExitRSI:  
                execute_order(cur_sid, 'order_target_value', 0, True, context)  
#  
#  NOW CHECK ENTRY RULES  
#  
        if context.portfolio.positions[cur_sid].amount == 0:  # NO POSITION CHECK ENTRY RULES  
            if rsi_c[-1] < context.OS and data.can_trade(cur_sid):  # LONG ENTRY  
                execute_order(cur_sid, 'order_value', context.TradeAmount, True, context)

            elif rsi_c[-1] > context.OB and data.can_trade(cur_sid):    # SHORT ENTRY  
                execute_order(cur_sid, 'order_value', context.TradeAmount, True, context)

    record(long=context.longs, short=context.shorts, net=context.netposn)  
    if context.Mode == "SSB":  
        record(rsi=rsi_c[-1])

    if get_datetime().date() == context.EndDate:  
       log.info("Total Trades: " + str(context.WinningTrades + context.LosingTrades))  
       log.info("Total Winnning Trades: " + str(context.WinningTrades))  
       log.info("Total Losing Trades: " + str(context.LosingTrades))  
       log.info(" ")  
       log.info("Total Long Trades: " + str(context.WinningLongTrades + context.LosingLongTrades))  
       log.info("Total Winnning Long Trades: " + str(context.WinningLongTrades))  
       log.info("Total Losing Long Trades: " + str(context.LosingLongTrades))  
       log.info(" ")  
       log.info("Total Short Trades: " + str(context.WinningShortTrades + context.LosingShortTrades))  
       log.info("Total Winnning Short Trades: " + str(context.WinningShortTrades))  
       log.info("Total Losing Short Trades: " + str(context.LosingShortTrades))
1 response
Ernesto Perez

Hi Steve,

The Backtesting with Zipline notebook in the Tutorials and Documentation folder (inside the research environment) has a simple example of how to backtest an algorithm in research. In general, we recommend using the IDE environment for backtesting instead.

It is possible to import a list of securities in a csv file using the fetch_csv function, but it requires a bit of work. I would recommend you to check out the fetch_csv documentation here. Specifically, you want to look at the section explaining how to import securities' data, and how data.fetcher_assets is used to reference the securities listed in the csv's symbol column.

Regarding the TypeError, it happened because sid is a built-in function in the platform, and you had not overwritten its value before trying to use it as a key to context.Positions.loc. I think you meant to use s, which you set to cur_sid.sid. In general, it is good practice to avoid using built-in function names as variables.

Additionally, I wanted to point out that lines 104 through 125 might not be behaving as you expect. Your algorithm is checking if a position exists on the same minute-bar in which it placed the order for it, so new_posn will always be 0 and the values you store in your Positions DataFrame will not be the ones you expect.

You would need to schedule two functions. First, one that handles your ordering logic, then a second one (allowing some time for orders to fill) that checks the order status and records values into your Positions DataFrame.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.