also i see required variables such as data_frequency are not exposed to the quantopian script. so it seems not possible .....

Use the following after checking out the code from github:

# -*- coding: utf-8 -*-  
"""
Created on Thu Mar 20 12:17:18 2014

@author: Suminda  
"""

# Delete unused imports once algo is finished

from datetime import datetime

from zipline import *  
from zipline.algorithm import *  
from zipline.api import *  
from zipline.api import *  
from zipline.data import *  
from zipline.errors import *  
from zipline.finance import *  
from zipline.gens import *  
from zipline.protocol import *  
from zipline.sources import *  
from zipline.transforms import *  
from zipline.utils import *  
from zipline.version import *

class commission:  
    def __init__(self):  
        self.PerShare = PerShare  
        self.PerTrade = PerTrade

commission = commission()

class slippage:  
    def __init__(self):  
        self.VolumeShareSlippage = VolumeShareSlippage  
        self.FixedSlippage = FixedSlippage

slippage = slippage()

# Logging. Following imports are not approved in Quantopian  
####################################################################################

import logbook  
import sys

log_format = "{record.extra[algo_dt]}  {record.message}"

zipline_logging = logbook.NestedSetup([  
    logbook.StreamHandler(sys.stdout, level=logbook.INFO, format_string=log_format),  
    logbook.StreamHandler(sys.stdout, level=logbook.DEBUG, format_string=log_format),  
    logbook.StreamHandler(sys.stdout, level=logbook.WARNING, format_string=log_format),  
    logbook.StreamHandler(sys.stdout, level=logbook.NOTICE, format_string=log_format),  
    logbook.StreamHandler(sys.stderr, level=logbook.ERROR, format_string=log_format),  
])
zipline_logging.push_application()

log = logbook.Logger('Main Logger')

# Cut and past between the ### line below to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)  
####################################################################################

# Place Quantopian approved imports here

def initialize(context):  
    pass


def handle_date(context, data):  
    order(symbol('AAPL'), 10)


####################################################################################  
# Cut and past between the ### line above to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)

if __name__ == '__main__':  
    import pylab as pl  
    # Any other non approved imports to make things work or to plot results  
    start = datetime(2008, 1, 1, 0, 0, 0, 0, pytz.utc)  
    end = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc)  
    # Reference any stocks data you need below  
    data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,  
                           end=end)  
    data = data.dropna()  

    algo = TradingAlgorithm(initialize=initialize,  
                            handle_data=handle_date)  

    results = algo.run(data)  
    results.portfolio_value.plot()  

    pl.show()  

You need to add following patch to api.py file:

class sid:  
    """Default sid lookup for any source that directly maps the  
    sid a symbol.  
    """  
    def __init__(self, sid_to_symbol):  
        """Initialise the maping of sids to symbols  
        :Arguments:  
            sid_to_symbol : dict  
                Contains the maping  
        """  
        self.sid_to_symbol = sid_to_symbol  
    def __call__(self, sid_num):  
        """Retrive the symbol corresponding to the sid  
        :Arguments:  
            sid_num : int  
                sid for which the symbol needs to be retrived  
        :Returns:  
            Mapped symbol  
        """  
        return self.sid_to_symbol[sid_num]

__all__ = [  
    'sid',  
    'symbol',  
    'slippage',  
    'commission',  
    'math_utils',  
    'batch_transform',  
    'FixedSlippage',  
    'VolumeShareSlippage'  
]

Following improvements are needed:

• symbol and sid need to return a security object. Implication is that you cannot call any of the security objects methods as this is represented as a string.

Hope the above helps.

if you want to save the results use replacing last part of the file above with this.

####################################################################################  
# Cut and past between the ### line above to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)

if __name__ == '__main__':  
    import pylab as pl  
    from pandas import ExcelWriter  
    import matplotlib.pyplot as plt

    # Any other non approved imports to make things work or to plot results

    start = datetime(1998, 12, 31, 0, 0, 0, 0, pytz.utc)  
    end = datetime(2014, 3, 22, 0, 0, 0, 0, pytz.utc)

    # Reference any stocks data you need below  
    data = load_from_yahoo(stocks=['SPY', 'XLY', 'XLP', 'XLE', 'XLF', 'XLV', 'XLI', 'XLB', 'XLK', 'XLU'], indexes={}, start=start,  
                           end=end)  
    data = data.dropna()  
    algo = TradingAlgorithm(initialize=initialize,  
                            handle_data=handle_data, capital_base=10000, instant_fill=True)

    results = algo.run(data)

    results.to_csv('results.csv')  

NB: This does not work with Zipline version 0.6 so you have to get the latest from Github.

Complete api.py file is below so you might not make a mistake:

#  
# Copyright 2014 Quantopian, Inc.  
#  
# Licensed under the Apache License, Version 2.0 (the "License");  
# you may not use this file except in compliance with the License.  
# You may obtain a copy of the License at  
#  
#     http://www.apache.org/licenses/LICENSE-2.0  
#  
# Unless required by applicable law or agreed to in writing, software  
# distributed under the License is distributed on an "AS IS" BASIS,  
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
# See the License for the specific language governing permissions and  
# limitations under the License.

# Note that part of the API is implemented in TradingAlgorithm as  
# methods (e.g. order). These are added to this namespace via the  
# decorator `api_methods` inside of algorithm.py.

import zipline  
from .finance import (commission, slippage)  
from .utils import math_utils


from zipline.finance.slippage import (  
    FixedSlippage,  
    VolumeShareSlippage,  
)


batch_transform = zipline.transforms.BatchTransform


def symbol(symbol_str, as_of_date=None):  
    """Default symbol lookup for any source that directly maps the  
    symbol to the identifier (e.g. yahoo finance).

    Keyword argument as_of_date is ignored.  
    """  
    return symbol_str


class sid:  
    """Default sid lookup for any source that directly maps the  
    sid a symbol.  
    """  
    def __init__(self, sid_to_symbol):  
        """Initialise the maping of sids to symbols  
        :Arguments:  
            sid_to_symbol : dict  
                Contains the maping  
        """  
        self.sid_to_symbol = sid_to_symbol  
    def __call__(self, sid_num):  
        """Retrive the symbol corresponding to the sid  
        :Arguments:  
            sid_num : int  
                sid for which the symbol needs to be retrived  
        :Returns:  
            Mapped symbol  
        """  
        return self.sid_to_symbol[sid_num]

__all__ = [  
    'sid',  
    'symbol',  
    'slippage',  
    'commission',  
    'math_utils',  
    'batch_transform',  
    'FixedSlippage',  
    'VolumeShareSlippage'  
]

If you want to use sid you have do the following at the top of your script file:

sid = sid({24 : 'AAPL'})  

then you can use sid as usual as long you give the mapping for all the symbols you like above.

Another important thing to remember is that you have to have the Yahoo ticker list for all the tickers you would be using. (This appears towards the bottom of the file.)

data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,  
                           end=end)  

thanks Suminda! I actually spent the last 3 hours writing a framework that should work in both places. I just got it working in quantopian, so now I need to get zipline operational to test with. after that, I'll make a github project for it.

the main difference with your code is i'm a python noob so I didn't think about wild-card importing, and i wrote shims for things, plus a new "Framework" class as a facade to abstract all the platform shims/glue away from the user's algorithm.

though i'm doing this for use in visual studio, so it's probably not very 'pythonic'

anyway, i'll post it once i get it tested under zipline.

trying to read your code Suminda... maybe too pythonic for me :P like i don't understand

sid = sid({24 : 'AAPL'})  

wouldn't I want to use the context.mySid=sid(24) ?

i also like how you pass the initialize and handle_data methods to the framework. i'll have to do that too.

In you python script you have to use

... sid(24)

but Zipline does not know about sids so you have give a mapping before this will work. So when you define the sid object you have to pass the mapping to its constructor.

yeah i'm running into that problem now. also i see that the yahoo data loader is returning the data as a single dataframe. pretty disappointing to see all these discrepancies.

i guess it's working now.. no need to do any code changes to switch to/from quantopian.

fyi i circumvented the whole sid problem by just using sid to be a string on zipline, and int on quantopian. everything still works.

i'll put this on github in the morning. honestly going through all this code makes me either hate zipline's codebase, or python in general. not sure which. I probably won't continue improving this. it's just too retarded how the runtimes are so different.

Thanks for all this Suminda! It will be a big help for setting up a decent local env.

actually not sure there's interest in a cross-plat framework anyway. so I'll paste my current framework below. on this github will be my continued work, but i'll be abandoning zipline use because of it's various differences/defects.

current code that works on both is:

# Import the libraries we will use here  
import datetime  
import pytz  
import math  
import numpy  
import pandas  
import scipy  
import scipy.stats  
import zipline

#from zipline import *  
#from zipline.algorithm import *  
##from zipline.api import *  
#from zipline.data import *  
#from zipline.errors import *  
#from zipline.finance import *  
#from zipline.gens import *  
#from zipline.protocol import *  
#from zipline.sources import *  
#from zipline.transforms import *  
#from zipline.utils import *  
#from zipline.version import *





#global constants  
global true  
true = 1  
global false  
false = 0




#quantopian shims  
class Shims():  
    class Context():  
        def __init__(this , portfolio = zipline.protocol.Portfolio()): #, tradingAlgorithm = zipline.TradingAlgorithm()):  
            this.portfolio = portfolio  
            #this.tradingAlgorithm = tradingAlgorithm  
            pass  
        pass



    class _Logger():  
        '''shim for exposing the same logging definitions to visualstudio intelisence'''  
        #def __init__(this):  
        #   pass    

        def error(this, message):  
            print("{0} !!ERROR!! {1}".format(this.framework.get_datetime(),message))  
            pass  
        def info(this, message):  
            print("{0} info {1}".format(this.framework.get_datetime(),message))  
            pass  
        def warn(this, message):  
            print("{0} WARN {1}".format(this.framework.get_datetime(),message))  
            pass  
        def debug(this, message):  
            print("{0} debug {1}".format(this.framework.get_datetime(),message))  
            pass  
        pass



    class _TradingAlgorithm_QuantopianShim:  
        '''shim of zipline.TradingAlgorithm for use on quantopian '''  
        def __init__(this):  
            #this.logger = Shims._Logger()  
            #this.logger = log  
            pass  


        def order(this,sid,amount,limit_price=None, stop_price=None):  
            '''  
            Places an order for the specified security of the specified number of shares. Order type is inferred from the parameters used. If only sid and amount are used as parameters, the order is placed as a market order.  
            Parameters  
            sid: A security object.  
            amount: The integer amount of shares. Positive means buy, negative means sell.  
            limit_price: (optional) The price at which the limit order becomes active. If used with stop_price, the price where the limit order becomes active after stop_price is reached.  
            stop_price: (optional) The price at which the order converts to a market order. If used with limit_price, the price where the order converts to a limit order.  
            Returns  
            An order id.  
            '''  
            sec = this.context.framework.targetedSecurities[sid]  
            #this.logger.info(sec)  
            order(sec,amount,limit_price,stop_price)  
            pass

        def order_percent(self, sid, percent, limit_price=None, stop_price=None):  
            """  
            Place an order in the specified security corresponding to the given  
            percent of the current portfolio value.

            Note that percent must expressed as a decimal (0.50 means 50\%).  
            """  
            value = self.context.portfolio.portfolio_value * percent  
            return self.order_value(sid, value, limit_price, stop_price)

        def order_target(self, sid, target, limit_price=None, stop_price=None):  
            """  
            Place an order to adjust a position to a target number of shares. If  
            the position doesn't already exist, this is equivalent to placing a new  
            order. If the position does exist, this is equivalent to placing an  
            order for the difference between the target number of shares and the  
            current number of shares.  
            """  
            if sid in self.context.portfolio.positions:  
                current_position = self.context.portfolio.positions[sid].amount  
                req_shares = target - current_position  
                return self.order(sid, req_shares, limit_price, stop_price)  
            else:  
                return self.order(sid, target, limit_price, stop_price)

        def order_target_value(self, sid, target, limit_price=None,  
                               stop_price=None):  
            """  
            Place an order to adjust a position to a target value. If  
            the position doesn't already exist, this is equivalent to placing a new  
            order. If the position does exist, this is equivalent to placing an  
            order for the difference between the target value and the  
            current value.  
            """  
            if sid in self.context.portfolio.positions:  
                current_position = self.context.portfolio.positions[sid].amount  
                current_price = self.context.portfolio.positions[sid].last_sale_price  
                current_value = current_position * current_price  
                req_value = target - current_value  
                return self.order_value(sid, req_value, limit_price, stop_price)  
            else:  
                return self.order_value(sid, target, limit_price, stop_price)

        def order_target_percent(self, sid, target, limit_price=None,  
                                 stop_price=None):  
            """  
            Place an order to adjust a position to a target percent of the  
            current portfolio value. If the position doesn't already exist, this is  
            equivalent to placing a new order. If the position does exist, this is  
            equivalent to placing an order for the difference between the target  
            percent and the current percent.

            Note that target must expressed as a decimal (0.50 means 50\%).  
            """  
            if sid in self.context.portfolio.positions:  
                current_position = self.context.portfolio.positions[sid].amount  
                current_price = self.context.portfolio.positions[sid].last_sale_price  
                current_value = current_position * current_price  
            else:  
                current_value = 0  
            target_value = self.context.portfolio.portfolio_value * target

            req_value = target_value - current_value  
            return self.order_value(sid, req_value, limit_price, stop_price)

        pass

    class _TradingAlgorithm_ZiplineShim(zipline.TradingAlgorithm):  
        '''auto-generates a context to use'''  
        def initialize(this):  
            #delay initialize until start of first handle-data, so our portfolio object is available  
            #this.__isInitialized = false;  
            this.context = Shims.Context()  
            this.context.tradingAlgorithm = this  
            #this.context.portfolio = this.portfolio  
            pass

        def handle_data(this,data):  
            this.context.portfolio = this.portfolio  
            #if not this.__isInitialized:  
            #    this.__isInitialized=true  
            #    this.context.portfolio=this.portfolio  
            this.context.framework.update_timestep(data)  
            pass  
        pass


class FrameworkBase():  
    def __init__(this, context, isOffline):  
        this.isOffline = isOffline  
        this.context = context  
        this.tradingAlgorithm = Shims._TradingAlgorithm_QuantopianShim() #prepopulate to allow intelisence  
        this.tradingAlgorithm = context.tradingAlgorithm  
        this.targetedSecurities = {}  
        this._targetedSecurityIds=[0]  
        del this._targetedSecurityIds[:]  
        if is_offline_Zipline:  
            this.logger = Shims._Logger()  
            this.logger.framework = this  
        else:  
            this.logger = log  
        pass  
    def initialize(this):  
        #do init here  
        if this.isOffline:  
            #passed to the run method  
            this._offlineZiplineData = this.abstract_loadDataOffline_DataFrames()  
            this._targetedSecurityIds = list(this._offlineZiplineData.columns.values)  
        else:  
            this._targetedSecurityIds = [sec.sid for sec in this.abstract_loadDataOnline_SecArray()]  
            #this.tradingAlgorithm.logger.info(len(this.securityIds))  
            #this.tradingAlgorithm.logger.info(this.securityIds)  
        pass

    def abstract_loadDataOffline_DataFrames(this):  
        '''return a pandas dataframes of securities, ex: using the zipline.utils.factory.load_from_yahoo method  
        these will be indexed in .securityId's for you to lookup in your abstract_handle_data(data)'''  
        return pandas.DataFrame()  
        pass  
    def abstract_loadDataOnline_SecArray(this):  
        '''return an array of securities, ex: [sid(123)]  
        these will be indexed in .securityId's for you to lookup in your abstract_handle_data(data)'''  
        return []  
        pass

    def abstract_update_timestep_handle_data(this,data=pandas.DataFrame()):

        '''find the securities you loaded by .securityId.  
        if the security isn't present in data  
        , it's temporally unavailable (not yet listed or already removed from the exchange)  
        '''  
        pass

    def update_timestep(this,data):  
        '''invoked by the tradingAlgorithm shim every update.  internally we will call abstract_update_timestep_handle_data()'''  
        this.targetedSecurities.clear()  
        for qsec in data:  
            if not this.isOffline:  
                sid = qsec.sid  
            else:  
                #if offline zipline, qsec is a string ex: "SPY"  
                sid = qsec;  
                qsec = data[qsec]

            if len(this._targetedSecurityIds)==0 or this._targetedSecurityIds.index(sid)>=0:  
                this.targetedSecurities[sid]=qsec  
        this.data = data  
        this.abstract_update_timestep_handle_data(data)  
    pass

    def get_datetime(this):  
        if is_offline_Zipline:  
            if len(this.targetedSecurities)==0:  
                return datetime.datetime.fromtimestamp(0,pytz.UTC)  
            else:  
                return this.targetedSecurities.values()[0].datetime  
        else:  
            return get_datetime()  
        pass

#entrypoints  
def initialize(context=Shims.Context()):  
    '''initialize method used when running on quantopian'''  
    context.tradingAlgorithm = Shims._TradingAlgorithm_QuantopianShim()  
    context.tradingAlgorithm.context = context  
    context.framework = constructFramework(context,false)  
    context.framework.initialize()

    pass

def handle_data(context=Shims.Context(),data=pandas.DataFrame()):  
    '''update method run every timestep on quantopian'''  
    context.framework.update_timestep(data)  
    pass

global constructFramework  
def initalize_zipline():  
    '''initialize method run when using zipline'''  
    tradingAlgorithm = Shims._TradingAlgorithm_ZiplineShim()  
    context = tradingAlgorithm.context;  
    context.framework = constructFramework(context,true)  
    context.framework.initialize()  
    tradingAlgorithm.run(context.framework._offlineZiplineData)  
    pass




##############  CROSS PLATFORM USERCODE BELOW.   EDIT BELOW THIS LINE

class ExampleAlgo(FrameworkBase):  
    def abstract_loadDataOffline_DataFrames(this):  
        '''only called when offline (zipline)'''  
        # Load data  
        start = datetime.datetime(2002, 1, 4, 0, 0, 0, 0, pytz.utc)  
        end = datetime.datetime(2002, 3, 1, 0, 0, 0, 0, pytz.utc)  
        data = zipline.utils.factory.load_from_yahoo(stocks=['SPY', 'XLY'], indexes={}, start=start,  
                           end=end, adjusted=False)  
        return data  
        pass  
    def abstract_loadDataOnline_SecArray(this):  
        '''only called when online (quantopian)'''  
        return [  
                sid(8554), # SPY S&P 500  
                ]  
        pass

    def abstract_update_timestep_handle_data(this, data = pandas.DataFrame()):  
        ''' order 1 share of the first security each timestep'''  
        if(len(this._targetedSecurityIds)>0):  
            this.logger.info("buy {0} x1".format(this._targetedSecurityIds[0]));  
            this.tradingAlgorithm.order(this._targetedSecurityIds[0],1)  
            pass  
        else:  
            this.logger.info("no security found this timestep");  
    pass  

##############  CONFIGURATION BELOW

def constructFramework(context,isOffline):  
    '''factory method to return your custom framework/trading algo'''  
    return ExampleAlgo(context,isOffline);

############## OFLINE RUNNER BELOW.  EDIT ABOVE THIS LINE  
is_offline_Zipline = false  
if __name__ == '__main__':  
    #import pylab  
    is_offline_Zipline = true

if(is_offline_Zipline):  
    initalize_zipline()  
    pass

fyi the github repo has an improved (but quantopian only) version of my framework now. I need this as an intraday/live trading framework, so I'll be continually updating it until it's on par with my previous (interday only) framework.

@Jason: Nice work! But why can't you use zipline.api for this? Should reduce your code burden.

Hi Thomas, I didn't know such a thing existed (you replied acknowledging my rant about no docs) I searched online and I see this: https://github.com/quantopian/zipline/blob/master/zipline/api.py but it's not obvious to me how that would help? Could you please elaborate? Sorry I'm not a python person so I might be missing something obvious to the PEP inclined.

also fyi during my search just now I found what appear to be the built html version of the docs: http://zipline.readthedocs.org/en/latest/ Is that reasonablly up to date?

it looks like with zipline.api I should be able to add this to the offline-zipline version, is this correct?

import zipline.api  
symbol("SPY")  
#etc for the quantopian slippage/commision/etc global methods  

Unfortunately the readthedocs is not up-to-date. We'll add new docs to zipline.io.

zipline.api is work in progress so that's why it's not advertised but the idea is to be able to write Quantopian algos that can be run on zipline (and vice versa). Thus if you do (although explicit imports are to be preferred in Python):

from zipline.api import *  

You should get some of the functionality like the order methods, symbol() etc (if your zipline is from github master). See here for a simple example of instantiating an algorithm like that:
https://github.com/quantopian/zipline/blob/master/zipline/examples/quantopian_buy_apple.py

ok thank you! I'll work on getting the project zipline compatable again after I finish adding features.

Check out our new blog post on this topic:
http://blog.quantopian.com/unifying-zipline-quantopian/

OK Great

BTW, are the docs updated for the IPython Magic? (Pl. assume no knowledge in Python, IPython, Zipline, Quantopian)

Not yet, but they will be for the 0.6.2 release.

OK.

Another thing is we are in no man's land since the batch_trasform is deprecated and buggy and history is not yet fully functional and implementation is quaky and does not support minute mode at least. When writing an algo for live trading without backfilling it will not run immediately.