Spy on Fred - pipeline syntax help?

Back to Community

edited

Hello,
I've been lurking here for a while now, and I'm still learning some of the syntax.... I would like to use Quandl datasets that have monthly economic information (FRED unemployment rate data for example). It is marginally working from some of the nice examples and tutorials. The backtest runs, and plots the data I want to use, but still has some issues.

1: In order to use the dataset, it requires putting into a pipeline? Is there a way to use it directly in an algorithm? (the unrate data doesn't have an example algorithm, the attached backtest once it's working is one simple way it might be used) For this simple type of use, it's not for filtering stocks, more for algorithm decisions over time.

2: In the attached backtest, there must be a simpler way to get it into pipeline. the syntax suggested in documentation doesn't work: www.quantopian.com/data/quandl/fred_unrate pipe.add(fred_unrate.value.latest, 'fred_unrate')

  #How to store the value into pipeline???????  
    # unrate  = fred_unrate.value[-1:]  
  #running it through the SMA function works, crazy I know, but hey we're new here....  
    unrate = SimpleMovingAverage(inputs=[fred_unrate.value], window_length=1)  
    pipe.add(unrate, 'unrate')  
unrate_sma10 = SimpleMovingAverage(inputs=[fred_unrate.value], window_length=200)  
    pipe.add(unrate_sma10, 'unrate_sma10')                               #(unrate is monthly, 10mo ~= 200day)

3: Further down in the algorithm, I'd like to compare the value and the SMA of the value, both were stored in the pipeline. Referring to them this way works to plot them, but doesn't return the current value that could be compared

record(unrate=context.fred[:1].unrate,unratesma=context.fred[:1].unrate_sma10)

    unrate=context.fred[:1].unrate  
    unratesma=context.fred[:1].unrate_sma10

               # Sell all of our shares by setting the target position to zero  
  #Want to Do This -->  \/\/\/  
     #       if unrate>unratesma:           # Only if unemployment is increasing  
  #But I don't get how to reference the value from pipeline  
            order_target_percent(context.spy, 0)

Looking in the debugger, I see that these are BoundColumns data, but how do I get a value out of it?

Thanks! In Advance! this Quantopian thing is really cool by the way...

9 responses

Jamie McCorriston

Hi Dale,

The trick with the unemployment rate is that it's not a per-asset dataset. This means that you can't simply use .latest to get the latest factor. I'm attaching a backtest that has a CustomFactor that maps the latest unemployment to all equities. This means that our pipeline will output a copy the unemployment rate for each asset. I'll show this in a notebook right after. To get the actual value in your algo, you'll want to look at a single arbitrary row from the pipeline output, which you already do with the lines:

unrate=context.fred[:1].unrate  
unratesma=context.fred[:1].unrate_sma10

Right now, we don't have a good way of handling datasets that are not n x number_of_assets. Pipeline was initially designed to handle input data that is per-asset. The solution I offered works for now, but we hope to implement a better design for datasets of different dimensions going forward.

Let me know if this helps.

Note: The fact that this worked with SimpleMovingAverage was surprising, I think it wasn't an intentional part of the design that it worked, but it looks like you can use it the way you are for the time being ;)

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Jamie McCorriston

Here's the notebook that shows the unemployment rate being returned for every asset.

Disclaimer

Dale Klein

Thanks Jamie for looking at this and the insights. I see how it's stored in the pipeline now.

I think I'm still stuck on getting the value back out and use it. Uncommenting line 92 still doesn't run, unrate holds a series, not a value.

Ahah! The following works to retrieve the value stored in the pipeline. The final square brackets are required. Hey, trial and error is one of my key skills...

    unrate=context.fred[:1].unrate[0]  
    unratesma=context.fred[:1].unrate_sma10[0]

unrate=context.fred[:N].unrate[0] would also work, where N is an integer of any of the assets in the pipeline.

Jamie McCorriston

Hi Dale,

I'd recommend changing it to this:

unrate=context.fred.unrate.iloc[0]

That should just get you the 0th element in the context.fred.unrate Series.

Disclaimer

Dale Klein

Ahah! thanks. Appreciate the help Jamie.

Thomas Goldenberg

So I was able to get the fred_unrate data as suggested, but what about other fred data sources? I was unable to figure out how to get these into the pipeline in the same way. Here are some of the other data sets:

from quantopian.interactive.data.quandl import rateinf_inflation_usa # Inflation rate  
from quantopian.interactive.data.quandl import fred_gnp # GNP  
from quantopian.interactive.data.quandl import fred_gdp # GDP  
from quantopian.interactive.data.quandl import fred_icsa # Unemployment Claims  
from quantopian.interactive.data.quandl import adp_empl_sec # Total Jobs  
from quantopian.pipeline.data.quandl import fred_unrate # unemployment rate

Anyone have any suggestions on how to incorporate these data sources into pipeline? I would like to get the most current value of each of these when evaluating whether to buy a stock or not.

Thank you.

Sean Ferguson

Has this method been updated since this thread started? I'm interested in using fred_unrate data but not sure if attaching it to each asset in the Pipeline is still the most practical way or if it's been updated since 2017 as mentioned being in the works above. I haven't seen any other posts about it.

Dan Whitnable

@Sean Ferguson The method alluded to by Jamie is currently still the best approach to importing 'macro economic data' (such as the fred_unrate) or data which isn't directly associated with a specific security. One can think of the Quantopian database as a big dataframe. The rows are the securities and the columns are the data associated with each security. One can add custom data as columns in that dataframe but one cannot add rows or securities. All data must be tied to a security.

Disclaimer

Sean Ferguson

@Dan Whitnable

Just making sure. I am however running into some issues manipulating the unrate pipeline data through CustomFactors, but I'll open a new thread, it's a little different than what's going on here. Thanks for clarifying!

You've successfully submitted a support ticket.

Our support team will be in touch soon.