This is very exciting!

I am still encountering timeout issues with some of the 3rd party datasets (namely stock twits). Is there a timeline on a fix for this as well? I know something was in the works, but I thought it might have coincided with this announcement.

Thanks

Hi Ben,

I saw a PR for performance improvements go in this weekend, so that tells me progress is being made but these types of performance problems can prove to be nasty and not straight forward so I can't commit to a timeline. Generally these pieces of work are separate and will be delivered on their own timelines.

Thanks for hanging in there!
Josh

There are 20 fundamental data fields that are not supported in live
trading with pipeline because they are not properly adjusted. The
pipeline API uses data that is both split and dividend adjusted

... does this mean we will soon be able to get split/div adjusted data natively in quantopian, without having to rely on pipeline ?

Hi Florent,
Yes, we are working on delivering fully adjusted data in the backtester. I can't give an exact date for delivery, but it's in progress.

This might be a dumb question, but if shares_outstanding is impacted is market_cap impacted?

Awesome, finally a reason to play with the pipelines

@Matt, not a dumb question.

Shares outstanding are impacted because the share count itself changes.

In the case of market cap, the market cap doesn't itself change because it's the product of the shares outstanding and the price per share. So when the shares outstanding increase, the price per share decreases by the same proportion. There is no discontinuity the market cap value.

Hope this helps,
Josh

Does this mean shares outstanding from traditional fundamentals queries has also been wrong all this time? Or is this somehow a specific fault in pipeline?

Also, does this mean that all the example pipeline factors which calculate MarketCap as shares_outstanding * close are all wrong?

@Josh Makes sense. Thanks for the explanation.

Simon,

A couple of elements to answering your question:

1) This problem isn't apparent in the get_fundamentals() queries used in algorithms because that function returns "today's best known value" for the requested metric. It doesn't provide a time series response. So the day before a split, it provides "today's shares outstanding" and it's accurate and then the day after a split it provides "today's shares outstanding" and it is split adjusted and indeed accurate.

Pipeline is the first instance of this data being exposed in an algorithm beyond this initial use case of a time series being exposed in algos for this data so it became a problem in it's use in pipeline.

2) The historical data for time period from 2002 - 2014 for fundamentals is generated based on the split adjusted data provided to us by Morningstar on June 1, 2014. So the per share metrics and the shares outstanding are split adjusted as of June 1, 2014. One of the pieces of work ahead of us is to un-adjust this data in our construction of that historical time period. So this part of the problem is independent of it's use by pipeline.

3) We discovered that we're not processing the updated values from Morningstar in our processing since June 1, 2014. The data comes, but we have a bug that improperly delays the reflection of this new data point (sent to us as a correction). This bug is in the process of being fixed but further diminishes the accuracy of the data.

I wouldn't say the data is "all wrong". But if a company has had a split, the data is not reliable for that market cap calculation.

Given the workaround for the lack of USEquityPricing.open prices (i.e. using the prior day's close), some values could be significantly affected by a split when using raw data instead of split adjusted data.

For example, this example in the 101 Alphas Project chat has the following formula for alpha factor no. 101 which is supposed to be ((close - open) / ((high - low) + .001)):

out[:] = (close[-1] - close[-2]) / ((high[-1] - low[-1]) + .001)  

I guess I'm confused. The help page clearly states:

Our data uses adjusted close prices. That means that the effect of all stock splits and merger activity are applied to price and volume data. As an example: A stock is trading at $100, and has a 2:1 split. The new price is $50, and all past price data is retroactively updated 2:1 (volume data is correspondingly updated 1:2). In effect, that means you can ignore stock splits unless you are storing prices in a live trading algorithm.

Further down, under the section discussing dividends, we have:

While other corporate events, such as splits and mergers, are handled by adjusting historical prices and volumes...Dividend events modify the security price and the portfolio's cash balance.

But then we have some exceptions described above, and on https://www.quantopian.com/posts/the-pipeline-api-dividends-and-splits-what-you-need-to-know, where there is discussion of other fine points, such as:

The history function doesn't account for dividends at all, and reports a trailing window of fully split-adjusted prices.
We have an in-process project to change the data object to align with pipeline. The data object will become split and dividend adjusted to the date for which the object is being called.
Fundamentals data is not split or dividend adjusted.
Data brought into research through the store is not split or dividend adjusted by Quantopian.

Given the risk of users ending up with "garbage in garbage out" or wasting time trying to compare "apples to oranges" you might consider creating a definitive section of your help docs that tabulates all of this information, and keeps it up to date, including flagging bugs (and their expected resolution dates, if known). Personally, I find it challenging to get it all straight.

Generally, I'd recommend some sort of public-facing bug tracker, as well (perhaps you could leverage your presence on github, https://github.com/quantopian). I think you need to move beyond the practice of posting on forums, "Oh, by the way such-and-such is broken." It would be better to say, "The problem has been logged as bug #57 on our bug tracker." Then, information germane to the bug could be posted there, including temporary work-arounds and details on how the bug was resolved and the testing performed to show that the fix was effective.

Another thought is that you might somehow poll users regarding the readiness of features for real-money trading. It seems there should be a comment/vetting period, no? And maybe Quantopian would run real money first, and publish the results (I don't see any mention that you've ever run real money with pipeline)?

This appears OK regarding splits. For example, Apple had a 7-1 split on 2014-06-09, so it is a good test candidate.

from quantopian.pipeline import Pipeline  
from quantopian.pipeline import CustomFactor  
from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline.data.builtin import USEquityPricing as bar  
from quantopian.pipeline.factors import SimpleMovingAverage, Returns

class ReturnFactor(CustomFactor):  
    inputs = [bar.close]  
    window_length = 2

    def compute(self, today, assets, out, close):  
        out[:] = close[-1] / close[-2] - 1

def initialize(context):  
    pipe = Pipeline()  
    attach_pipeline(pipe, 'pipe')  
    returns = Returns(window_length=2)  
    returns_pipeline = ReturnFactor()  
    sma_1 = SimpleMovingAverage(inputs=[bar.close], window_length=1)  
    sma_5 = SimpleMovingAverage(inputs=[bar.close], window_length=5)  
    pipe.add(returns, 'returns')  
    pipe.add(returns_pipeline, "ReturnFactor")  
    pipe.add(sma_1, "SMA_1")  
    pipe.add(sma_5, "SMA_5")  
    context.aapl = sid(24)

def handle_data(context, data):  
    output = pipeline_output('pipe')  
    log.info("pipe aapl SMA1: {0:.2f}".format(output.loc[context.aapl].SMA_1))  
    log.info("pipe aapl SMA5: {0:.2f}".format(output.loc[context.aapl].SMA_5))  
    log.info("data aapl close: {0:.2f}".format(data[context.aapl].price))  
    log.info("pipe aapl return: {0:.2%}".format(output.loc[context.aapl].returns))  
    log.info("pipe aapl return factor: {0:.2%}".format(output.loc[context.aapl].ReturnFactor))  
    log.info("\n")

Per the logs:

014-06-04handle_data:46INFOpipe aapl SMA1: 637.54  
2014-06-04handle_data:47INFOpipe aapl SMA5: 631.65  
2014-06-04handle_data:48INFOdata aapl close: 92.12  
2014-06-04handle_data:49INFOpipe aapl return: 1.44%  
2014-06-04handle_data:50INFOpipe aapl return factor: 1.44%  
2014-06-04handle_data:52INFO  
2014-06-05handle_data:46INFOpipe aapl SMA1: 644.82  
2014-06-05handle_data:47INFOpipe aapl SMA5: 635.81  
2014-06-05handle_data:48INFOdata aapl close: 92.48  
2014-06-05handle_data:49INFOpipe aapl return: 1.14%  
2014-06-05handle_data:50INFOpipe aapl return factor: 1.14%  
2014-06-05handle_data:52INFO  
2014-06-06handle_data:46INFOpipe aapl SMA1: 647.35  
2014-06-06handle_data:47INFOpipe aapl SMA5: 638.24  
2014-06-06handle_data:48INFOdata aapl close: 92.23  
2014-06-06handle_data:49INFOpipe aapl return: 0.39%  
2014-06-06handle_data:50INFOpipe aapl return factor: 0.39%  
2014-06-06handle_data:52INFO  
2014-06-09handle_data:46INFOpipe aapl SMA1: 92.23  
2014-06-09handle_data:47INFOpipe aapl SMA5: 91.54  
2014-06-09handle_data:48INFOdata aapl close: 93.70  
2014-06-09handle_data:49INFOpipe aapl return: -0.27%  
2014-06-09handle_data:50INFOpipe aapl return factor: -0.27%  
2014-06-09handle_data:52INFO  
2014-06-10handle_data:46INFOpipe aapl SMA1: 93.70  
2014-06-10handle_data:47INFOpipe aapl SMA5: 92.32  
2014-06-10handle_data:48INFOdata aapl close: 94.25  
2014-06-10handle_data:49INFOpipe aapl return: 1.60%  
2014-06-10handle_data:50INFOpipe aapl return factor: 1.60%  
2014-06-10handle_data:52INFO