Updated on 01/19/2016 to fix a bug with the not_near_earnings_announcement filter.

Here is the backtest.

I joined the webinar, it was very useful!

Thanks for the webinar, it answered a few questions I had about Alphalens.

What is missing now is the ability to backtest algos that are trading daily with > 100 long/short positions over longer periods. I'm hitting memory limit pretty fast (and >4Gb in my browser) with an interesting alpha factor I'm trying to test from 2004 to now. Running a tearsheet on them in research is also out of the question...

But all in all, this workflow is really promising! How about using output variables from alphalens to feed into the optimization API?

Thanks for the webinar--it was very informative. I have a few comments:

I believe you have a bug in your not_near_earnings_announcement filter. The & should be a |.

Jess mentioned that the max drawdown on the strategy was quite low, but so was the annual return. Shouldn't the max drawdown be evaluated relative to the annual return as well as the total capital?

Hi David,

Nice catch. That should indeed be an | in the not_near_earnings_announcement filter. I'll fix the notebook and backtest, thanks.

Regarding max drawdown, it's certainly sensible to evaluate max drawdown relative to annual return. In fact, the Calmar ratio included in the Pyfolio table of metrics does just that.

@Jamie
The Sentdex signals are actually generated already via moving average crossovers on the raw sentiment data, so further smoothing of the data is only lagging the signals more, which is probably a decent starting point for the such short forward predictive power. I would personally not smooth the signal at all, just go purely off of .latest. The initial Apple data is going to be very noisy, Apple was highly volatile initially, and the starting data will definitely be a mess because there aren't enough prior datapoints to properly compute those MAs that will be accounting for the crossovers. Yes, it's noisy, but, later, one of the main complaints against the data was that we weren't actually turning over and changing baskets frequently enough. This is why.

Through making the simple .latest change to the sentiment signal, I ended up facing a new error, so I am not sure on performance changes at the moment, will work on it.

For example, let's instead look at BAC (a much less-volatile sentiment company, but still high volume), and choose a time frame that is slightly after right when we start collecting data:

date_mask = sentiment.asof_date.isin(  
    get_trading_days(pd.Timestamp('2014-06-01'), pd.Timestamp('2014-12-01'))  

Then run the rest the same. I am currently unable to share since there're updates being made to sharing, but here's an image of a completely un-smoothed BAC signal:
https://i.gyazo.com/08898047f0705d1ab1510a581934f21e.png

Much smoother, and most companies are going to be more like this.

To put things in perspective, AAPL was the #1 most "sentimented" company for a very long time. Things have slowed down in the last year or two for them, but they were #1 and very volatile at the time, especially back in 2013.

@Vladimir
My takeaway from the webcast was not that this specific strategy's performance is what will get you capital. Instead, it was that it's the level of research and confirmation into the strategy's thesis that's required. If anything, it appears they invalidated that specific strategy that solely used sentiment signals as being valid, and it was via alphalens that you're able to see why and where things went wrong.

edit: updated what happens when modifying to .latest rather than smoothing.

@Harrison,

What is the look back period on the MA you use to generate the signal?

Edit: Also, is there a reason for the scale of -3 to 6?

Thanks

@Frank

Look back period
The MAs are not time-based, they are sentiment-update-based. The following moving averages are used to generate sentiment:
100, 250, 500, and 5,000. These are applied to the raw sentiment values that come in from reading text.

Depending on how many sentiment updates a company gets, the 5000MA might account for the last 6 months, or it could account for the last week. I chose this as the most reasonable since, if there isn't much sentiment/reports on the company, then sentiment as a signal also reasonably wouldn't be changing much.

The Scale
As for the scale of -3 to 6, it is important to note that -3 is just as "negative" as +6 is "positive."

I did -3 to +6 because I found the differences between the negative numbers were not as meaningful, but the differences on the positive side were, so I just wanted a bit more granularity on the positive side than the negative side.

This signal was originally just created for my own uses with the Quantopian Fetcher to play with the signal data in a backtest, and I used it with the fetcher in a Quantopian tutorial series. From there, Quantopian suggested it could be added to the data here, so it was added.

Personally, I'd rather the Sentdex data on here actually be the raw sentiment data, resampled daily as a mean with the accompanying sentiment volume resampled daily as sum to match, and then let the users decide what to do with it. The user could go with the sentiment ma crossover with a factor if they wanted...or something else entirely. Would certainly make it a more versatile dataset, as well as not be leaving out volume.

A perfect example with why you want the volume is why we want to avoid earnings announcements...etc. If suddenly volume jumps 4x, your strategy might not even be using the sentiment as predictive, but instead just the volume of sentiment to know some news likely just broke and you want to avoid for now til things calm down.

I think we were just trying to keep it as simple as possible to begin, plus we ran into a billion roadblocks along the way getting things working right. Maybe it's time we re-look into bringing in the raw values as described above.

Very helpful as always. Thank You Harrison!

Hi Harrison,

That's great insight, thank you! As you mentioned, the webinar was meant to be more of a general overview of the workflow that takes advantage of all the tools on Quantopian as well as insight into the types of strategies that we're looking for. If I'm being honest, I chose the Sentdex dataset because it was a signal-based dataset that I hadn't yet explored. Knowing what you've shared here about the collection process, I probably would have done things differently!

Here's a notebook that better illustrates how I would first look into the signals.

Main modifications:

• Removal of SMA being applied to the signal, since the signal is
  already the return of SMA data.
  
• Illustration of a different company (BAC) for a graph of raw sentiment signals, at a later date, to show both
  the general high-volatility of AAPL, but also avoiding the initial
  data, which, since it uses MAs to calc the signal, is going to be
  very noisy.
  
• dates: This runs start_date='2015-01-01', end_date='2016-01-01'. This
  a full year that contains a neutral market overall, but also includes
  ups and downs that give us some good environments to test in.
  
• New backtest attached via PyFolio which works straight off signals
  (removing the SMA factor, and then also not working with buckets
  since we can't currently organize those right).

Further issues:

• Currently, the quantiles in Quantopian's Alphalens tear sheet
  (.tears.create_factor_tear_sheet) is not lending itself to proper
  sorting, and we have an issue where an entire bucket can be filled
  and never change rank, since many companies may fall in there.
  Alphalens already has a new property for bins which lets you
  explicitly define their edges (rather than evenly dividing your
  data). This is absolutely necessary to properly test this, since we
  do not have an even spread of sentiment signals. That said, with
  these dates, we can at least get away with 3 buckets like was
  initially shown, but this isn't quite how you'd trade this, and these
  buckets aren't really fairly divided.
  

And the backtest that uses a strategy that more represents how one might want to use Sentdex. See above post for breakdown of changes.

@Vladimir
I did not even notice his algorithm here had commissions explicitly set to 0. They did address commissions in the webinar, did you follow along?

In the webinar, they set commissions to 0.001, no idea why he's set his shared algo here to 0.000, that's what I copied to start modifying. I am guessing their actual Q Fund isn't paying the default IB rates that you might pay on a personal account, but maybe I am wrong.

Attached backtest with set_commission(commission.PerTrade(cost=0.001)) as per their video.

@Harrison,

According to Important Rules of Quantopian Open Contest (current way to choose managers)

algorithm must use the default commission model.

So the line 31 of your's code

set_commission(commission.PerTrade(cost=0.000))  

should be commented or deleted.

' Here is backtest of your's latest algo from 2013-06-03 to 2016-12-21 with default commissions.
Equity curve speaks for itself.

@Vladimir
It's a strategy that is looking at a single factor to make decisions, doing nothing else and using almost no logic. It's already doing solid in sideways or dropping markets, and is on the way to being a fantastic hedge. Further improvements would be something along the lines of stop-loss and addressing possibly a strategy shift in a clearly up trending market, and reserving this for times of uncertainty.

I'd also suggest having a filter in place to see if price has already moved and if you're too late with the sentiment signal. Sometimes sentiment comes first, or while price is rising, but sentiment boosts can also come about after the fact, based on on pricing having changed and people getting worked up about it. Given the massive spread we saw in sentiment and returns, we'd definitely want to begin hunting down as many of those causes as possible.

@Vladimir

Also, I just noticed what you posted actually wasn't the same as above. You're using default commissions again. Again, I am not sure if the Q Fund actually pays the same commissions as a personal interactive brokers account would. The posted algo had 0.000 set comms, but in the webinar they used 0.001 commissions. Here's the above algo, with the dates you used, from 2013-06-03 to 2016-12-21, using 0.001 commissions as the webinar suggested.

Of further interest would be why the trades appear to basically cease around Jan 2016.

@Harrison

In backtest below I compared it to SHV - Barclays Short-Term Treasury Bill ETF with default commissions..

@Jamie,
In attributes preferred by the Quantopian Investment Management Team I do not see any related to profitability.
Is that correct?

@Vladimir
The webinar's attempt was to show you what you need to research and consider in order to get an allocation, not to show you an actual strategy that would necessarily get you funded. This was stated within the first few minutes of the webinar.

Next, the question on commissions was also addressed in the webinar. You're consistently ignoring me here, and I am not sure why. Your latest back test again leaves in that default IB personal account commission rate. Again, it's unclear with 0.000 vs 0.001 vs going with an Interactive Broker's personal account defaults for commissions. Trade friction-less first to validate a strategy, and then work on dealing with the friction.

As seen in the notebook, and noted in the webinar, there's a large spread still on the signals in terms of cumulative returns and predictive power. While the mean is positive, there's plenty in the negative pulling the strategy down. Being wrong there, especially on the shorts, is detrimental, especially in a 2013-2016 bullish market range.

Other factors and filters will almost certainly need to be added to this specific strategy to get profitability where you want it, but yes, you want to retain the low beta and low volatility.

In the end, the whole point I received from the webinar is that you want to be analyzing your strategy far beyond, and ideally also far before you're worried about, the % return at the end of a back test.

Jamie & Jess -

Here's an algo to consider. I will post the tear sheet, as well. It looks to me to be reasonably attractive for your fund; it would be interesting to hear your perspective.

I do not have any out-of-sample results yet, but I don't think there is any gross over-fitting at play here.

Grant

Ref.: https://www.quantopian.com/algorithms/5883523a801d3c438641fed3/5884a39bb07bf961362be5f6

Here's the tear sheet. --Grant

I'd also be very interested to hear what we're to be looking out for with convex optimization specifically in a strategy. Seems like the optimization API is relatively easy to produce very appealing results.

Hi Harrison,

At some point, I could start a separate thread, so as not to hijack the main thrust here, which is to talk about workflow, tools, and algo evaluation process at a higher level. One interesting thing would be to sort out how to circle back and generate an input to alphalens using the signal in the algo. As I've been told, this should be doable, and might provide some insight, since alphalens is not restricted to daily bars (but maybe daily signals, due to memory limitations). Before getting too excited, it would be nice to get some "warm fuzzy" feedback regarding the potential for fundability, since maybe there is a fatal flaw, relative to the requirements.

@Grant
I think probably both a separate thread and this question remaining here makes sense.

Here, I would like to hear what to look for within alphalens and pyfolio with regards to convex optimization in a strategy and how to validate/invalidate it. From what I can tell, with your submission for example, it seems to hitting all the right notes, but I too would like to see a green light from high command that suggests it's an acceptable approach. I would hope the optimization API is at least a slight nod toward people using it for fund-ability.

I'd also be interested in a new thread dedicated more generally to convex optimization applied to Quantopian. When I was poking around the optimization API threads, it looks like you've been playing with this topic for a while. I'd love to hear what you've learned if you're happy to share.

@ Harrison,

I suggest any discussion of minutiae of the CVXPY-based code I posted above be done on:

https://www.quantopian.com/posts/long-short-algo-w-slash-cvxpy

Hey Harrison,

We're about to be releasing a template algorithm for the lecture series that uses the optimization API in addition to implementing a multi-factor model. It may provide some context/answers for your questions.

Hi Delaney -

I suppose this is not the forum to go into any depth, but I'd be interested in the following topics:

• How best to combine factors based on daily bars (e.g those defined within pipeline), with ones that run on minute bars (e.g. the example I posted above).
  
• How to run the Quantopian optimization API over an expanding window, for smoothing (e.g the example I posted above). I think this entails running the factors over a series of look-back windows, combining them, running the optimization for each look-back window, and then combining the results (e.g. in the example I show, a weighted average of the portfolios is performed).
  

Hi Jamie & Jess -

I posted another example for you to analyze, if you care to:

https://www.quantopian.com/posts/contest-algo

It ran in the Q contest, under https://www.quantopian.com/live_algorithms/574e1c554ac0dc970700023e (note that for the posted example, I simply changed to context.leverage = 1.0). It might be a nice example for you to discuss with the "crowd" since there are in-sample and out-of-sample data. On the surface, it would appear to meet the basic criteria, but must have fallen short in the details (e.g. too few stocks, NASDAQ only, hedging instrument, etc.).

Hi Jamie & Jess,
Great webinar in terms of explaining the typical workflow and how to use and interpret the results of alphalens - thanks!
I'm just battling to get this example to run over a longer time period. As soon as I increase the time span from 1 to 2 years the notebook just hangs on the run_pipeline stage. For example, at the moment I've set it to run 2014-2016 and have left it for over an hour and no results yet. Could this be an issue with internet speed or processing power on my side, or is it more likely to be processing power / memory on your server?
Thanks!

Hi Richard,

That definitely sounds odd. Did it work for the same period as the backtest posted here? I would recommend emailing in to [email protected] so that our support team can help sort things out.

Hi, could someone explain to me why the return in the backtest (2%) does not match the spread between top and bottom quantiles computed by alphalens? Looking at the cumulative returns of the top and bottom quantiles the return should be about 3%+3%=6%. Where does this inconsistency between alphalens and the backtest come from? I tried setting the commissions and slippage to 0 and rebalancing as close as possible to the open, but it's still nowhere near 6%. Am I missing something?

Hello Christian,

Great question. In practice backtests are the final step of the quant process, and are used to determine whether the theoretical returns of an alpha factor can survive the messy conditions of the real market. Two of the big factors that will reduce returns are slippage and commissions as you mentioned. Another one is liquidity. Alphalens makes no assumptions about ability to buy/sell, so sometimes you may not be able to fill orders in the backtest. I would also ensure rebalancing frequency matches alphalens, and in general that the way you're ordering stocks is the same. Does alphalens weight securities or assign equal weighted positions? Etc.

We're hoping to release documentation and template algorithms using our optimize API soon. Those will also have another source of mismatch from alphalens as the optimizer will be deciding position weights based on a set of constraints. Alphalens has no sense of a risk model so will not be able to model this.

Hi Delaney,

Thank you for your reply. From what I can tell, alphalens is rebalancing and weighting positions in the same way as the backtest. So I guess most of the difference is caused by liquidity issues as you suggested. I see a few warning messages saying "your order failed to fill by the end of day and was canceled". What exactly is causing this?. Do you know where I can find out more about the modelling of liquidity in quantopian/zipline? Thanks

Hi Christian,

The warning you see is most likely due to the liquidity of the security you are trying to trade. Orders will remain open until there is enough volume in a minute-bar to fill them, and orders that remain open until the end of the day are automatically canceled.

The amount of volume that your algorithm can take per minute-bar is set by the slippage model you are using. If you don't specify a slippage model, the backtester will use the default. You can learn more about the slippage and commissions models in lessons 9 & 10 from the Getting Started tutorial, as well as the documentation in our Help page.

Additionally, If you are interested in a more in-depth explanation of the default slippage model I would suggest going through this blog post.

A question about the allocation criteria, if I may. Suppose an algo has a relatively low beta (0.01-0,.2), but is not perfectly hedged (60/40 or 70/30, instead of 50/50). Would it still be eligible for allocation, if furnnishing good retruns and low volatility?

I second that question. I'm using the Optimization API and currently specifying a dollar neutral tolerance of 0.1. Does the team look for a tighter dollar neutral strategy?

I started paper trading the same strategy with a tighter 0.01 tolerance so that I may be able to use that out-of-sample performance in an effort to get an allocation. But in the case of the contest, the strategy has better performance metrics (volatility, beta, sharpe etc.) when it isn't perfectly hedged. I'm curious if it's such an important characteristic of an algorithm you'd consider for allocation, why is it not included in the contest scoring/badge process?

Jessica Stauth covers this topic in her QuantCon talk. It was published on the Quantopian blog recently: https://blog.quantopian.com/making-grade-look-inside-algorithm-evaluation-process-quantcon-2017/

I am sorry, Josh, but Jesicca's talk does not really provide a (specific enough) answer to my question.

We know they want dollar neutrality, the question is how tight of a tolerance do they require? Is 55/45 long/short ok, even 60/40?

Antony, dollar neutrality and a beta of 0 are not necessarily the same thing. A beta above 0 indicates a bias to being long the market, not necessarily holding more longs than shorts... if the factor construction has a low beta characteristic.

Many factors (such as most fundamentals, low-volatility, etc.) will prefer stocks with a low inherent beta. When built into a market neutral strategy this will require more longs than shorts in order to have a beta of 0. Tim and I (and many others) can make a perfect 50/50 long/short portfolio based off of the factors we've developed but this may come at the cost of having a negative beta - which may be worse than having a slightly net positive exposure.

This is what we want Quantopian to directly answer (and something that has been asked about in other posts), in addition to a beta target of 0, what type of dollar neutrality tolerance (if any) is allowable?

Thanks to Dan Dunn for pointing me to this thread!

I know that we haven't put out specific and complete cut-off threshold values for all the metrics we consider in our algorithm selection -- apologies if that has been frustrating, which is not our intention!

Today a good indicator that I'll see your algo (if it holds up in 6 months of out of sample) is that your algo's trailing 6 and 12 month beta to SPY is less than 0.3 (ideally less than 0.10 is what I like to use today). We prefer as close to 0 net exposure as possible as well, let's say same order of magnitude, but again "today" I'm not filtering on net exposure. That said, if we select your algorithm for our diligence process, you can be sure that we'll ask you if you've tried optimizing to a tolerance of less than 10% net exposure, and if you can see how that looks - so might as well try now!

We think of both market beta and dollar exposure as forms of model risk and as such we give preference to algos that minimize both, while its certainly true that you can do one without the other and vice versa, either intentionally or because you aren't explicitly matching the beta exposure of your long and short books.

Please let me know if that is helpful guidance or what other questions you might have about the selection process.

Thanks Jessica, that's helpful - special thanks for giving us a number at 10%!

I'll keep the market_neutral = opt.DollarNeutral(tolerance=0.1) for the contest entries because it does a better job tracking a beta of 0, and I'll keep a paper trading version at a tolerance of 0.01.

Hopefully we'll be talking in ~6 months as part of your due diligence!

Very helpful discussion, thanks a lot, everyone!

Is it possible to get some quick feedback before waiting for the 6 month period. If the algo is not going to make the cut because it violates some important criteria, getting to know that ASAP will cut down on iteration time (6Month * N) and help us refocus to fit the criteria. Since the algorithm selection process is somewhat automated, making some of that scoring visible to the algo developer will make a day/night difference in visibility on what to improve in an algo in a timely manner. It benefits both Quantopian and the developer if we can recalibrate and refocus at the earliest.

Hey Coyote,

You're totally right, optimizers are meant to tweak the weights to be within risk constraints while preserving the objective alpha. If the objective alpha was not built with the constraints in mind and has heavy biases, the optimizer will either not be able to do much, or distort the alpha beyond all profitability. If you find your alpha to have large exposures, it's often better to think about a better alpha design than to try to fit a square peg in a round hole. The optimizer is a final step once you have a combination of alphas that average out to fairly low exposures.

One objective measure of "square peg round hole" is to look at the alphas chosen by the optimizer versus what was available. A perfectly constructed alpha should result in N from the top, and N from the bottom of the ranking being chosen by the Optimizer.

Actually this is known as the transfer coefficient, basically how much of your alpha (strength measured in information coefficient 'IC') transfer to the actual positions. http://www.institutionalinvestor.com/article/1027267/the-cost-of-constraint.html#.WXEL9hPyvUI

The higher the transfer coefficient the less your constraints are warping your alpha. I think it's canonically measured via Spearman Rank Correlation (https://www.quantopian.com/lectures/spearman-rank-correlation), but you could also use something like MSE to estimate. Basically a distance metric between vectors should get at the same idea.

Hi. I just copied and pasted the lines one by one in a notebook and I get the following error with this line:

sentiment_2014_df = bz.compute(sentiment_2014_expr)

ValueErrorTraceback (most recent call last)
in ()
----> 1 sentiment_2014_df = bz.compute(sentiment_2014_expr)

/usr/local/lib/python2.7/dist-packages/multipledispatch/dispatcher.pyc in call(self, *args, **kwargs) 162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/build/src/qexec_repo/src/blaze/blaze/interactive.pyc in compute(expr, **kwargs) 203 raise ValueError("No data resources found")
204 else:
--> 205 return compute(expr, resources, **kwargs)
206
207

/usr/local/lib/python2.7/dist-packages/multipledispatch/dispatcher.pyc in call(self, *args, **kwargs) 162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/build/src/qexec_repo/src/blaze/blaze/compute/core.pyc in compute(expr, d, return_type, **kwargs) 410 d4 = d3
411
--> 412 result = top_then_bottom_then_top_again_etc(expr3, d4, **kwargs)
413 if post_compute_:
414 result = post_compute_(expr3, result, scope=d4)

/build/src/qexec_repo/src/blaze/blaze/compute/core.pyc in top_then_bottom_then_top_again_etc(expr, scope, **kwargs) 151 # 1. See if we have a direct computation path with compute_down
152 try:
--> 153 return compute_down(expr, *leaf_data, **kwargs)
154 except NotImplementedError:
155 pass

/usr/local/lib/python2.7/dist-packages/multipledispatch/dispatcher.pyc in call(self, *args, **kwargs) 162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/build/src/qexec_repo/src/blaze/blaze/server/client.pyc in compute_down(expr, ec, profiler_output, compute_kwargs, odo_kwargs, **kwargs) 240
241 if not ok(r):
--> 242 raise ValueError("Bad response: %s" % reason(r))
243 response = serial.loads(content(r))
244 if profiler_output is not None:

ValueError: Bad response: Computation failed with message:
AttributeError: 'frozenset' object has no attribute 'resources'
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/server/server.py", line 647, in compserver
**compute_kwargs),
File "/home/databazaar/.venv/lib/python3.5/site-packages/multipledispatch/dispatcher.py", line 164, in __call_
return func(*args, **kwargs)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/compute/core.py", line 424, in compute
d2 = swap_resources_into_scope(expr, d)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/compute/core.py", line 367, in swap_resources_into_scope
return toolz.merge(expr._resources(), scope)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in _resources
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/.venv/src/toolz/toolz/dicttoolz.py", line 38, in merge
for d in dicts:
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in _resources
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/.venv/src/toolz/toolz/dicttoolz.py", line 38, in merge
for d in dicts:
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in _resources
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/.venv/src/toolz/toolz/dicttoolz.py", line 38, in merge
for d in dicts:
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in _resources
return toolz.merge(arg._resources() for arg in self._inputs)
File "/home/databazaar/.venv/src/toolz/toolz/dicttoolz.py", line 38, in merge
for d in dicts:
File "/home/databazaar/databazaar-clone/submodules/blaze/blaze/expr/core.py", line 239, in
return toolz.merge(arg._resources() for arg in self._inputs)

Don't know if anybodyelse has had it.

I'm getting the same error message:

ValueError: Bad response: Computation failed with message:  
AttributeError: 'frozenset' object has no attribute 'resources'

Happens in the original notebook when I try to execute:

sentiment_2014_df = bz.compute(sentiment_2014_expr)  

@Jamie,

I get the same errors when cloning the notebook and running all cells without making any changes. Would you be able to have a look?

Also, in the notebook you say:

Let's see how many securities are covered by this dataset between
12/2013 and 12/2014.

However, I don't see this timeframe defined in the next or earlier cells. Am I missing something?

Thanks for a super useful video/webinar (more of this please). I just wish I could run the NB as well. Perhaps the NB was updated to use a different dataset due to this problem?