I'll definitely play around with this. Does the data go back any farther than 3 years? Did you code the digitizer or download it?

I couldn't find data going farther back than three years. You can see the image on the bottom of that page (http://money.cnn.com/data/fear-and-greed/) only stores three years worth of data. The digitizer I used is actually online, it's here: http://arohatgi.info/WebPlotDigitizer/.

Hi Gus,

Here's another source of data:

http://www.aheadofthecurve-thebook.com/

I read the book awhile back...the author claims some level of predictability of the market (although on a time scale perhaps not of interest to active trader types). The data and claims on the website might be another source of inspiration for you.

Grant

I cloned your algorithm and used it with monthly consumer confidence data since 2001 (http://www.econstats.com/r/usind__m5.htm)

Instead of just buy/sell if index is above/below 50, what I did was weight the confidence and allow for short sales. So for instance, if the confidence is 51, then long 1/50th of your bet size. Alternatively, if the index is 30, then short 20/50th of the bet size.

The bet size is static so sometimes its more or less that the total capital you have access to. Also, confidence interval spikes above 100 about 25% of the time with a highest value of 117.9 in mid 2001. Therefore, the algorithm uses a bit of leverage at times.

I think most of these confidence intervals are at best a lagging indicator and at worse just noise. It seems like consumer confidence never really recovered after the 2008 crash and has been flat (around 50-60) since then. Therefore, the index really missed out on the market rally since then.

Thanks Grant, that looks like it has some good ideas. I will check it out.

Branko, yeah, I agree that it's just a lagging indicator. It is interesting that the consumer confidence data and the CNN index don't correlate too well. Although the CNN one uses a mathematical formula based on a few different things. I think you are also right in using a weighted bet rather than just betting the same amount every time in an indicator like this where seeing confidence is possible. I guess the theory is that, for example, if there is a 50.5% chance a security's price will rise, then it's probably not worth it to bet much.

The result looks good only because this index is calculated from stock index, e.g., http://money.cnn.com/investing/about-fear-greed-tool/index.html, so they should be highly correlated.

You're right Raullen, the index's value is dependent on the market's current state. But this shows that the Fear and Greed index is not a better than average indicator and is purely a reflection of the current (or realistically, past, due to delay) state and not the future state in any way.

Hello. Thank you so much for this. I am a total quant layman, but a long time conservative investor (not trader). I did wonder though if the CNN index was a good indicator if you buy when the index falls to 25 or below and sell at 85 or above, and just hold in between. I tried to plot the index against the S+P but couldn't even figure that out. If someone can and can compare that to the price of the S+P, I wonder if it would show greater returns than simply dollar cost averaging in every month. Thanks for all your great work.

Thanks for the nice reply Jeff. So on my backtest charts above, the S&P is the benchmark in the top graph (the red line) and the Fear and Greed index is the purple line in the lower graph. You can compare them from there if you'd like, or if you are interested in modifying my algorithm, you can clone it and change whatever you like. Let me know if you need any help with that.

Hello everyone - Great place and Great people.Good work Gus.

Gus: Please post the URL for the site containing the data for the Indicator. Thanks.
Al

Hey Alan,
The site is in my code (click "Source Code" above the graph): http://money.cnn.com/data/fear-and-greed/

Jeff,
Did you get an answer to your question? I have had the same one only I look at 80 greed and 20 fear as triggers since that mark is hit more frequently than your limits.

John

I like the basic idea here... I modified it to weight the buys and sells, including short sales.

Why not take it one step further? It seems to me that when people are fearful, they not only sell securities, but they also shift that money into bonds and gold. So I am considering both positions -- Greedy = sell AGG + GLD and buy SPY (or DIA, or similar), Fearful = sell SPY and buy AGG + GLD

[I plan on extending this a little further by comparing IBS of ETFs]

I have a question: is there a way to do this with current F&G index data? (I notice this would be very easy to grab from the websites source code -- it is simply a number in an li tag, so a simple script to hit the website and parse/search for that section of HTML would do the trick)

I have another question (which I admittedly haven't researched yet in the documentation): is it possible to set timings of these transactions, such as what to do On-Close or On-Open of markets? [PS. I'm a total noob. Not to coding, but to quants and financial in general]

Hi Matthew,

Here is a previous thread that shows how to send orders on market open and market close events. This code should be helpful for your strategy. If its not what you're looking for, let me know.

With the current F&G data, you should be able to implement your greedy/fearful strategy if you modify the logic in the ordering function. You can add the ETFs for gold and bonds in the initialize method.

If you discover another interesting strategy or find a fun phenomenon, feel free to start a new thread!

Best,
Alisa

Thanks for the response... I have not yet implemented the complete strategy, as it was a little too complex for my first foray into this system... however, I did modify this F&G strategy by adding the additional ETFs (plus extras, to test multiple ones) and then using a reverse-strategy for those hedges (gold and bonds); ie. when fearful, sell all equities and buy gold/bonds.

Although this results in lower overall performance, I believe it would also be much less volatile in the long run (hard to tell from this backtest because it only covers a 3 year time span in which the general market was up-up-up for the most part... and also because gold was in an inflated bubble which burst a bit in 2011/2012, so that also has an impact)

It's a very naive, basic strategy that simple works on the fundamentals... but, it does seem to work.

Another thought... it seems to me, that there is still a flaw in weighting based on fear/greed (or consumer confidence, for that matter), because those are lagging indicators (in other words: they tell recent history, they don't necessarily predict anything)

I'm going to test something else: what happens if you sell when it is a mildly fearful index (ie. things are just starting to hit the fan, but there's still room to fall), and buy when it is mildly greedy... but do the contrarian moves when it hits an extreme threshold of fear or greed (aka. a "peak" or a "bottom" -- for example, when everybody was saying "Buy real estate!" in 2005, I knew for certain that it was NOT a good time to buy real estate. Confidence/greed would have been way in the green, but the correct move would have been sell or short when it comes to RE)

In other words -- I'm going to test following my algorithm (combined with a multi-ETF IBS strategy), but throw in some contrarian moves; in other words, there would be 3 actual thresholds: for example, 50 = neutral, 80 = strong buy; 20 = strong sell; > 80 = sell; < 20 = buy.

Not sure if it's just a coincidence coinciding with the bullish market, but doing some tweaking related to the above (contrarian stance on overly-greedy or over-fearful indicator) did seem to improve performance a little bit... moreso, I'd like to try it with real-life forward-moving data, not just backtesting (to see if it is predictive in any way, rather than just reactive)

Does anyone have or can anyone put the daily CNN fear index into an excel worksheet?

I haven't tried it, but since the code is calling a fetch to a "CSV" ("comma separated values") page, you should be able to download that at https://gist.github.com/gusgordon/7615f5b91f3cba1e7ff5/raw/261a3213b20f7cc7d2ee52be2cdc81c49f69a4de/gistfile1.txt and then load it directly into Excel.

Thank you so much, this is great!

Hi Gus, Thanks for the great info and analysis! the WebPlotDigitzer is really cool.

I actually tried using it to create a CSV file but the data came out with duplicate dates and did not exactly account for weekdays only (market trading hours). You must have prepped the data in Excel to the best of your ability.

If so you alter the data did you do so in Excel?

Thanks.

Dear Gus, I do appreciate your link above and I was considering to put CNN index on my market monitoring model but lack of raw data. Just wonder could you fill the data from 2013 July to 2014 Feb. I am grateful to your help and would like to thank you in advance :)

Does this source code still work? It has multiple errors when trying to "Build Algo"

CParserError: Passed header=0 but only 0 lines in file
File test_algorithm_sycheck.py:13, in initialize
File algoproxy.py:1391, in fetch_csv
File requests_csv.py:364, in init
File requests_csv.py:247, in load_df
File requests_csv.py:370, in fetch_data
File parsers.py:400, in parser_f
File parsers.py:198, in read
File parsers.py:479, in __init_
File parsers.py:586, in make_engine
File parsers.py:957, in __init_
File parser.pyx:477, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:4434)
File parser.pyx:599, in pandas.parser.TextReader._get_header (pandas/parser.c:5831)

I am a long time algo developer but first time Quantopian user. Thanks!

Hello Nakul,

Out of curiosity I saved the data with a .CSV extension.....

P.

I was referring to the original backtest with SPY only, not the GLD/TLT. I was able to use the source code from the last posted backtest to access the Fear & Greed index. I note the Sharpe/Sortino is lower then the original strategy posted. Even still, I'm surprised it was this high, unfortunately no down market data in the last 3 years...

Am I crazy or are the values in Risk Metrics changing? The Sharpe and Sortino ratio are higher today (1.04 & 1.63) then the first time I posted the backtest?

Nakul,

I played your setting with CSV import, the F_G curve it not right. It can be easy to spot by comparing FG curve in previous message and the first message. Please check.

When I switch to
fetch_csv('https://gist.githubusercontent.com/gusgordon/7615f5b91f3cba1e7ff5/raw/261a3213b20f7cc7d2ee52be2cdc81c49f69a4de/gistfile1.txt',
date_column='date',

Then fgindex seems okay.

I built a trend following trade system based ob CNN fear and greedy index,

• when fgindex drop 13.5 from top, then sell
  
• when fgindex increase 13.5 from bottom, then buy.

The result is not bad.

Cool, and by the way, the code can be much simpler now since order_target_percent has been added: https://www.quantopian.com/help#ide-ordering

Instead of counting how many shares we own, we can simply just check if the indicator is above or below the threshold, then call

order_target_percent(context.security, 1)  

if we want to go long, and
order_target_percent(context.security, -1) if we want to go short.

Hi all. Thank you again for trying to explain how to use the FG index vs. SPY. Honestly, I am way out of my league as far as the technical aspect goes. I'm pretty seasoned as far as investing. I do still wonder what would happen if you sold at 80 and bought at 20 only. If you buy at 20 and the fg index goes to 70, for example, you don't buy. Yes the fg index is lagging, or current, but it seems like it may help people buy low and sell high. If anyone with a technical bent wants to let me know how that strategy would have done vs. buying the first of every month, that would be great. I hope that using the fg index in this way provides some kind of advantage. Thanks.

Interesting post. I've thought about programmatically using the fear and greed index to set allocations on betterment.com. A few things to remember with the benchmark are:

1.) Dividends: When you sell your positions you are no longer earning them - I don't think this benchmark is including dividends which are currently 1.86% annually with SPY
2.) Taxable events: When you sell, you will have to pay capital gains, in most cases short term. Deferring taxes usually works to your advantage in the long run
3.) Commissions: Although this can be mitigated with an commission free ETF broker.

Instead of selling completely, I wonder what the returns would be like if you basically set tactical allocations instead:

During times of greed switch to 90% bonds, 10% stocks: F&G index over 80
During times of fear switch to 10% bond, 90% stocks: F&G index below 20
For everything else, keep it 50/50 and re-balance quarterly

I suspect this will fail to beat the market, due to the large bond holding. However your volatility will be greatly reduced. Reducing volatility drag is one way to beat the market (if you can do it).

Perhaps a mix if 70% stock, 30% bonds during the times the F&G index is between 20 and 80.

There are tons of books out there that espouse the virtues of passive investing so I won't go into that, but thanks to Gus for doing this test. Fascinating.

Thank you for posting this Gus. It is obviously an older thread but have been looking at the Fear Greed index lately. But what I found the most fascinating was the plotting site you posted http://arohatgi.info/WebPlotDigitizer/ as I regularly come across charts where I would LOVE to extract data.

My question is what method did you use to extract the Fear Greed index. I have uploaded the picture, made the axis's, and then have tried a few ways to pull the data but each time I get as much bad data as good.

If you still use this tool I would LOVE some guidance as to what you do.

Thanks,
Dave

I'm new to Quantopian and curious, is there any way I can have it fetch Real time data from the CNN index or is it just more of a reacting algorithm itself?

If it can be done though, what ways would I have to change the Algo and CSV? I'm not necessarily asking for a new written code, but maybe a point in the right direction.

Thanks!

Thanks Dave! That tool is definitely very cool. It is a little tricky to get data, but it can all be done with Python or even Excel. I remember first running the digitizer to get the raw data. Then you have to normalize the x and y values to your scale and dimensions - for example, on the x-axis, we want dates in this case. I set up some sort of thing to convert the value on the x-axis to a date based on the fraction of the way through it was the date range I wanted, if that makes sense. The resolution was really high, so some days had multiple data points. Since I only wanted the first value (or the average value) for each day, I removed the excess points. That's what I remember, hope that makes sense.

Not sure about getting bad data - I didn't have a problem with that. Try playing with the digitizer settings, maybe. Let me know if you have any luck!

Alec, the F&G index, from what I gather, is a reacting value that is supposed to show the sentiment of investors. If it were very accurate, then it would explain how investors were feeling, and the stock market would follow that signal. It seems like an OK signal to me - though this data is over 2 years old now. The best way to get data in a live algo would be to get the signal from CNN's page, and put that into a CSV for your algorithm to use. This could be done either automatically or by hand each day. The CSV would have to be updates around 2 AM EST or earlier, because that's around when Quantopian will grab the CSVs for the day. See the docs on fetcher and the notes on live trading for more info.

@Yg care to explain....these... I dont really understand... and wat it means...
context.buy = 25
context.sell = 79
context.mode = 1
context.prev = 52
context.delt = 13.5
context.max = 0
context.min = 100

Thanks, Gus, for the idea!

I realize this thread is quiet since long time, but I was curious and wanted to throroughly test it.
And then:
- I have compiled 10 years of FGI (not interested in daily precission, as will show later);
- Defined time windows from 30 to 360 d in steps of 30 d in which to test a predefined buy/sell strategy based on the FGI against just following the S&P (the strategy is defined below); these time windows are the duration of time during which the strategy is compared against the S&P option;
- Scanned all the possible starting dates for the above time windows which are compatible with the duration of the FGI data;
- Run the strategy for all those starting dates and for all the predefined time windows;
- Statistically checked for whether the strategy performs better than the S&P option.

The strategy is based on the following rules:
- if the FGI is bearish and reaches a certain value, then sell a certain fraction;
- if the FGI is bullish and reaches a certain value, then buy a certain fraction.
This strategy has four parameters: the two trigger points in FGI, and the two trade fractions.
I have created all possible strategies independently looping over these four parameters from 10 to 90 in steps of 10.

And the result is that ... in these last 10 years there is no statistical proof (at 95%) that any of these 78732 strategies performs better than simply following the S&P.
Which is a pity, as I expected that somehow there would be more embebded value in the FGI!

The Notebooks are:
- FGI R0, massaging the raw FGI data
- FGI R1, completing the creation of a suitable FGI data list
- FGI R2, the core running the strategies,
- FGI R3, where the statistical analysis is done.

The uploading seems not to be working, so here are the scripts:
- https://app.box.com/s/tci8o2km9gk14zsaqtef6d8fkk1m44g6
- https://app.box.com/s/jm71xf7tuoans022rg96vz9f9ebwlarx
- https://app.box.com/s/1hjtnd7cfko3qxlwjgxqpqds5zzl9837
- https://app.box.com/s/mqdboelgh2s5wewelq0iw61islvni6gb