Just for testing. ...

I cloned the algo "Google Search Terms predict market movements",
but instead of using Google Search, I am using the 1 month of News sentiment data on Apple from historical FinSentS data.

import numpy as np  
import datetime

Average over 2 weeks, free parameter.

delta_t = 5  
def initialize(context):

This is the search query we are using, this is tied to the csv file.

context.query = 'Sentiment'

Use fetcher to get data.
News sentiment data extracted from the files and format it in another excel file. This is purely a test data, please feel free to use.

Note: To my understanding it is a daily score that represent average of sentiment scores of Articles mentioning Apple in Business and Financial News.

fetch_csv('http://data.infotrie.com/Quantopian_sentiment_apple5_1daylag.csv',  
           date_format='%Y-%m-%d',  
          date_column='Date',  
          headers={'User-Agent': 'Blah'},  
          symbol='Sentiment',  
)  
context.order_size = 1000  
 context.sec_id = 24  
context.security = sid(24) # Apple  

def handle_data(context, data):  
c = context  

if c.query not in data[c.query]:  
    return  

Extract Sentiment scores of search query.

indicator = data[c.query][c.query]  

Buy and hold strategy that enters on the first day of the week and exits after one week.

#   if data[c.security].dt.weekday() == 0: # Monday  
# Compute average over weeks in range [t-delta_t-1, t[  

mean_indicator = mean_past_queries(data, c.query)  
if mean_indicator is None:  

#log.info('error1')  

return

Exit positions

amount = c.portfolio['positions'][c.sec_id].amount  
log.info('amount before')  
log.info(amount)  
log.info(mean_indicator)  
log.info(indicator)  

if (amount < 0) and (mean_indicator < indicator):  
    log.info('cover short')  
    order(c.security, -amount)

if (amount > 0) and (mean_indicator > indicator):  
    log.info('cover long')  
    order(c.security, -amount)  

# amountA = c.portfolio['positions'][c.sec_id].amount  
# log.info('amount after')  
# log.info(amountA)

Long or short depending on whether sentiment is clearly down or up, respectively.

#if indicator > mean_indicator and indicator < 35:  
if indicator < 35:  

#if (indicator < mean_indicator) and (indicator < 35):  
   order(c.security, -c.order_size)  

else:  
   if indicator > 70:

# if (indicator > mean_indicator) and (indicator > 70):  
         order(c.security, c.order_size)  

@batch_transform(window_length=delta_t+1, refresh_period=0)
def mean_past_queries(data, query):
# Compute mean over all events except most current one.
# log.info('error2')
return data[query][query][:-1].mean()

This is quite simple and not very realistic...

That sounds like an interesting data set and algo. Did you get it to run? I'd love to see the backtest.

Hi Dan

Back test in there

Interesting stuff. I definitely look forward to seeing more.

I modified your test a little. I shortened the test period (just so the graph would be better) and I put in some rudimentary borrowing limits.

@Dan, in your backtest, see how context.portfolio.cash increases when you short the stock. How does that make sense? That's the kind of thing I was talking about in the other thread.

Hi Dennis,

When you short a stock, your cash balance does indeed increase, because you get the cash proceeds of the short sale.

The cash attribute on the portfolio doesn't represent the value of your portfolio, it represents cash on hand. For total value, you need to look at the pnl attribute on the portfolio.

See the attached backtest, which demonstrates this by buying one share of Apple per day during a month when Apple went down, and graphing various attributes of the portfolio object throughout the month.

jik

@JIK, I just re-read the Help & API page for portfolio.cash. You are correct, it defines portfolio.cash as "The current amount of cash in your portfolio."

Maybe I'm being naive but how does that definition help in the creation of a trading algorithm? A far more useful definition would be "The current amount of cash available in your portfolio."

Take a look at the attached backtest. I modified your "short Apple" backtest. My version is "short SPY, long SH". It shorts SPY and goes long SH every day. Notice how the built-in portfolio object provides no useful information.

By the end of the backtest here are the overall position values:

SPY: ($5,316.80)  
SH:   $5,368.00  

But the portfolio object has this to say:

capital_used: 190.43  
cash: 1190.43  
pnl: 241.63  
positions_value: 51.20  

Clearly I've used more than $10,000 in capital (not $190). And I've far exceeded the available cash which should have only let me commit to a $1000 position before needing to borrow on margin.

I have already found a way to calculate things using a different method in my algorithms. But your Help Docs and examples are promoting the use of portfolio.cash without explaining that it doesn't measure anything useful.

Dennis

If folks are interested, we have real time financial sentiment data available to test. http://psychsignal.com/analysis

Our data correlates well against the VIX and two widely followed sentiment surveys put out by the NAAIM and AAII Have a look at the comparison http://psychsignal.com/survey

Feel free to get in touch with me [email protected]

Dennis - I definitely get the point that you're making, and I agree with it. I haven't yet put together the big picture of how we're going to implement the change. It's one of those things that I want to change as rarely as possible, and that means I have to work it through a number of future scenarios and make sure it works. I'm particularly focused on live trading and how to best represent to the algorithm what your broker will and won't permit. That's a long way of saying "yes, but it will take a bit."

James, that looks interesting, and I'll take a look.

Thanks Dan, we're very early... still building out the site and API but would love to have some early beta testers if anyone is curious. If not I'll come sneak back over in August when things are more polished!

Hi all,

I am following a particular stock on the Singapore Market SGX.
Singapore Telecom.

I have extracted the data for the last month and spent some time on them.

Please find from this link my analysis on excel :
https://dl.dropboxusercontent.com/u/99980873/Singtel%20Test.xlsx

I proceed step per step, below a brief description of my analysis

• 1) Raw data

I have selected the interesting data per date

1) Sentiment score (End of day one)
2) Last spot price of the day
3) Number of Good words and bad words
4) Volume of transactions

• 2)Data used for the graph

I have cleaned the data in order to produce a graphical view in order to detect strategies.

i.e:
In the second graph,
I kept only the sentiment score between [1;30] and [70;100]

To me, it could be the relevant signals.
Score = 0 means “No news, no articles”
Score between [40;70] are not significant "Sentiment is So So "

• 3) and 4) Graph generation and tentative of strategies identification

I have produced a first graph and then a second one in order to identify signals

• 5) Raw code and result on the strategies

With the identified signals, I have manually tested the strategy.

Result was not good !
For 1 month data and a simulation with 10.000 stocks, PNL (-1000 SGD)

• 6) Adjustment on the strategies

I have challenged a bit the data and I found the below:

Sentiment score is being produced at midnight (German time), that is 6am Singapore time

I can then put my order at the opening time and put the reverse order at the closing time.

In Singapore, the market hours

Singapore Exchange (SGX)

Pre Open Session : 8.30am - 8.59am
Matching : 8.59am - 9.00am
SGX Trading Hours : 9.00am - 12.30pm
: 2.00pm - 5.00pm
Pre Close Session : 5.01pm - 5.05pm

Matching : 5.05pm - 5.06pm
No Orders Accepted on POEMS : 5.06pm - 5.15pm
Orders Accepted for next Trading Day : 5.15pm onwards

The adjusted strategy is now positive.

For 1 month data and a simulation with 10.000 stocks, PNL (+3600 SGD)
I will test it on 6 months data !

I will see how it can be coded in Quantopian and will update you

Many assumptions have been done for this analysis, for example:

• Brokerage fees and clearing not considered in the models
  
• Orders can be placed without issue few second after market opening and few second before Market closing
  
• Market price is not extremely volatile at the opening and closing time

Cheers
Daniel

Sentiment Data available from this link,

https://dl.dropboxusercontent.com/u/99980873/Historical%20data%20-Sentiments.xlsx

https://dl.dropboxusercontent.com/u/99980873/Singtel%20Test.xlsx

Sorry for the previous dead link
Daniel

Hey Daniel,

I too was looking at the algorithm based on google search data. You have to excuse me, as I am very new at this and I was hoping you could answer a few basic questions about what and how you're amending the base platform of this original algorithm.

First, you used "fetcher" to acquire sentiment on apple. How and does one get this data say if I am curious using this same algorythm but with the price of Oil? I am lost. Please help.

Hello,

The financial sentiment score is being provided per a market data provider named FinSentS.

From their historical data,
I am not able to find any of the Oil underlyings ie :"Brent" "WTI"

As I am focus on Equities market. I can't help you on that.

You should try to contact them in order to request these data.

Daniel

Dear All,

Long time, I did not post ...
From below, you can find a new set of data on SP500 stocks

https://www.dropbox.com/s/4i7lttkoxnolnyl/New%20Set%20of%20Data%20for%20SP%20500.xlsx

Daniel

Dear All,

FinSentS Open Beta Web application has been released.
You can test the data from the web application

http://portal.finsents.com/

Daniel

Hi,

Because a lot of traders have requested a gold sentiment index.

As wishes,
https://www.dropbox.com/s/ghua8n52lq9fq5a/gold_sentiment.csv

G10 currencies will come asap.

Daniel

Dear All,

I have received the below update from INFOTRIE website.

D

Hello,

You probably have heard about the recent attribution of the 2013 Nobel Prize in Economics to three people: Eugene F. Fama, Robert J. Shiller and Lars Peter Hansen.
This is a true "Financial" Nobel Prize, and a praise for their research on how financial markets work and assets such as stocks are priced.
Allow me to make a parallel with what we are doing at InfoTrie FinSentS in the field of Sentiment Analysis, Big Data and Financial Engineering:

■ Firstly, let's take University of Chicago Professor Eugene Fama, known as the "father of the efficient markets hypothesis," which asserts that all information is efficiently priced into the markets making it incredibly difficult to profit off of trading in the short run. If Fama is fully right, if markets are really efficient, our Sentiment Analysis is worthless!

■ But if you listen to the other Nobel Prize, Yale Economist Robert Shiller, one of the "father of behavioural finance", then it is another story! Since is book "Irrational Exuberance" (2000) he has been well known for its ability to recognize asset bubbles. I don't know whether Shiller ever looked at Sentiment Analysis, but it for sure broadens the toolbox to analyse bubbles.

■ In a more neutral way, I wanted to thank Hansen for his development of econometric techniques for analyzing data and asset prices. He balances the position of the first two laureates, and that highlights the importance of the usage of statistical methods in today's Financial Engineering.

So regardless of whether you trust Fama, Shiller or simply use the techniques developed by Hansen,
I strongly invite you to make up your on mind on Sentiment Analysis, and register for free on our Open Beta Portal: portal.finsents.com

Thank you,
Best regards,
Frederic GEORJON
CEO of InfoTrie Financial Solutions,

https://www.quantopian.com/posts/7d-sma-of-sentiment-as-an-overbullish-or-overbearish-indicator

http://stocktwits.com/FinSentS

Hi all, I wanted to share an interesting blog post BigML recently published using our data: http://blog.bigml.com/2014/05/09/predicting-stock-swings/

The summary is that our data is very useful in predicting volatility.. which is what our current client feedback has said all along.

I think in general it makes sense that any sentiment source (apart from our own) would be predictive of volatility rather than price. I'm not saying sentiment can't be price predictive but the low hanging fruit seems to be volatility. If the market is emotionally excited then volatility should follow.

If you want to play around with our data feel free to create an account here: https://psychsignal.com/users/sign_up then download a historical data set here: http://psychsignal.com/docs There's no cost or subscription involved.

James

http://www.slideshare.net/erictham/real-time-sentiment-analysis-of-twitter-feeds-with-the-nasdaq-index

FinSents is now on Quandl