SMA filter added

Leverage = 1.0

Notebook for the algorithm above.

Short and Long.

SHOULD always be hedged

Took a look at the one marked Leverage = 1.0. Good efforts. This is just to encourage attention to detail ...
Max Margin: -16919 _ Max Lvrg: 1.69 2016-03-09 _ An intraday spike.
In the custom chart you can click legend items to toggle everything off except leverage and that date looks like a flat line at 1. Then you can move sliders to zoom in on that date, it eliminates smoothing, and there is no visible spike. The code is recording leverage each morning, so that margin at 169% of the initial capital was not carried overnight, no charge from the broker, I hear, for intraday borrowing.
Ok, no shorting in this version of the algo, so the leverage spike means what happened is that there was an attempt to sell some longs that were partial fills while simultaneous buying of longs were succeeding. Negative cash. The orders were all completed by end-of-day, no warnings about unfilled orders at the end of the day and here there isn't a scheduled task that would quash such warnings.
Thus none of this is a problem unless you encounter unexpected market conditions and that never happens--I mean, which are guaranteed. So you could wind up incurring surprise margin costs and not be real happy about it. Therefore you can protect yourself from it if you wish.
There's at least one other instance of margin like that, 1.51 on 2015-01-09.

To protect against unwanted overnight margin and the associated costs, I haven't put a lot of thought into this so someone else might want to cover it very well or have a better bad idea (from the movie Argo not to be confused with the movie Algo about us, which hasn't been made yet). Just on the fly, consider:
Schedule a check maybe an hour before market close for open orders.
If there aren't any, all done.
If there are open orders, calculate from the order objects using filled and amount values and current prices how badly unfilled orders could result in end-of-day margin, worst case scenario, given current cash.
If there are any worries, cancel orders, sell any longs necessary to compensate. Keep in mind that this new attempt to sell longs might also be partial.

If you'd like to see those partial orders, use track_orders. It'll light up with negative cash whenever those happen.
Having just done that, I can see now it is very few minutes involved at this level of initial capital and orders so schedule position closing first and wait a few, that always helps.

Thanks for bringing this up! Great catch. I have needed to track open orders in the past but its been a while. I'll add that in when I have time.

Jacob, would you shed some light on what the random forest model is doing and what its inputs are? I was assuming that it would be taking sentiment as an input, but it looks like it is just using 3 days of price and volume.

Works With Robinhood Instant

Added 2 more scheduled functions for opening and closing positions.
Added minutely monitoring of leverage - Max and Min leverage is recorded now.

Can grab price and volume in one history call if you want to, then access them like this:

        z = data.history(sec, ['price', 'volume'], context.history_range, '1d')  
        price_changes, volume_changes = np.diff(z['price']).tolist(), np.diff(z['volume']).tolist()  

Has the buy/sell logic been reworked to avoid going over the defined leverage?

@blue, correct, Im dyslexic and sometimes (particularly at night) everything needs to be on a new line as much as possible. I'm just starting my junior year in college (stats and computational math) and, for the most part, everything I know is self taught. I'm sure I will find much neater notation as time goes on. I will admit, as of right now, this algorithm is in "hobby project" notation (it's a mess).
Thanks for the tip - J

@Delman: @ one point in the history (I believe) the algorithm over leverages by 1%. The algorithm does not check for open orders... it just assumes that 5 minutes is a long enough time to sell off all positions that are meant to be sold - which is generally true

ALWAYS HEDGED I generally do not agree with the idea of ALWAYS hedging. It simply doesn't agree with my philosophy and it certainly isn't logical to hedge at all times (in a very bullish market for example). That said, when you are dealing with millions of OPM that may drive one to think irrationally about their exposure to risk. At any rate, here it is, alot of people have viewed this thread and will see it in the future so for all the new people: *don't take this and enter into the contest: it won't win, it's dishonest, and you will be wasting quantopians money as they use up server space to evaluate an algorithm that might be able to finish in the top 30% *

Thanks for this, I will say I look at algorithms all day on Quantopian and it is so rare that I see one worth cloning. This algorithm has potential, it's probably only one of a handful that I would consider worth putting time into. I will attempt to trade this live if it still has good risk/returns after I add intelligent ordering logic, position sizing, and stop loss and check for overfitting

How does this work during crashes (flash and otherwise) or bear markets (2008/09)?

LONG ONLY WITH SMA FILTER MADE TRADEABLE
Increased the min prediction value to 1.5
Sacrificed returns for a lower max draw down and lower beta.

I agree with you that we will know in the next 2 years (and probably sooner, my guess) how much the current market is overextended, given the underlying fundamentals.

Since I'm new to Quantopia, is there any way to add exogenous variables to the trading algorithms?

@Jacob - does your algo utilize RB margin?

@Bob Katz Exogenous variables are something that should be covered in an entirely separate forum thread. My only advice is to come up with an idea and just do it. I doubt many people will know what causally independent variables are at all.

I'm working on setting up an algorithm that applies the "laws" of wave mechanics to stocks over a specified window and trades based on the "behavior of the wave".
Do i understand wave mechanics? More than most - then again that's not too hard to achieve.
Only recently have i gotten a meaningful output. When I started this project I knew pretty much nothing and got more errors and useless out put than I'd like to admit, but I kept at it.

Generally, when you have an inquiry about topics that are even remotely complex, those who have the knowledge you seek are too busy working and those who don't understand, can't help you even if they wanted to.

To give you a quick answer: Technically using a system random as an input for a variable results in an pseudo-exogenous variable - at least as per the definition: A variable that effects a model without being effected by the model and whose method of generation is unknown to the creator.

In the end, this is python - you could build a neural network with hidden layers if you really wanted to... you probably won't have the ram required, but you could do it.

@Jay Maloney: No, look at the leverage - it never (ok it does once) goes over 1.0 . Implementing robinhood gold is well documented on quantopians forums

NOTEBOOK

Hi, Jacob. Nice post! To give you a little fun support of 65% penalty, Confucius (a Chines philosopher two thousand years ago) said "Three People One is a Teacher". Here is quick link:
http://www.chilture.com/images/large/Calligraphy/confucius_three_people_LRG.jpg

This algorithm looks really cool. Is there an issue with the fact that it will never hold a given security for more than one day, or is that how you intended. For example, in the evaluate() function, you have the following code:

if prediction > -0.5:  
    print(str(sec.symbol) +  " | " + str(prediction))  
    if sec not in context.portfolio.positions:  
        context.longs.append(sec)  

so that it is not possible for any position we currently hold to be added to context.longs, and then in the sell() function:

for sec in context.portfolio.positions:  
    if sec not in context.longs:  
        order_target_percent(sec, 0.0)  

this part doesn't allow any security that is currently owned to be held onto to unless it is in context.longs, but that condition is impossible because of the above part. Basically, I figure you may have intended this to be the case, but I think not when I realize the second line of the sell function is redundant here since it is impossible for anything to be in context.longs if it is currently in our portfolio.

I hope I was clear here, but let me know if there are any follow-up questions, as I am still relatively new to Quantopian.

Is this data free to use - or do you need a subscription?

@tyler, its free.
@Dan The algorithm only sells once a day... in the morning, 5 minutes later it buys stocks. It does nothing else until the next day.

@Jacob
I realize that it only sells once a day. My point is that it is impossible for the algorithm to hold on to a security for more than one day. If it determines it should by AAPL for example, then the next day if must sell AAPL, even if the algorithm thinks it should keep it longer. This could result in a stock that is continually being bought and sold on alternating days when it would produce more profits if it was bought once and held onto for the entire period and then sold once it is determined it will no longer be profitable.

@Dan Codos

Ah i see what you are saying. Your concerns and reasoning are totally valid.

The algorithm does hold for more than one day: See the last cell of the notebook attached

Im working on a SMA for sentiment intensity of tech stocks right now and that is essentially how it will trade. If the hype is rising, buy the stock, if the hype starts to die - sell the stock.

EDIT: Dan is right - the logic says it shouldn't hold a stock for more than one day...

Gut Feeling: "The get in and get out quickly" behavior that is currently implemented will be better since people are greedy and tend to over hype a stocks movements. you have no idea how many times I saw: "AMD will go to $30 per share." (before ryzen) meanwhile all the fundamentals and historical data pointed to $12-13 per share upon the ryzen announcement and ~$18 to $24 by the end of the year.

I knew what I was talking about back then because I'm a total tech nerd. They just wanted their long investments to do extremely well regardless of the facts available to them.

@Dan Codos

That makes sense. I also ran a quick backtest and saw that the returns weren't as good when that issue was "fixed" and was wondering why it would be the case. I'd be interested to see if what you first mentioned had any merit though, where you put a stricter set of qualifications that would allow a security to be held onto for more than one day, so it would be a "get in and get out" with exceptions for extremely hyped stocks.

As far as not evaluating the positions in the portfolio already. What about this approach?
1. Don't remove stocks we already hold from evaluation.
2. When looking at the prediction do something along the lines of:

if prediction > 1.0:  
    if sec in context.holding_dates:  
        if prediction < (context.holding_dates[sec.sid] * 0.75):  
            continue  
    print(str(sec.symbol) +  " | " + str(prediction))  
    context.longs.append(sec)  

1. Then in sell add something like:
   context.holding_dates.pop(sec.sid, None)
2. Finally in buy:

    for sec in context.longs:  
        order_target_percent(sec, leverage/len(context.longs))  
        # record the number of days we have held this security  
        if sec in context.holding_dates:  
            context.holding_dates[sec.sid] += 1  
        else:  
            context.holding_dates[sec.sid] = 1  

I won't be able to work on anything until tomorrow, but currently does anyone have a backtest I can toy around with that implements the 'follow the hype train' approach and proper order management? I'd like to try implementing a trailing stop.

@ Luke Izlar; you had mentioned something similar that you were working on. Any progress?

@Dan Codos: Thats what I'm working on now, see the notebook attached.

@louis lombard: You are using up more ram storing information that is already stored.
Using the state of portfolio itself does the trick. see my reply to delman

@Delman woodrum ...... All you have to do is remove line 94 from the post labeled Works With Robinhood Instant and then fix the indentation.

Notebook: I know that during the JNUG event the volume was low (<10). This only supports my assertion that a "high volume" be an initial requirement.

Have you put thought into just implementing a trailing stop onto the higher returning code? That might be more profitable than the current strategy where the return is sacrificed for lower drawdown, while still maintaining a low drawdown. Food for thought :>

EDIT:
Just realized that @delman is working on that. Let me know how it goes!

@Perry Cheves I do have an implementation of a trailing stop in progress, and should have something at least partially done by the end of the week if work isn't too crazy. In the meantime, try smashing this code from @David Edwards into your algo and seeing if you can get at least a baseline working with dirty code.

def initialize(context):  
    context.stock = symbol("SPY")  
    set_benchmark(context.stock)  
    context.stop_price = 0  
    context.stop_pct = 0.95  
    schedule_function(buy, date_rule=date_rules.month_start())  

def handle_data(context, data):  
    set_trailing_stop(context, data)  
    if data[context.stock].price < context.stop_price:  
        order_target(context.stock, 0)  
        context.stop_price = 0  
    record(price=data[context.stock].price, stop=context.stop_price)  

def buy(context, data):  
    order_target_percent(context.stock, 1.0)  
def set_trailing_stop(context, data):  
    if context.portfolio.positions[context.stock].amount:  
        price = data[context.stock].price  
        context.stop_price = max(context.stop_price, context.stop_pct * price)  

Trailing Stop Loss

@Delman Woodrum I've been trying to implement these pieces for the past 30 hours and it seems there is some trouble with multiple stocks. The code you sent only works with one stock while this algo uses multiple stocks based on their sentiment. Right now I'm trying to figure out a way to set a stop on each individual position using similar code but no dice so far. Right now I'm thinking about making each open position something along the lines of:

context.stocks = context.portfolio.positions[get_open_orders()].symbol  

The get_open_orders query returns a dictionary that contains a list of orders for each ID, oldest first. So perhaps using this and looping the other code over the returned dictionary might work. Just a concept, this has been pretty difficult thus far. Let me know what you think and if you make progress.

UPDATE:

So I got something sorta working, except it sells all of the stocks that are currently in the portfolio:
2015-01-07 06:31 PRINT AABA | [ 0.27] 2015-01-07 06:31 PRINT JPM | [ 0.73] 2015-01-07 06:31 PRINT FB | [ 0.66] 2015-01-07 06:31 PRINT I sold a bad stock!:AABA 2015-01-07 06:31 PRINT I sold a bad stock!:JPM 2015-01-07 06:31 PRINT I sold a bad stock!:FB Here is the algorithm with it implemented. It's almost there! We just need to fix this one issue.

> from quantopian.pipeline import Pipeline, CustomFilter  
from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline.factors import Latest  
from quantopian.pipeline.data.builtin import USEquityPricing  
from quantopian.pipeline.data.psychsignal import aggregated_twitter_withretweets_stocktwits as st  
from sklearn.ensemble import RandomForestClassifier  
from quantopian.pipeline.factors import SimpleMovingAverage  
from quantopian.pipeline.filters import Q500US  
import numpy as np

def initialize(context):  
    #Long only  
    set_long_only()  
    #I use robinhood when long only  
    set_commission(commission.PerShare(cost=0.0, min_trade_cost=0.0))  
    #ADDED TO MONITOR LEVERAGE MINUTELY.  
    context.minLeverage = [0]  
    context.maxLeverage = [0]  
    context.securities_in_results = []  
    attach_pipeline(Custom_pipeline(context), 'Custom_pipeline')  
    #PREFFERED CLASSIFIER  
    context.model = RandomForestClassifier()  
    #YOU WANT A HIGH NUMBER OF ESTIMATORS AND SAMPLES TO ENSURE CONSISTENT BACKTEST PERFROMANCE.  
    context.model.n_estimators = 500  
    context.model.min_samples_leaf = 100  
    context.lookback = 3  
    context.history_range = 5  
    #find the stocks we want to buy/sell  
    schedule_function(evaluate, date_rules.every_day(), time_rules.market_open())  
    #close positions  
    schedule_function(sell, date_rules.every_day(), time_rules.market_open())  
    #open positions  
    schedule_function(buy, date_rules.every_day(), time_rules.market_open(minutes = 5))

    context.stop_price = 0  
    context.stop_pct = 0.95  



def Custom_pipeline(context):  
    pipe = Pipeline()  
    sma_10 = SimpleMovingAverage(inputs = [USEquityPricing.close], window_length=10)  
    sma_50 = SimpleMovingAverage(inputs = [USEquityPricing.close], window_length=50)  
    pipe.add(st.bull_scored_messages .latest, 'bull_scored_messages')  
    pipe.add(st.bear_scored_messages .latest, 'bear_scored_messages')  
    #changed to be easier to read.  
    pipe.set_screen(Q500US()&\  
                    (sma_10 > sma_50)&\  
                    (0.35*st.bull_scored_messages.latest > st.bear_scored_messages.latest)&\  
                    # it's not just about there being more bullish posts than bearish... 2 is 100% greater than 1 - but 2 isn't exactly a record breaking number of posts.  
                    (st.bear_scored_messages.latest > 10)\  
                   )  
    return pipe

def before_trading_start(context, data):  
    context.longs = []  
    #For hedgeing...  
    results = pipeline_output('Custom_pipeline')  
    context.securities_in_results = []  
    for s in results.index:  
        context.securities_in_results.append(s)  
    #DEBUG  


def set_trailing_stop(context, data):  
    for sec in context.portfolio.positions:  
        price = data.current(sec, 'price')  
        context.stop_price = max(context.stop_price, context.stop_pct * price)  


def evaluate (context, data):  
    if len(context.securities_in_results) > 0.0:  
        for sec in context.securities_in_results:  
            recent_prices, recent_volumes = data.history(sec, 'price', context.history_range, '1d').values, data.history(sec, 'volume', context.history_range, '1d').values  
            price_changes, volume_changes = np.diff(recent_prices).tolist(), np.diff(recent_volumes).tolist()  
            X,Y = [],[]  
            for i in range(context.history_range-context.lookback-1):  
                X.append(price_changes[i:i+context.lookback] + volume_changes[i:i+context.lookback])  
                Y.append(price_changes[i+context.lookback])  
            context.model.fit(X, Y)  
            recent_prices, recent_volumes = data.history(sec, 'price', context.lookback+1, '1d').values, data.history(sec, 'volume', context.lookback+1, '1d').values  
            price_changes, volume_changes = np.diff(recent_prices).tolist(), np.diff(recent_volumes).tolist()  
            prediction = context.model.predict(price_changes + volume_changes)  
            if prediction > -0.5:  
                print(str(sec.symbol) +  " | " + str(prediction))  
                context.longs.append(sec)  

    # #if you arent a fan of puting all of your eggs in one basket:  
    # if len(context.longs) < 2:  
    #     context.longs = []  
    # for sec in context.portfolio.positions:  
    #     set_trailing_stop(context, data)  
    #     if data.current(sec, 'price') < context.stop_price:  
    #         order_target(sec, 0)  
    #         context.stop_price = 0  
    #         print("I sold a bad stock!:" + data.current(sec, "symbol"))  
    #     # record(price=data[sec].price, stop=context.stop_price)  

def sell (context,data):  
    for sec in context.securities_in_results:  
        set_trailing_stop(context, data)  
        if data.current(sec, 'price') < context.stop_price:  
            order_target(sec, 0)  
            context.stop_price = 0  
            print("I sold a bad stock!:" + str(sec.symbol))  
        if sec not in context.longs:  
            order_target_percent(sec, 0.0)  


def buy (context,data):  
    for sec in context.longs:  
        order_target_percent(sec, 1.0 / (len(context.longs) + len(context.portfolio.positions)) )  


def handle_data(context,data):  
    #if the current leverage is greater than the value in MaxLeverage, clear max leverage then append the current leverage. a similar method is used for minLeverage  
    #max leverage is set up as an array so that one can more quickly log all leverages  find the mean leverage if they want.  
    for L in context.maxLeverage: #ADDED BY JACOB SHRUM  
        if context.account.leverage > L: #ADDED BY JACOB SHRUM  
            context.maxLeverage = [] #ADDED BY JACOB SHRUM  
            context.maxLeverage.append(context.account.leverage) #ADDED BY JACOB SHRUM  
    for L in context.minLeverage: #ADDED BY JACOB SHRUM  
        if context.account.leverage < L: #ADDED BY JACOB SHRUM  
            context.minLeverage = [] #ADDED BY JACOB SHRUM  
            context.minLeverage.append(context.account.leverage) #ADDED BY JACOB  
    record(pos=len(context.portfolio.positions), resutls=len(context.securities_in_results), Min_Leverage = context.minLeverage[0], Max_Leverage = context.maxLeverage[0],Instantaneous_Leverage = context.account.leverage)  

This might be a little off target, but is there a way to store the price the stock was bought at in an array and then use that data as a baseline for your trailing stop/profit take?

I'm getting previous day's average price, and if the current price falls below that price *.95 then it will sell.

Reading further up the thread, the selling everything in the portfolio issue might have more to do with the algorithm sentiment logic then the stop loss. Try running a backtest weekly instead of daily and see if the issue persists, or if the stop loss executes properly in the middle of the week.

EDIT:

Also, instead of getting the previous days price, wouldn't it be better to simply log the purchase price once and use that as a baseline for the stop loss logic?

trading weekly makes no sense: all data would be meaningless given social media is about whats happening right now. trading on wednesday one week may be a great idea and then the next could be horrible.

5% is a great stop loss if you don't know what you are doing. We have specifically targeted stocks that are meant to rise as determined by other people and "verified" by a simple machine learning method: why would you set your stop loss at such a large loss? 3.33-2.5 is a much better stop loss.

NOTE: adding stop losses doesnt make sense in terms of evaluating an algorithm unless they are necessary for its survival. If I want to analyze the raw performance based ONLY on psychsignal data and ML models: a stop loss will ruin that data. If one is trying to evaluate different filtering mechanisms to yield above average ML model yield, stop losses will again ruin the data. given I'm doing both of these things: I'm not posting the algorithm - it's not relevant to what I'm doing. That said: here are the returns of different stop methods. Obviously the returns increased - that's the point of stop losses:

So it looks like, over the long term, having a stop loss implemented is indeed an advantage, if I'm reading the chart correctly.

And as far as trading weekly goes, it only makes sense for testing to see if the stop loss is making stops rather than the algorithm exiting on other factors.

Of course stop losses work... if they didn't work, people wouldn't use them. and as I said above, trading weekly makes ZERO sense - ever. you can't properly evaluate any of that data - its all meaningless. Imagine an Ai designed to detect earthquakes by looking at seismic data as it comes in: It wouldn't make much sense if you only ran it every monday...

Yes. You can use weekly trading to TEST the stop loss and TEST it only to see if it is indeed being triggered. No for live trading, but TEST purposes only.

Nice work Jacob, you should write conclusions in your notebook too so you don't have to rerun all the code when looking back :)

IDK why one would test a stop loss in an environment that they will never actually use. Theoretically I have already answered this question months ago: I wrote an algorithm that traded AFTER movements and it lost all of its money in 6 months. Of course the stop loss will be triggered... Why wouldn't it. I've already showed that stop loss would be triggered in the trading daily algorithm alone. If its being triggered in an algorithm that holds stocks for 24 hours, it definitely going to be triggered when stocks are held for an entire week (since that week also contains those same first 24 hours).

Like I said: of course the stop loss works: that's why they exist. but trading weekly on this data that is pseudo-random is a total violation of so many parts of modern probability theory and thus any data about stop losses that one could obtain is worthless because the data used to test the stop losses is worthless. will the stop loss work, YES. will it matter, NO.

Ill check since I have something that I think is worth testing that revolves around trading weekly.

The notebook attached is for anyone wasting their time with weekly trading.

I ran a backtest of this all the way back to 2013 and realized that the algorithm really didn't start to work well until the last year or two. Do you think this is a problem with the algorithm being overfit for recent data, or that the Twitter information has gotten much better in the last two years. I can't think of an easy way to test this, but if it is the latter then there is nothing to worry about, otherwise, the algorithm could just be riding a short-term high.

I believe the author addressed this somewhat in the original post:

"It should be noted that it seems that the number of posts increased a lot in 2015, I am only testing from 2015 onward as the conditions before this time are not favorable - especially before 2014 when 2-3 results is a common maximum number of results for weeks at a time."

So it seems that it may be due to a lack of information, rather than overfitting or a temporary trend.

since we are at an imminent market crash, how will this do through that?

This is the only historical test we can do right now....

You can always just paper trade it through today, seeing as the market's tanking.

Has anybody live testit this algo on a live account?

also, has anybody thought about Trading options at the same time to hedge against potential market loss.

1. I am paper trading it currently, and it is performing well.
   
2. You can't trade options on Robinhood (which is what is compatible with this algo).
   

Nice, Im thinking to build a similar algo, using AI and deep Learnig for Day Trading.

Someone mentioned hedging so I figured I would post this: NOT always hedged.

Returns of over 400% in 2 years are possible if you remove line 99 - 100
BUT you open yourself up to possibility of large losses.

@luis de la rosa : before you start day trading, take a look at this notebook.... day trading yields far less returns - a large majority of the returns come from after markets movements.

Jacob, I will paper trade with this algo and keep you updated. Thanks for the advise.

I have been live trading this algorithm on Robinhood for a little bit and a couple times I have had issues with all of the sell orders being cancelled right after the market opens. The orders do not even show up in Robinhood, I just see in the Quantopian dashboard that it says the orders were cancelled. The buy orders have all worked normally, assuming I manually can sell all the previous positions in time. Has anyone else had this issue or know what could be causing it?

Has anyone else had this issue

A couple of days last week so I'm using this now to retry in case it happens again (the cancels didn't occur today).
https://www.quantopian.com/posts/robinhood-rejecting-slash-cancelling-too-many-orders-any-reason-why-this-is-happening-1#58b5a527ebd94569ab31dee4

Robinhood said "Our records show that your orders were cancelled by you in the app". Until just now when I dug around in the phone app I wasn't aware that there is a way to cancel unfilled limit orders. My code isn't sending the cancels near market open, does cancel at end of day to avoid the logging of unfilled orders.

Hey Jacob,

I ran some test with the algo and you are doing a nice job. Now I think I found an issue .\ if you run a test during the market crash of 2008 you get a negative return of almost -50% from 2005 to 2010. I personally think this algo works fine in a boom market, but in a market crash, it may not perform well.
I notice the algo start reporting returns mid-2015 when the boom market started. have you noticed this?

let me know what you think
I may be wrong :-)
keep-up the good work.

@luis
I noticed the same thing and someone pointed out the following from the original post:

It should be noted that it seems that the number of posts increased a lot in 2015, I am only testing from 2015 onward as the conditions before this time are not favorable - especially before 2014 when 2-3 results is a common maximum number of results for weeks at a time.

That's not to say it won't do poorly in a bad market, but we can't go off the backtest. With something like social media that has changed so much in recent years, it is hard to know what exactly the cause of increased performance is.

I'm pretty confident that the increase in performance is due to the increase in volume. We shall see.

Anyone trading this right now should understand that the risk is extremely high and this is essentially gambling (we know it works, but we don't have enough data to confidently say that it should keep working) - just look at my notebook about the JNUG data anomaly.

@dan codos - what stocks were canceled - I could be stocks with low volume. Stock prices change a lot at open and the order may never be getting filled... other than that IDK.

I have a few friends who are interested in this stuff and will most likely be live trading this algorithm with ~ 5K (long only).

This looks interesting, thanks for sharing.

Is someone able to clarify what the "prediction" variable represents? Based on the code it looks like we're feeding in historical price change and volume data X into the classifier and then requesting to predict future price change Y, is that correct? I would have thought that "prediction" would be the predicted price change for the next day but that doesn't seem to be the case. It it just some sort of score rather than a value directly related to price?

@jacob,
Thanks for the clarification.

@Hayden McMaster: The best way to explain this is to have the API documentation explain it
RandomForestClassifier

@jacob

Have you try deep learning? to have the algo use previus prediction to learn from past unsuccessful trades. ( something like: alfa go)

it may work.

I just started learning programming and this idea is what drive me.

@jacob, thanks for that. I did have a look at the documentation which is what lead me to think that prediction would be the predicted price move. Based on the documentation, my understanding is that RandomForestClassifier is typically used to classify the data by class labels, but if the parameters to Y are real numbers then it does a regression. In our case Y is daily price moves so I would expect prediction to be an estimated price move based on regression, correct? I also tried RandomForestRegressor which seemed to give similar results.

I was trying to understand the rationale behind selecting the cutoffs for prediction. If prediction is an estimated price move it would make sense to me to have the cutoff based on a % move rather than a per share $ price move. When I tried that the whole algorithm falls apart so I suspect I'm misunderstanding something somewhere.

Yea, I'm currently working with a developer to build a virtual deep learning machine using AWS. will see how it does.

@luis de la rosa you may find this link interesting.

deep learning is not possible on quantopian - you would need far too much ram

Just want to add to this for newer members wondering what you can do, don't let the deep learning limitation talk dissuade you unless you're also into extreme number crunching. Especially since the recent doubling of ram on Quantopian, as I keep throwing more and more code into a minute, I haven't been hitting a ceiling and am impressed.

I started paper trading this algorithm two days ago and ever morbingnit keeps selling positions it doesn't own - I assume those are shorts which I can't do in robinhood? What is interesting though is that it seems to believe that it no longer holds positions in stocks that im never seeing a sell order for that I bought the day prior

@ Blue : i deleted the comment, It can be done - i did it on the most basic level i could think of. so technically i was wrong.
it didnt do anything - but it ran lol.

@ Josh Christensen: I need to know which one you are paper trading... some versions had very little logic involved in making orders.

@jacob shrum I found the same problem as @Josh Christensen with the Robinhood version of the algo.

I was trying to use the optimizer API to place constraints on Beta. However, I get the following error:

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''  

when i do the following:

     constrain_beta = opt.WeightedExposure(  
         loadings=pd.DataFrame({'market_beta': context.market_beta.fillna(1.0)}),  
         min_exposures={'market_beta': 0},  
         max_exposures={'market_beta': 0.5},  
     )  

Is there anything that i'm doing wrong?

Your random forest isn't doing anything. You're feeding in a single row of data with a single value to predict, which it does flawlessly since you're giving it 500 trees. It then just returns the value you had it learn, because that's the only row it has ever seen. To validate this, I've removed any mention of the random forest model and instead fed forward the value that you had wanted it to learn to predict, and it has performed the same. Not saying the algo doesn't work, it's just not because of machine learning.

Actually, you can run it with default trees.. you get the same result... so its not due to the number of trees either.

I take a learn as I go approach to this stuff since I'm teaching myself (im a junior in college and academia is hideously slow IMO). I'll take care to include that in my posts from now on since I'd rather not have people think I'm posting this with any "authority" for lack of a better term.

Also, that ML model is taken right from a post by someone at quantopian. I doubt its "not doing anything" the model is vital to its performance as far as I understand it.

Thanks Jacob, it was an interesting post where I personally learnt from too!

Good stuff,
So I rewrote this portion : (0.35*st.bull_scored_messages.latest) >( st.bear_scored_messages.latest) )
to this: ( (st.bull_scored_messages.latest)/(( st.bear_scored_messages.latest)+(st.bull_scored_messages.latest)) > 0.74 )
Essentially the same result.

Anyhow, my suggestion is that you take that >x where x equals a base value (0.74) +- a beta filter. For example, more bearish SPY or QQQ is, the higher the x value needs to be, the more bullish the market is, the lower that value is. This will help you harness beta surges and diminish beta downturns .

I will work experimenting w/ this, but I am unsure how to pull the sentiment on one particular stock.

@Ryan Clous:
from odo import odo
from pandas import pd
build the data frame with all the attributes you want and then just reference them as you go.

Notebook attached has everything you'd need i believe.

i assume youre doing more than something that just requires
... pipe.add(memes , ['Best Memes']) ... context.results = pipeline_output('Custom_pipeline') ... context.results['Best Memes']

I am still having the issue where the algorithm is selling stock that the portfolio does not and never has owned, but these are not short positions. Additionally, it is somehow not holding a position in some stocks that it never sold. Its almost like the algorithm is selling stocks that it never owned instead of selling stocks that it did own, and for some reason it thinks that that process is liquidating its position.

This is for both the unhedged version and the robinhood version.

I know someone else is experiencing this paper trading it as well. Can't find any explanation for what I am seeing.

I started this one on 8/3 - so it filled its first orders on 8/4. Here are the buy/sell orders for 8/4 and 8/7.

8/4:
4 V
4 JPM
7 A
ILMN 2
EXEL 17
RIG 48
SYMC 14
C 6
BAC 17

8/7:
CLVS -4
WMT -4
XEL -6
S -38
CTSH -4
INTC -9
CSCO -10
X -14
AET -2
EXEL -12
MSFT -4
BAC -13
NVDA 23

It thinks its only current position is 23 NVDA

This means it believes it never sold V or JPM or a number of others, but doesn't currently hold positions in those? Additionally it thinks it sold MSFT, a position it never held.

Solution

Thanks, I will look into the solution later today. I briefly looked it over and it seemed to make sense. I am still very new to python so I appreciate the help.

Question: why restrict this to the Q500 instead of the Q1500?

Yeah, it's mainly because you are feeding in a single row of data. You can't expect the random forest to learn anything other than what you gave it with a single row.

@Josh Cristensen I may make a notebook on Q1500 vs Q500, but, in general, the Q1500 is far too volatile. Almost all algorithms do better when only looking at the Q500.... that said i havent actually checked....
checking now

Edit: Q500 is better in every way.

Tried the q1500, could be of some use with more extensive modeling, but underperformed in my permutations.

Is there a way to pull the trending stocks from stocktwits? I think those might be nice to add to the universe for some alpha discovery.

As for the ordering problem, I just defined sell at open and buy 30 minutes later. Took a 50% hit on returns, but it is still a decent return.

Is there a way?

of course.

you just have to be determined - stocktwits has an API. I'll be on vacation for the next 4 days (ill be working - I hate vacations, but the fam wants to go on vacation - laptop lifestyle FTW) . if I find time, I'll do that and make a post about how to do such a thing - shouldn't be too hard.

Edit... id be surprised if "Trending" was not just that with the highest "total_scanned_messages"....

@Kyle

If the machine learning is being fed with just a single row of data, can it still learn from that? Or is it ineffective?

Hey everyone,
If you replace the random forest classifier with from sklearn.tree import DecisionTreeClassifier, you can run backtests on these faster. No difference in results.

Is it still effective, using only one input to make predictions? It seems like it.

@jacob
I tried to add the top ten securities w/ total scanned messages to a new pipeline, but I cannot do 2 pipelines. I can figure this out, but it would take me a long while as I just learn as I go and am a huge novice when it comes to these things.

Furthermore,I actually saw some promise with working with the default_us_equity_universe_mask, but I got caught up on an error on the model fit, so I had to scratch it.

@jacob
Also trending has a minimum price of $5 on ST

@jacob
Can't seem to run the algo. It was working fine the last few days.

Got the following error message when trying to backtest:
TypeError: Expected str, got unicode
There was a runtime error on line 33.

Hi @King Kantona,

I am live trading it, and it got stopped with the same error this morning.

I've received an email from Quantopian telling me than "We wanted to notify you that due to a problem with our Pipeline API this morning, your live trading algorithm(s) raised an error and stopped. "

I have deployed it again now and it is working Ok.

            if Y[0] > -0.5:  
                print(str(sec.symbol) +  " | " + str(Y[0]))  
                if sec not in context.portfolio.positions:  
                    context.longs.append(sec)  
            #For hedging...

            if Y[0] < -1.0:  
                if sec not in context.portfolio.positions:  
                    context.shorts.append(sec)  

According to this code, doesn't it bias the entries to large stocks because the -0.5 and -1.0 price changes aren't normalized?

Anyone else using this in robinhood live and have it not purchase any stocks this morning?

Thanks Honver,
Yes it is biased , you could potentially change it to a percentage increase to really normalize the data.

In lieu of brokerage shutdown, has anyone figure out how to get this data outside Quantopian?

You cant, its against the TOS.

I'm not wanting to use Quantopian's license of the data, but rather get the data from some place outside of Quantopian so I can still trade on it.

I contacted Psychsignals about licensing the data directly. It was $1k/month, which is high when the algo I'm using doesn't scale to the point to make it worth it.

Not sure is psychsignals will allow it, but I would be interested in splitting a subscription w/ some people.

My backup plan is to build the data myself. I have a basic proof-of-concept running right now.

A few comments on this algorithm.

First, thanks Jacob for sharing your work. It is always interesting to review other's idea.

1- I agree with Kyle on the fact that the RandomTreeClassifer does nothing. The algo feeds it a single sample and expects it to learn the next day price change. This does not work. Train a regressor on a single sample and regardless of the test sample, it will return the training target. See my notebook on this.

2- Based on the above, the test that the algo does at around line 95: if prediction > -0.5: essentially means that you are testing that the price change of the previous day must be -0.5$ or more. This is an absolute price change, regardless of the underlying equity. This test makes no sense. A stock of 100$ will have the same test as a stock trading at 10$.

3- I don't understand validity of the sma filter, its rational. A stock that has been trading higher in the last 10 days than the previous 50 is worth considering but not otherwise? Please explain. I know it works better with it, but still I don't understand it.

/Luc

Regardless of the above, there seems to be some alpha signal in tweets.

So that's how you get all those plots in an Y x Z formation... Thanks for that. I wanted to post information on alpha's between different data sets but the notebook was far too long and you couldn't see more than one graph at a time, making the comparison nearly impossible to make.

Jacob,

Oupps.. I misread your message and went on to a super long answer. I'll leave it below anyways.

Yeah, for the graphs, its pretty cool. I learned how to do it from another Quantopian (can't remember who).

I had to read through my Notebook again. Not much text in there... It is a little bit cryptic. But the gist of it is that the notebook tries to explore if the data(bear score message, bull score message, and the various moving averages) can be combined to create predictive signals. Similarly to your algo, It calculates two signals (amongst others):

prices['feat_3'] = prices['sma10'] - prices['sma50']
prices['feat_5'] = prices['bull']*0.35 - prices['bear']

Then, it calculates y, where y is 1 if the stock went up and -1 if it went down. We could use the actual gain, but it tougher to see. Then I generate the plots. The plots show clouds of dots, where the light green ones are when the stock went down, and dark green ones are for when the stock went the other way. If the clouds are somewhat separable (even if a little bit) then it is a good indicator that the underlying signals have some predictive power. Otherwise, no.

For example, feat_1 and feat_3, we see that most of the outliers are light green, and as such there is a chance that whenever the two indicators fall in that region, we could assume that the stock will tank. In this case, when the number of bear counts is high. We can also see that the number of bull counts is not very indicative.

Hope it helps.

/Luc

According to your Sharpe-Ratio, Did you consider the Buy- Sell price range and commission fee?

Thank you so much.