Having a tough time putting skeletons in my closet

Back to Community

posted

What is really needed here is a generalized skeleton to speed up development

For the record:
I am a programmer who has worked professionally in many many languages since attended Bronx Science in 1980s, and professionally not much shorter than that. So I am not thick or hard to teach. My last job was in medical research, and I have no degrees but am autodidactic

Python is my newest language to the list, and so i am not up to full speed on it yet
(though for me, the learning curve is not as great as it would be for others)

In my pocket i have about a dozen or so trading ideas of which i would LOVE to share with Quantopian
but the way things are set up, its a horribly slow slog than it has to be

a few skeletons that take care of various things would help a lot...
I would be up in days instead of weeks if i had this to go by

as great as people think the tutorial is, and maybe even the documentation, they are not
its not... compared to other things i have worked with over the past 40 years, it really isnt.

you probably lose a lot of people who would have great ideas, except that your own docs are a big barrier to entry
There is not one full example with everything needed for a valid contest submission.
Algorithm stuff is mixed with notebook stuff, and like the military, reasons why and explanations are really lacking.

I am spending all my time trying to cobble together something that will be acceptable and zero time working on a strategy or abstracting a strategy to be allowed, as the tutorials are misleading and missing stuff. I have been trading various ways for years, and would like to use that knowledge.

Of the 4 tutorials:

None show slippage or commission or how to include it
Not one example in any tutorial uses handle_data(context, data)
Using schedule_function is the only example
None show which pipeline to use for the contest vs otherwise except the last one
Putting it all together doesn't actually put it all together!!
Notebook and algorithm are mixed throughout so its not clear where you should be putting things together
The first lesson says algorithm, but actually spends four lessons on research with notebook and without clarity
All examples assume day period trading, no intraday examples, of which you lose huge opportunities
Almost nothing is explained or given context, just hanging examples with no reason in one sentence
In lesson 1 of tutorial 4 we find out lesson 11 is where you find if all your work is ok for the contest!!

Maybe your using this as a filter to keep out all but the most determined...
but it certainly makes it very hard for people who are new to Quantopian to actually use things and get results without being discouraged

I am not the kind that gives up easy, so that wont probably happen to me, but it sure would be nice if things were organized better
you have great comprehensive documentation, which says its out of date and sends you to the very very short form documentation.

So far i have spent four days and the best introductions i have found are out of date you tube tutorials!!

If there was a couple of decent skeletons of whole algorithm applications that would go very far in helping one understand what the final thing has to be like. The incremental stuff without targeting the contest from scratch just means that after someone does a lot of work, they are going to end up having to refactor all their work again. Research on finding things is separate from algorithms, and should be clearer on how to adapt what the example there is to the actual things people will have to use. Alphalens looks great, but is going to be a bear to figure out how to apply it to a specific application. I'm leaving it out as i dont want to waste a lot more time on something that prevents me from having any results. hopefully i can figure it out later to use it...

I really really want to do this, and really enjoy playing with data, information and more so if what i do can be useful rather than just curiosity
I even have my own stock database going back from the 1990s when i started doing this with other programming languages out of sheer curiousity and interest...

i hope this doesnt just sound like an old man kvetching - i dont want to be THAT old man..
but one great full structured skeleton that is not out of date would be stupendous...
as it speeds up creating things by a mile, insures faster success, and goes a long way for us code jockeys to get going asap!

If i end up making one, i will post it for everyone to use... but looks like it will be a while reinventing the wheel before i get a cart moving

6 responses

Viridian Hawk

Right now the platform seems to be in a transition period. The contest criteria are a bit different than the criteria they are using to assess strategies for their fund, so you basically have to approach those two things differently.

None show slippage or commission or how to include it

For contest, submissions you do not need to do anything regarding slippage. The default slippage and commission settings are automatically applied. If you're developing a strategy for the Q Fund or for one of the "Tearsheet Challenges" then you can set slippage and commissions to 0. I guess the direction they're going is that they've decided that modeling slippage has never been particularly successful and so they are throwing in the hat.

Not one example in any tutorial uses handle_data(context, data)

This works the same as a scheduled function that is set to run every single minute of the trading day. I'd avoid using it. Quantopian is useless for developing/researching high frequency strategies, because it lacks bid-ask data, only has 1-minute granularity, and the fill model isn't particularly accurate. If you're trading every minute then you're generating so many trades that it is amplifying any mistake in the fill model over actual changes in share prices.

I'd stick to strategies that rebalance once a day and hold stocks for at least a week or two.

All examples assume day period trading, no intraday examples, of which you lose huge opportunities

Like I said, you can't really simulate intraday strategies with 1-minute OHLC bars. You need tick-level quotes to get anywhere close to accuracy.

For the Q Fund they're looking for strategies where it doesn't matter if you're 1-5 days late entering a position. That's because it can take them that long to leg into the target position without causing excessive slippage.

Viridian Hawk

My contest submissions generally are based off a template like this. I don't have any prior experience with Python, so I might have an idiosyncratic way of doing things.. But I hope this helps.

Ken Rumments

Viridian, this is AWESOME!

Thanks for answering so fast, and on a Saturday too!
I get the feeling, like me, you like tech and finance :)

Ken Rumments

Hi Veridian,
here is some of my attempt.. Idiosyncratic stuff is fine, it teaches as well. I just added Python to my dev invironment, created a subsystem for this stuff, and put in pydev for my eclipse set up (which is getting rather fat with coldfusion, java, php, javascript, python, et al)

as you can see i am trying to use Exponential Moving Average. Its easy to realize that all moving averages are weighted, its just the ones billed as unweighted are weighted evenly and so, are equal to no weighting... ;)

I am hoping I have the decay rate for the averaging right for the standard version so i can match it to my research in TOS (think or swim)

My next biggest challenge is that i require backdata... ie. data from before the current day as well..
and i have yet to figure out how to do that well with the history.

as an aside question, how have you done in the contest? been a contender? been a participant hoping for better? etc.

base_universe = QTradableStocksUS()  
beta = SimpleBeta(target=symbol('SPY'), regression_length=60)  

ExpMovAvg10  = ExponentialWeightedMovingAverage(inputs=[USEquityPricing.close], window_length=10, decay_rate=(1 - (2.0 / (1 + 15.0))))  
ExpMovAvg15  = ExponentialWeightedMovingAverage(inputs=[USEquityPricing.close], window_length=15, decay_rate=(1 - (2.0 / (1 + 15.0))))



return Pipeline(  
    columns={  
        'beta' : beta,  
        'ExpMovAvg10': ExpMovAvg10,  
        'ExpMovAvg15': ExpMovAvg15,  

    },  
    screen=base_universe  
)

Viridian Hawk

I'm on and off the leaderboard. 12th place today. 7th place yesterday. 36th place on the "all time" list, though I anticipate I'll be moving up now that I have a really solid submission on there.

To get the close price from a previous day, I believe you'd define a CustomFactor like this one:

class Previous(CustomFactor):  
    def compute(self, today, assets, out, inputs):  
        out[:] = inputs[0]

That simply returns the first value in the time series for each asset. Then you can use it in your pipeline something like:

price_5_days_ago = Previous(inputs=[USEquityPricing.close], window_length=5)

There are a lot of useful CustomFactors that people have posted on the forums, but it can be hard to find them. I wish there was a repository of them somewhere.

Ken Rumments

WOWSERS... well then i am honored you took the precious time to help! And i do hope i can give you some serious (friendly) competition.

I am ramping up on my Python now... just got the 3.8 system installed... put in the virtual system to use... setting my Eclipse pydev...
put in community version of visual studio as well, because i have a titan X card (3000+ cuda cores), to do machine learning in python
as i may have mentioned, i have been writing code since about the late 1970s early 1980s... started on mainframes.. i used to play with AI,
but it was such a bear (no finance pun intended) given you had to write everything yourself from scratch... and never had time later to play
with the libraries.

My system is a homebuilt, water cooled I7 with an ASUS board for overclocking and 32gb of ripjaw memory (4ghz)..
i recently lost my last job (of 15 years) and am hoping to switch over to more interesting things and more for me than for others.

back to quantopian..

I too wish that the documentation, examples, were better... and their cookbook is not much of a good one..
at first i thought it was their way of keeping the unserious people out of the game and wasting their resources :)

sadly its going to take me a bit more to figure out the whole thing and get my current ideas and things i noticed into a code form..
but if i can, at least then i can perhaps plink along with my cuda cores and see what i can do there...

its a wide open area, like standing on the shores looking out at the sea and wondering what lands will be found..
you know they are there, as things keep washing up and hinting at the possibilities...
though the work world thinks i am too old to learn new things (really? going from a world in which the games were pong to titan X cards, and keeping up at the top of my field doing med research and you have a youngin saying i dont want to learn? really? ha ha ha ha ha!! at least i have what they dont - and wont have for a long time - experience).

all the best...
lets see what stupid question will come up next..

You've successfully submitted a support ticket.

Our support team will be in touch soon.