I understand that training TensorFlow (Keras) models on Quantopian is WAY too processor intensive but serving them does not take much processing and can be done with much less processing than most of the models currently running in Q.
I'd suggest seriously looking at serving only with Tensorflow and forget modeling for now. The problem becomes much more manageable.
https://www.tensorflow.org/serving/serving_basic
I'm working on a TS/Keras model now that I'd love to be able to run in Q
As far as I can tell, Q is completely stalled in their development efforts other than a singular focus on expanding to global equities presumably enabled by their work with Factset. Reading between the lines, the 1337 Street Fund is not yet throwing off enough profit to fund a roadmap to expand the platform capabilities. Q is still in start-up mode, since their inception in 2011 (although lots of progress has been made). There was also a major shift toward a SaaS model with the soft launch of Quantopian Enterprise (still maintaining the Quantopian Community-supported 1337 Street Fund). And after more than 6 months of a new contest, less than 200 participants... I'm wondering if the writing is on the wall? Perhaps there just aren't enough folks using the platform productively to justify pouring more money into it...
Requests for TensorFlow and the like have been languishing on the forum for several years now. It would seem it just ain't gonna happen...
Sorry to be so negative, but if Q isn't going to tend to their own forum, when there are valid, thoughtful posts, then somebody should. They can always follow up to correct anything I've said...
Hi John,
Thanks for pointing out TensorFlow Serving, I hadn't seen that before. Adding that functionality isn't currently on our roadmap, but I'll bring it up internally.
I also have a couple of questions for you. You mentioned training the model offline - are you using your own data? And if so, would it be possible to train it once and then upload the OOS predictions to an FTP server and load them into a Q Pipeline using Self-Serve? Uploading ML signals from models trained off-platform was one of the use cases we wanted to support when we set out to build Self-Serve, so I'm wondering if there's a way you can use your model on Q today.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Jamie,
You could do it that way and its how I plan on backtesting for now but in practice with live data that would be a real pain since the the signals are based on seconds and minutes but the models can last for months. It really shouldn't be too much of a lift to include the correct libraries to run these models on Q. The one thing that I would wonder about it where the serialized models need to live and how to grab them.
Seems like to do ML offline, it would be good to have offline access to the QTradableUS point-in-time. I'm not so familiar with ML, but I'd think a starting point would be to narrow the universe to include only stocks of interest.
If I'm not mistaken, if you do ML offline or any processing offline for that matter, you're on your own with regards to sourcing data (price, fundamentals or alternatives), as Quantopian policy does not allow export of their internal data. As far as I now, you can only have access to Q data on their online research / backtest platform. Offline access to the QTU point-in-time would be a good start, but I doubt if that would be happening!
Uploading ML signals from models trained off-platform was one of the use cases we wanted to support when we set out to build Self-Serve
Would be interesting to have a discussion about this. I might be interested in giving it a try, but not without formal guidance/support from Q. If we can't even get traction on a free, offline QTU data feed (which is just a list of stocks, for goodness sake), then it's a non-starter.
In the end, one goes down a path of running full-up algos offline and just uploading the signals to Q. Either Q wants it and supports it, or not. @ Jamie - what's on the roadmap?
Based on the QTU description in the documentation (1st pass, 2nd pass, 3rd pass), wouldn't one be able to create a type of synthetic proxy universe for the QTU that should get pretty close to the real thing, no?
Regarding historical price & volume data, it looks like Quandl provides this for free for most equity markets? My understanding of Q's dataset licensing agreement with the vendors (e.g. Morningstar) is that they are able to provide some for free on a 'rent' basis, but that it wouldn't be free if users could download the full database locally?
So I'm still in the middle of building out my model for this but preliminary results look good. Yes, you need the data offline. I ended up buying the EOD data for the modeling. I'm pretty confident that the numbers will match up since EOD data is pretty standard but it they don't for whatever reason, then what Jaime is suggesting would be another alternative.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.
In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
