Hey everyone,
As many of you know, Quantopian currently runs on Python 2.7. In the coming months, we will be rolling out upgrades to the Quantopian platform that will move us from Python 2.7 to Python 3.5. Python 2 will no longer be officially maintained as of Jan 1, 2020 (referred to as the “end of life” or “EOL” date). As a result, we are planning to cut off of Py2.7 and upgrade to Python 3.5 prior to that date.
Upgrading libraries on Quantopian is a complex task. Millions of Quantopian backtests and notebooks are currently written in Python 2.7. While some of these will be able to run in Python 3.5 without any changes, we are expecting a significant number to require at least minor syntax changes. In an effort to ease the burden of making these changes, we have come up with an upgrade plan that we would like to share with you.
Why Python 3.5?
When we set out to upgrade Quantopian to Python 3, the first thing we had to decide was the minor version of Python 3 we wanted to target. We ultimately decided on Python 3.5 (for context, the newest version is Python 3.8). The biggest reason why we chose Python 3.5 over a newer version was to cut down on the scope of the project and the magnitude of the change. Upgrading to a newer version of Python 3 would have required us to upgrade pandas, numpy and other libraries, which are core dependencies of the Quantopian API. Upgrading these libraries would have required us to make API changes that would be likely to break a lot of user code. At some point, we would like to upgrade to a newer version of Python 3 and newer versions of pandas and numpy. But for now, we wanted to start with an incremental upgrade to Python 3.5 to ease the transition and to allow us to shift our focus back to adding data to Pipeline. For the time being, we are staying on our current versions of pandas and numpy (0.18.1 and 1.11.3, respectively). It’s also worth noting that the whitelist of supported libraries (and their versions) on Q will remain unchanged.
Steps to Python 2.7 —> 3.5 Upgrade
- We plan to update the algorithm IDE to allow running backtests in either Python 2.7 or Python 3.5. We are currently testing this feature, and we hope to make it available to you soon. We plan to release to a small group of alpha users in the coming days, with a public release coming soon after.
- We plan to update our Jupyter-based "Research" environment to allow running notebooks in either Python 2.7 or Python 3.5. We are currently working on this feature, and we aim to roll it out as soon as possible.
- In addition to supporting Python 3.5 in the IDE and Research, we plan to release tools to make it easier to migrate existing Python 2 code to Python 3. In particular, we are aiming to release two tools to help smooth the transition:
- An "auto-conversion" tool based on 2to3 that will allow you to automatically migrate existing Python 2 code to equivalent Python 3 code. We have been testing a prototype of this tool internally, and we've found that the vast majority of algorithms on the platform can be migrated successfully using the conversion tool. We are still exploring the best way to expose the conversion tool in the IDE interface, but we are leaning toward making an interface where you can click a button and get a Python 3 copy of your code in the IDE.
- A "Python 2 Compatibility" extension for Research that will enable running many notebooks that were written for Python 2, in Python 3. Most notably, the extension will allow Python 2-style
printstatements in Python 3.
- An "auto-conversion" tool based on 2to3 that will allow you to automatically migrate existing Python 2 code to equivalent Python 3 code. We have been testing a prototype of this tool internally, and we've found that the vast majority of algorithms on the platform can be migrated successfully using the conversion tool. We are still exploring the best way to expose the conversion tool in the IDE interface, but we are leaning toward making an interface where you can click a button and get a Python 3 copy of your code in the IDE.
- Finally, we will phase out the Python 2.7 environment from Quantopian. In expectation, this will happen in mid to late December. Prior to the Py2.7 EOL date (Jan 1, 2020), we plan to remove the Py2.7 options from the IDE and Research. We plan to leave the auto-conversion tool available for you to continue using indefinitely so you can easily convert any old algorithms to Python 3 beyond the Python 2 EOL date.
Contest
At a certain point, we will stop accepting Py2.7 submissions and require all new submissions to be written in Py3.5. Currently, there are hundreds of submissions running in Python 2.7. To avoid resetting everyone's contest score, we are planning to use the auto-converter to allow any submissions that were written in Python 2.7 to stay running in the contest beyond the end of Python 2.7 on Quantopian. When the time comes to drop Python 2.7 support from Quantopian, any previously submitted contest entries that were written in Python 2.7 will continue to run. Our plan is to use the auto-conversion tool to upgrade Python 2.7 contest algorithms in memory (as opposed to storing the Py3 version persistently) prior to running the nightly simulation. This way, your originally submitted code will stay the same and we can run your algorithm in Python 3.5.
The hope is that no one will have to do any work to keep existing submissions running. We will provide more info on this at a later date.
More Info Coming
Currently, we have a version of the Python 3.5 backtesting working in the IDE. If you would like to help test this feature, please let me know (email [email protected] and mention that you'd like to be added to the Python 3 alpha) and we can add the feature to your account. Soon, we will open up the feature to everyone.
The goal of sharing this plan is to keep you informed of upcoming updates as we further develop our plan. As noted earlier in this post, some steps in the plan are still in progress. We plan to provide updates as we make progress on the work.
