Introducing Pyenv pip-upgrade Plugin
In my previous articles (Configuring Python Workspace and Configuring Python Workspace: Poetry), I have described how I use pyenv to create several virtual environments. With the lapse of time, the tools that you install in these environments become outdated and you need a tool to update them. I develop a pyenv plugin that updates all packages in all or particular pyenv environments and in this post I describe how to use it.
Table of Contents
Introduction
In my setup, some of pyenv environments are used to install several python tools. For instance, I install Jupyter-related tools into pyenv virtual environment called jupyter. Obviously, with the lapse of time the feature set of these tools improves and in order to facilitate them all packages of these environments need to be updated. This means that I have to activate the environment and update all outdated packages manually. So as there is no pip command to update all packages simultaneously, each time I have to open the stackoverflow answer and copy-paste the bash one-liner to upgrade all outdated packages. Looking at the score of this answer, currently at 2370, I conclude that there are a lot of people facing the same issue.
I have looked for available solutions to facilitate this process and managed to find the pyenv pip-update plugin that can be used to update Python packages in all pyenv environments excluding system. Unfortunately, it cannot update packages of a particular environment, therefore I have decided to develop my own pyenv plugin which I named pip-upgrade. During the development, I have made several improvements that further distinguish my plugin from the pyenv pip-update. In this article, I describe my plugin and the improvements made. In brief, pip-upgrade is a pyenv plugin that allows you to update all packages of particular or all pyenv virtual environments excluding system. It supports both conda- and pip-based environments.
Installation
Before installing the pip-upgrade plugin, make sure that pyenv is installed and configured in your system. Then, run the following command:
$ git clone https://github.com/zyrikby/pyenv-pip-upgrade.git $(pyenv root)/plugins/pyenv-pip-upgrade
This command will clone the repository containing the pip-upgrade plugin into the directory where all pyenv plugins are stored in your system. Now, you can start using the plugin. There is no need to update plugin separately, during pyenv update (pyenv update self) all the plugins are updated automatically if there are any changes.
Usage
It is quite easy to start using the pip-upgrade plugin. In its simplest form, to update a particular environment you just need to run the following command:
$ pyenv pip-upgrade <pyenv_env_name>
For instance, in my case to update all packages of tools3 virtual environment, I have to run the following command:
$ pyenv pip-upgrade 3.8.5/envs/tools3
Bear in mind, you have to specify the full names of environments, you cannot use aliases. Later, I explain why I have made this design choice. You can also update packages of several environments using one pip-upgrade command. Just list the names of environments you need to update splitting them with a space. It is also possible to update all environments excluding system at once. In order to do this, you need to run the following command:
$ pyenv pip-upgrade --all
By default, contrary to the approach described in the popular stackoverflow answer where the packages are updated sequentially, pip-upgrade updates packages of an environment using one single pip install --upgrade <pckg1> [<pckg2> ...] command. This approach will have one benefit with the new 2020 pip resolver. The new resolver will refuse to update packages with incompatible dependencies. The old pip resolver would update packages one by one overriding conflicting dependencies if they exist. Thus, these pip versions would usually finish without errors, however, they lead to inconsistencies after the update. However, you can force pip-upgrade to update packages sequentially by adding --sequential option to the pyenv pip-upgrade command, e.g.:
$ pyenv pip-upgrade --sequential --all
So as new resolver checks packages supplied only within the same command, the sequential update simulates the behavior of the old pip resolver (thus, may lead to the inconsistencies).
The pip-upgrade command supports autocompletion, so after you type pyenv pip-upgrade just press the
Implementation Details
As I mention in the README file, the development of this plugin has been inspired by the following projects/articles:
- pyenv-pip-update — a pyenv plugin to update all environments at once
- stackoverflow answer — stackoverflow answer how to update all packages using pip
- Bash Strict Mode — article about bash strict mode
I have already mentioned how the information from the stackoverflow answer is used. Bash strict mode allows you to write “more robust, reliable and maintainable” scripts, therefore I recommend everyone who develop bash scripts to read the article. However, in this section I want to concentrate on the implementation details of pip-upgrade and how it is different from pip-update.
Listing Pyenv Environments for Update
In order to get the list of pyenv environments, pip-update lists (using ls command) all the files located in the $(pyenv root)/versions/ directory, which is usually points to ~/.pyenv/versions/. For instance, in my case the output of this command looks in the following way:
$ ls -al ~/.pyenv/versions/
total 20
drwxrwxr-x 5 yury yury 4096 Oct 18 23:51 .
drwxrwxr-x 14 yury yury 4096 Oct 13 20:55 ..
drwxr-xr-x 7 yury yury 4096 Aug 18 23:02 2.7.17
drwxr-xr-x 7 yury yury 4096 Aug 18 23:02 3.8.5
drwxr-xr-x 7 yury yury 4096 Oct 10 12:41 3.9.0
lrwxrwxrwx 1 yury yury 47 Aug 18 23:02 ipython2 -> /home/yury/.pyenv/versions/2.7.17/envs/ipython2
lrwxrwxrwx 1 yury yury 45 Aug 18 23:02 jupyter -> /home/yury/.pyenv/versions/3.8.5/envs/jupyter
lrwxrwxrwx 1 yury yury 48 Oct 18 23:51 pyenv_test -> /home/yury/.pyenv/versions/3.8.5/envs/pyenv_test
lrwxrwxrwx 1 yury yury 44 Aug 18 23:02 tools3 -> /home/yury/.pyenv/versions/3.8.5/envs/tools3
As you can see, there are three different Python versions (2.7.17, 3.8.5 and 3.9.0) installed in my system, and there are four pyenv-virtualenv environments (ipython2, jupyter, pyenv_test, and tools3). The pyenv-virtualenv environments are represented as links to the environment directories for specific Python versions. Thus, if you run pyenv pip-update you will update all packages of these seven environments.
However, you can easily delete a link from this directory: rm -f ~/.pyenv/versions/pyenv_test. After this, if you check the list of available environments with the pyenv versions command, you will discover that it still contains the 3.8.5/envs/pyenv_test environment the link pyenv_test has been pointing to, while pip-update does not see it anymore:
$ pyenv versions
* 2.7.17 (set by /home/yury/.pyenv/version)
2.7.17/envs/ipython2
* 3.8.5 (set by /home/yury/.pyenv/version)
3.8.5/envs/jupyter
3.8.5/envs/pyenv_test
3.8.5/envs/tools3
3.9.0
* ipython2 (set by /home/yury/.pyenv/version)
* jupyter (set by /home/yury/.pyenv/version)
* tools3 (set by /home/yury/.pyenv/version)
Thus, if you try to update all packages of all environments using pip-update, it will miss packages of the 3.8.5/envs/pyenv_test environment.
Similarly, nobody prevents you from creating a couple of links (aliases) to the same environment directory. In this case, pip-update will try to update the same environment twice. Although this is not an error, it will require more time for update.
These previous examples are a bit artificial, meaning that if you do not manipulate links (aliases) manually, pip-update should work correctly. However, there is a case when pip-update definitely fails. Let me exemplify this by installing a miniconda environment and creating a new environment in it:
$ pyenv install miniconda3-latest
Installing Miniconda3-latest-Linux-x86_64...
...
Installed Miniconda3-latest-Linux-x86_64 to /home/yury/.pyenv/versions/miniconda3-latest
$ pyenv versions
* 2.7.17 (set by /home/yury/.pyenv/version)
2.7.17/envs/ipython2
* 3.8.5 (set by /home/yury/.pyenv/version)
3.8.5/envs/jupyter
3.8.5/envs/pyenv_test
3.8.5/envs/tools3
3.9.0
* ipython2 (set by /home/yury/.pyenv/version)
* jupyter (set by /home/yury/.pyenv/version)
miniconda3-latest
pyenv_test1
pyenv_test2
* tools3 (set by /home/yury/.pyenv/version)
$ pyenv activate miniconda3-latest
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(miniconda3-latest) $ conda create -n conda_env_test
Collecting package metadata (current_repodata.json): done
Solving environment: done
...
(miniconda3-latest) $ pyenv deactivate
Now, let us list files in the ~/.pyenv/versions directory:
$ ls -al ~/.pyenv/versions
total 24
drwxrwxr-x 6 yury yury 4096 Oct 21 18:23 .
drwxrwxr-x 14 yury yury 4096 Oct 13 20:55 ..
drwxr-xr-x 7 yury yury 4096 Aug 18 23:02 2.7.17
drwxr-xr-x 7 yury yury 4096 Aug 18 23:02 3.8.5
drwxr-xr-x 7 yury yury 4096 Oct 10 12:41 3.9.0
lrwxrwxrwx 1 yury yury 47 Aug 18 23:02 ipython2 -> /home/yury/.pyenv/versions/2.7.17/envs/ipython2
lrwxrwxrwx 1 yury yury 45 Aug 18 23:02 jupyter -> /home/yury/.pyenv/versions/3.8.5/envs/jupyter
drwxr-xr-x 15 yury yury 4096 Oct 21 18:23 miniconda3-latest
lrwxrwxrwx 1 yury yury 48 Oct 21 18:11 pyenv_test1 -> /home/yury/.pyenv/versions/3.8.5/envs/pyenv_test
lrwxrwxrwx 1 yury yury 48 Oct 21 18:11 pyenv_test2 -> /home/yury/.pyenv/versions/3.8.5/envs/pyenv_test
lrwxrwxrwx 1 yury yury 44 Aug 18 23:02 tools3 -> /home/yury/.pyenv/versions/3.8.5/envs/tools3
As you can see, the created conda_env_test environment does not appear in the list of links, although you can see it in the list produced by the pyenv versions command:
$ pyenv versions
* 2.7.17 (set by /home/yury/.pyenv/version)
2.7.17/envs/ipython2
* 3.8.5 (set by /home/yury/.pyenv/version)
3.8.5/envs/jupyter
3.8.5/envs/pyenv_test
3.8.5/envs/tools3
3.9.0
* ipython2 (set by /home/yury/.pyenv/version)
* jupyter (set by /home/yury/.pyenv/version)
miniconda3-latest
miniconda3-latest/envs/conda_env_test
pyenv_test1
pyenv_test2
* tools3 (set by /home/yury/.pyenv/version)
Thus, pip-update will not be able to update packages in this environment.
Definitely, the better way to get the list of environments is by using the output of the pyenv versions command. However, in its default version the output includes the system environment (pyenv environment pointing to the default Python installation in your system), all pyenv-virtualenv environments and their aliases. If we want to obtain a more usable output we should add two options: --bare - to remove the system environment from the list and to remove the highlighting (*) of the currently active environments; --skip-aliases - to skip aliases. In the pip-upgrade plugin, I use the pyenv versions command with both these options:
$ pyenv versions --bare --skip-aliases
2.7.17
2.7.17/envs/ipython2
3.8.5
3.8.5/envs/jupyter
3.8.5/envs/pyenv_test
3.8.5/envs/tools3
3.9.0
miniconda3-latest
miniconda3-latest/envs/conda_env_test
I should mention that this approach has one negative implication: you cannot update a particular environment using its alias; you have to provide full environment name. However, so as pip-upgrade supports autocompletion, this is not a big issue.
Update Packages Installed with Pip in Conda Environments
If you work with conda distribution regularly, you know that its repository does not contain all packages. Therefore, sometimes you still need to install some of them using pip tool. For instance, you cannot install the cowsay package with conda, you have to use pip (note that I install an outdated version of cowsay in order to show later the update process):
$ pyenv activate miniconda3-latest
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(miniconda3-latest) $ pip install cowsay==2.0.2
Collecting cowsay==2.0.2
Using cached cowsay-2.0.2-py2.py3-none-any.whl (6.7 kB)
Installing collected packages: cowsay
Successfully installed cowsay-2.0.2
If you try to update the environment with pip-update (which uses internally the conda update --all command), the cowsay package will not be updated:
(miniconda3-latest) $ conda update --all -y
Collecting package metadata (current_repodata.json): done
Solving environment: done
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(miniconda3-latest) $ pip show cowsay
Name: cowsay
Version: 2.0.2
Summary: The famous cowsay for GNU/Linux is now available for python
Home-page: https://github.com/VaasuDevanS/cowsay-python
Author: Vaasu Devan S
Author-email: vaasuceg.96@gmail.com
License: GNU-GPL
Location: /home/yury/.pyenv/versions/miniconda3-latest/lib/python3.8/site-packages
Requires:
Required-by:
As you can see the cowsay still has version equal to ‘2.0.2’. Therefore, packages installed with pip will not be updated in conda environments when using the pip-update plugin. At the same time, the pip-upgrade plugin updates packages both installed with conda and pip in conda environments:
$ pyenv pip-upgrade miniconda3-latest
[INFO]: Pyenvs to update: miniconda3-latest
[INFO]: Updating packages of 'miniconda3-latest'...
[INFO]: Updating conda environment: miniconda3-latest...
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
[INFO]: Updating packages: cowsay
Collecting cowsay
Using cached cowsay-2.0.3-py2.py3-none-any.whl (6.9 kB)
Installing collected packages: cowsay
Attempting uninstall: cowsay
Found existing installation: cowsay 2.0.2
Uninstalling cowsay-2.0.2:
Successfully uninstalled cowsay-2.0.2
Successfully installed cowsay-2.0.3
$ pyenv activate miniconda3-latest
(miniconda3-latest) $ pip show cowsay
Name: cowsay
Version: 2.0.3
Summary: The famous cowsay for GNU/Linux is now available for python
Home-page: https://github.com/VaasuDevanS/cowsay-python
Author: Vaasu Devan S
Author-email: vaasuceg.96@gmail.com
License: GNU-GPL
Location: /home/yury/.pyenv/versions/miniconda3-latest/lib/python3.8/site-packages
Requires:
Required-by:
