Python Dependencies for JavaScript Developers

Recently, I needed to update an old Python codebase which had not been touched in 3 years! Of course, the dependencies were so outdated that the Dockerfile of the project wouldn’t build. My JavaScript knowledge did not help me much:

• how could I do a yarn upgrade-interactive --latest to update dependencies?
• where’s the dependency lockfile?
• why do we need to compile C-libraries for Python dependencies?

Luckily for me, my colleague Ondrej patiently explained how to approach these questions.

Using a virtual environment

TL;DR: pip installs packages globally unless we activate a virtual environment.

In Python, the default package manager is call pip and it installs packages from the Python Package Index, shortened to Pypi. The equivalent in JavaScript land would be npm and npmjs.com.

By default, pip installs packages system-wide. This contrasts with the need to pass a --global flag to npm.

To remedy that, we can use a local Python environment (a bit like having our node_modules folder in a JavaScript project). The following command will create a virtual environment in a venv folder in our current directory:

# Create the venv folder container the virtual environment
$ python3 -m venv venv

To make sure pip knows it should install packages in this virtual environment, we must activate it. Depending on your shell, you should source the appropriate activate file in venv/bin:

• activate, for Bash and ZSH,
• activate.fish, for Fish Shell,
• activate.csh, for csh
• Activate.ps1, for PowerShell.

In my case, I will source the one for ZSH:

$ source venv/bin/activate

Now, pip install will install packages in the local folder instead of targeting the global scope. My “Python node_modules folder” is venv/lib/python3.8/site-packages/.

Dependencies in Python

TL;DR: use pip-compile to generate a lockfile and pip-sync to install dependencies from that lockfile

To deal with dependencies, we must install the pip-tools package in our virtual environment:

pip install pip-tools

This will allow us to use pip-compile and pip-sync later on.

Once that’s done, we could for example decide to start a Django project, “the web framework for perfectionists with deadlines”. The latest version of the package at the time of writing is 3.2.5. Let’s create our Python equivalent of a package.json file to express our project needs the Django package pinned to version 3.2.5:

# requirements.in
django==3.2.5

Note: the better equivalent to a package.json would be a setup.py file, since it contains additional fields like the name of the local package, its version, etc. but we just want to write down the dependencies.

EDIT #1: it’s apparently not common practice to pin dependency versions in requirements.in, just in case you really need to (if there are breaking changes for example). Thanks for pointing it out, Ondrej.

To generate a corresponding lockfile of the transitive dependencies of Django, we can run the following command:

pip-compile --output-file requirements.txt requirements.in

This will create the following requirements.txt file:

#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
asgiref==3.4.1
    # via django
django==3.2.5
    # via -r requirements.in
pytz==2021.1
    # via django
sqlparse==0.4.1
    # via django

Now that we have this file, we can finally retrieve and install the packages with the following command:

pip-sync

Note: if you try to run pip-sync without a requirements.txt, you will get the following error:

No requirement files given and no requirements.txt found in the current directory

It is indeed necessary to run pip-compile before.

Updating dependencies

Similar to npm oudated, we can list the outdated Python dependencies with

$ pip list --outdated

For example, after downgrading Django to 3.2.3 in my requirements.in, regenerating the lockfile with pip-compile, re-installing the dependencies with pip-sync, the command above will show:

Package    Version Latest Type
---------- ------- ------ -----
Django     3.2.3   3.2.5  wheel
setuptools 44.0.0  57.4.0 wheel

To update the dependency, I can change the version back to 3.2.5 in the requirements.in and go through the same dance:

$ pip-compile --output-file requirements.txt requirements.in
$ pip-sync
$ pip list --outdated
Package    Version Latest Type
---------- ------- ------ -----
setuptools 44.0.0  57.4.0 wheel

EDIT #2: a batteries-included equivalent to Yarn is called Poetry. Thanks for sharing, Dániel!

Wheels and Eggs

You might wonder what this “wheel” type is for the Django and setuptools packages.

They are still mysterious to me, but here’s my current understanding: in Python, some dependencies need to be built for specific architectures, as they might be written in Cython, “a superset of the Python language that additionally supports calling C functions and declaring C types on variables and class attributes”. Wheels and Eggs are packaging formats for these.

In particular, it is not a good idea to use Alpine Docker images for Python projects as a lot of packages don’t have pre-built wheels for that architecture: your Docker build step will need to compile the packages from source, slowing down your entire process (and requiring you to install the necessary build tools for this compilation to happen!).

I don’t think there is a similar “packaging format” concept in JavaScript land. But we do have packages shipping prebuilt binaries for specific architectures, for example Sharp, the Node.js image processing package. (And we probably have enough issues with CommonJS vs ES Modules as it is… :smile:)

Conclusions

1. All in all, a project needs to:

   • specify external dependencies and their desired version with semantic ranges
   • have a way to retrieve said dependencies
   • keep track of which versions were effectively installed to improve reproducibility.
2. Despite all these measures, the Docker file of an unmaintained 3-year old project could not build. Software rots!