Joachim Burket's Page

Managing dependencies with Poetry

 
  • Jul 02, 2021

Poetry is a tool for dependency management and packaging in Python. It allows to declare a project information and dependencies in the pyproject.toml file, and manage them (install/update).

Packaging and dependencies management is not always fun in Python. I am normally using pip to install package, and its requirements.txt to manage my projects dependencies. Combined with Virtual Environments, this is working pretty well, but there are some drawbacks that Poetry helps solving.

Poetry benefits

  • Configuration centralization: All the project information and dependencies are written in one file: the pyproject.toml file. Here is an example:

    [tool.poetry]
name = "My super project"
version = "0.1.0"
description = "This is a really great description, isn't it ?"
authors = [
    "Philip J. Fry <philip.fry@futurama.org> "
]
repository = "https://github.com/pjfry/super-project"
keywords = ["super", "project"]
  
[tool.poetry.dependencies]
python = "^3.8"	# the python version
django = "^3.2"
djangorestframework = "^3.12.0"
mysqlclient = "^2.0.3"
mod_wsgi = { version = "^4.8.0", optional = true }
  
[tool.poetry.dev-dependencies]
pylint = "^2.8.2"
isort = "^5.8.0"
black = "^21.5b1"
flake8 = "^3.9.2"
bpython = "^0.21"
pre-commit = "^2.12.1"
  
[tool.poetry.extras]
# allows to specify extra packages to install (for example) on prod server
mod_wsgi = ["mod_wsgi"]
  
[[tool.poetry.source]]
# allows to specify a custom pypi repository
name = "jfrog"
url = "https://myjfrog.ch/artifactory/api/pypi/my-pypi/simple"
default = true
  
[build-system]
requires = ["setuptools","poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
  
### We can also specify tools configurations (the one supporting it) ###
[tool.black]
line-length = 88
...
  
[tool.isort]
multi_line_output = 3
line_length = 88
...

    This greatly reduces the messy files in the root of our repository (like setup.py, setup.cfg, requirements.txt, requirements-dev.txt, etc..), and I like that a lot.

  • Better dependencies control:

    There are three big advantages in poetry:

    • When doing a pip freeze > requirements.txt, we end up with the list of all dependencies (our dependencies, and the dependencies of our dependencies). The requirements.txt quickly become a big boy, which doesn’t facilitate the dependencies management.
    • Poetry allows to specify the dependency min and max version, like django = "^3.2". This means that when running poetry update, Django won’t be updated to a major release (version will be >=3.2 and < 4.0)
    • Removing a dependency also removes unused dependencies (which pip doesn’t do)

  • Publishing packages to a repository: With pip, in order to publish a package to a repository, some setup files must be created. With poetry, it’s as simple as

    $ poetry publish -r <repository> --build

Starting a project with Poetry

Poetry installation

To install Poetry, there are a few possibilities:

  1. Install poetry globally using the installation script (the recommended method):

    $ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
$ export PATH="$HOME/.poetry/bin:$PATH"
$ poetry --version

  2. Create a virtual env, install poetry using pip, and then

    $ python -m venv venv
$ source venv/bin/activate
$ pip install --upgrade pip poetry
$ poetry --version

Project configuration

After poetry is installed, a pyproject.toml can be created in our project directory with $ poetry init. We can start adding dependencies in it, for instance:

[tool.poetry.dependencies]
pendulum = "^1.4"

After that, to install the dependencies specified in the pyproject file, the $ poetry install command is used. If poetry is installed globally (using the first method), poetry will created a virtual env in the {cache-dir}/virtualenvs folder. This can be changed in poetry config file (located in ~/.config/pypoetry/config.toml), either by changing the virtualenvs.path variable, or by setting virtualenvs.in-project to true.

If on the contrary poetry was installed into a virtualenv, it will install the libraries directly into this venv.

When the installation is done, a poetry.lock file is created. This file specifies the dependencies and their versions installed. It is highly recommended to add this file to the git repository, so that the project can be replicated with the same dependencies versions (and therefore avoiding unexpected bugs).

Finally, to use the virtualenv created, either activate it with $ source <venv_dir>/bin/activate, or with the $ poetry shell command.

Poetry useful commands

  • Initializing pyproject.toml file: 
    $ poetry init

  • Configuring poetry

    # List poetry configurations
$ poetry config --list
cache-dir = "$HOME/.cache/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"
   
# Modifying poetry configurations
$ vim ~/.config/pypoetry/config.toml
   
# Setting a config (this one will create the venv into the project dir)
$ poetry config virtualenvs.in-project true

  • Managing project dependencies:

    # install project dependencies (from pyproject.toml) in its venv. This command can only be runned when there is no `poetry.lock` file
$ poetry install
   
# update all project dependencies. When specifying dependency like `requests = "^2.25.1"`,
#  the package installed will be ">=2.25.1,<3.0.0"
$ poetry update
   
# add a package to pyproject.toml and install it
$ poetry add requests@^2.25.1
   
# remove a package
$ poetry remove requests
   
# show the list of installed packages (like pip freeze)
$ poetry show
  • Publishing a package to a local repository: 
    # configure local repository to publish to artifactory.
$ poetry config repositories.artifactory https://artifactory-url.com/api/pypi/pypi-local
   
# save credentials for the "artifactory" repository (artifactory)
$ poetry config http-basic.artifactory <username> <password>
   
# build and publish the package to a repository (the on configured in ~/.config/pypoetry/config.toml)
$ poetry publish -r <repository> --build
Search