The main resource is Anna-Lena Popkes’ post An unbiased evaluation of environment management and packaging tools. See also Modern good practices for python development.
python virtual environments
pyenv is a python version management tool, which makes it easy to switch between multiple versions of Python. It works alongside a virtual environment manager venv (or virtualenv) and a package installer pip. The uv project aims to replace all of these.
conda is its own distribution of python (and R), and contains the python version and packages all in one, as well as dealing with package management. It is commonplace in scientific computing.
uv
Replacing pyenv + venv + pip
uv python install 3.11 3.12 # install multiple versions of python, correct version specified in pyproject.tomluv venv # creates .venv/ virtual environment with all the packages
source .venv/bin/activate
uv pip install -e ."[dev]"mamba (conda, but quick)
mamba create -n new-env python=3.11
mamba activate new-env
pip install -e ."[dev]"This worked for me for a long time. I tried to move to micromamba, which does not have a base environment, but VSCode would not let me select micromamba environments as the python interpreter.
Time to try out pixi project, which is built by the mamba devs. It aims to overcome many of the issues whilst also being a multi-language package manager. You will be able to build pip and conda packages using pixi. It has VSCode support.
python package management
Everything should get a python package. We should constantly be refactoring model development notebooks into .py files.
However, python packaging is was a mess. Now uv will sort everything out.
mkdir <project name>
cd <project name>
uv init --lib # creates pyproject.toml and uses hatch for packaging
uv add <package> # creates uv.lock
uv add <dev package> --optional <group> # adds to dev-dependecies in pyproject
# OR
uv pip install # pip interface
uv add git+https://github.com/encode/httpx # git dependencies
# use uv sync or uv venv to make venv
uv run <script name>.pyI am periodically trying different tools (listed on PyPA) using the latest practices. Use a pyproject.toml, not setup.py.
Below are some working notes:
| package | comments | example repo |
|---|---|---|
| setuptools | Default | |
| hatch | Not bad, although need to enter dependencies manually (unless using uv). | tinygp |
| flit | Why aren’t more people using this? Hardly any projects are not pure python. This simplifies things massively. | bayeux |
| poetry | I’ve used it a lot. It’s easy but it’s kind of slow… to resolve dependencies. | GPJax |
| pdm | Not yet used | |
| uv | drop in replacement for pip (and soon everything else). rapid | uv |
standalone scripts
We can declare the dependencies for a script in the script itself via uv.