Simon Willison’s Weblog

uv: Unified Python packaging (via) Huge new release from the Astral team today. uv 0.3.0 adds a bewildering array of new features, as part of their attempt to build "Cargo, for Python".

It's going to take a while to fully absorb all of this. Some of the key new features are:

• uv tool run cowsay, aliased to uvx cowsay - a pipx alternative that runs a tool in its own dedicated virtual environment (tucked away in ~/Library/Caches/uv), installing it if it's not present. It has a neat --with option for installing extras - I tried that just now with uvx --with datasette-cluster-map datasette and it ran Datasette with the datasette-cluster-map plugin installed.
• Project management, as an alternative to tools like Poetry and PDM. uv init creates a pyproject.toml file in the current directory, uv add sqlite-utils then creates and activates a .venv virtual environment, adds the package to that pyproject.toml and adds all of its dependencies to a new uv.lock file (like this one). That uv.lock is described as a universal or cross-platform lockfile that can support locking dependencies for multiple platforms.
• Single-file script execution using uv run myscript.py, where those scripts can define their own dependencies using PEP 723 inline metadata. These dependencies are listed in a specially formatted comment and will be installed into a virtual environment before the script is executed.
• Python version management similar to pyenv. The new uv python list command lists all Python versions available on your system (including detecting various system and Homebrew installations), and uv python install 3.13 can then install a uv-managed Python using Gregory Szorc's invaluable python-build-standalone releases.

It's all accompanied by new and very thorough documentation.

The paint isn't even dry on this stuff - it's only been out for a few hours - but this feels very promising to me. The idea that you can install uv (a single Rust binary) and then start running all of these commands to manage Python installations and their dependencies is very appealing.

If you’re wondering about the relationship between this and Rye - another project that Astral adopted solving a subset of these problems - this forum thread clarifies that they intend to continue maintaining Rye but are eager for uv to work as a full replacement.

# 20th August 2024, 10:45 pm / packaging, python, rust, rye, uv, astral

Writing your pyproject.toml (via) When I started exploring pyproject.toml a year ago I had trouble finding comprehensive documentation about what should go in that file.

Since then the Python Packaging Guide split out this page, which is exactly what I was looking for back then.

# 20th August 2024, 12:12 am / packaging, python

cibuildwheel 2.20.0 now builds Python 3.13 wheels by default (via)

CPython 3.13 wheels are now built by default […] This release includes CPython 3.13.0rc1, which is guaranteed to be ABI compatible with the final release.

cibuildwheel is an underrated but crucial piece of the overall Python ecosystem.

Python wheel packages that include binary compiled components - packages with C extensions for example - need to be built multiple times, once for each combination of Python version, operating system and architecture.

A package like Adam Johnson’s time-machine - which bundles a 500 line C extension - can end up with 55 different wheel files with names like time_machine-2.15.0-cp313-cp313-win_arm64.whl and time_machine-2.15.0-cp38-cp38-musllinux_1_2_x86_64.whl.

Without these wheels, anyone who runs pip install time-machine will need to have a working C compiler toolchain on their machine for the command to work.

cibuildwheel solves the problem of building all of those wheels for all of those different platforms on the CI provider of your choice. Adam is using it in GitHub Actions for time-machine, and his .github/workflows/build.yml file neatly demonstrates how concise the configuration can be once you figure out how to use it.

The first release candidate of Python 3.13 hit its target release date of August 1st, and the final version looks on schedule for release on the 1st of October. Since this rc should be binary compatible with the final build now is the time to start shipping those wheels to PyPI.

# 6th August 2024, 10:54 pm / continuous-integration, packaging, pypi, python, adam-johnson

npm install everything, and the complete and utter chaos that follows (via) Here’s an experiment which went really badly wrong: a team of mostly-students decided to see if it was possible to install every package from npm (all 2.5 million of them) on the same machine. As part of that experiment they created and published their own npm package that depended on every other package in the registry.

Unfortunately, in response to the leftpad incident a few years ago npm had introduced a policy that a package cannot be removed from the registry if there exists at least one other package that lists it as a dependency. The new “everything” package inadvertently prevented all 2.5m packages—including many that had no other dependencies—from ever being removed!

# 16th March 2024, 5:18 am / packaging, security, npm

Python packaging must be getting better—a datapoint (via) Luke Plant reports on a recent project he developed on Linux using a requirements.txt file and some complex binary dependencies—Qt5 and VTK—and when he tried to run it on Windows... it worked! No modifications required.

I think Python’s packaging system has never been more effective... provided you know how to use it. The learning curve is still too high, which I think accounts for the bulk of complaints about it today.

# 22nd January 2024, 6:06 pm / luke-plant, packaging, python, windows

My User Experience Porting Off setup.py (via) PyOxidizer maintainer Gregory Szorc provides a detailed account of his experience trying to figure out how to switch from setup.py to pyproject.toml for his zstandard Python package.

This kind of detailed usability feedback is incredibly valuable for project maintainers, especially when the user encountered this many different frustrations along the way. It’s like the written version of a detailed usability testing session.

# 31st October 2023, 7:57 pm / packaging, python, usability

Dependency Management Data (via) This is a really neat CLI tool by Jamie Tanna, built using Go and SQLite but with a feature that embeds a Datasette instance (literally shelling out to start the process running from within the Go application) to provide an interface for browsing the resulting database.

It addresses the challenge of keeping track of the dependencies used across an organization, by gathering them into a SQLite database from a variety of different sources—currently Dependabot, Renovate and some custom AWS tooling.

The “Example” page links to a live Datasette instance and includes video demos of the tool in action.

# 11th August 2023, 3:54 pm / cli, packaging, sqlite, datasette

Rye. Armin Ronacher's take on a Python packaging tool. There are a lot of interesting ideas in this one - it's written in Rust, configured using pyproject.toml and has some very strong opinions, including completely hiding pip from view and insisting you use rye add package instead. Notably, it doesn't use the system Python at all: instead, it downloads a pre-compiled standalone Python from Gregory Szorc's python-build-standalone project - the same approach I used for the Datasette Desktop Electron app.

Armin warns that this is just an exploration, with no guarantees of future maintenance - and even has an issue open titled Should Rye exist?

# 24th April 2023, 4:02 am / armin-ronacher, packaging, pypi, python, rust, rye

Introducing PyPI Organizations. Launched at PyCon US today: Organizations allow packages on the Python Package Index to be owned by a group, not an individual user account. “We’re making organizations available to community projects for free, forever, and to corporate projects for a small fee.”—this is the first revenue generating PyPI feature.

# 23rd April 2023, 8:29 pm / open-source, packaging, pypi, python, psf

Boring Python: code quality. James Bennett provides an opinionated guide to setting up Python tools for linting, code formatting and and other code quality concerns. Of particular interest to me is his section on packaging checks, which introduces a whole bunch of new-to-me tools that can help avoid accidentally shipping broken packages to PyPI.

# 20th December 2022, 7:55 pm / james-bennett, packaging, python

How to create a Python package in 2022 (via) Fantastic tutorial on modern Python packaging by Rodrigo Girão Serrão. I’ve been meaning to figure out Poetry for a while now and this gave me exactly the information I needed to start figuring it out. Great coverage of GitHub Actions, Tox and pre-commit as well.

# 15th October 2022, 10:10 pm / packaging, python, github-actions

Should You Use Upper Bound Version Constraints? (via) Should you pin your library's dependencies using "click>=7,<8" or "click~=7.0"? Henry Schreiner's short answer is no, and his long answer is an exhaustive essay covering every conceivable aspect of this thorny Python packaging problem.

# 5th September 2022, 5:42 pm / packaging, pypi, python

Packaging Python Projects with pyproject.toml. I decided to finally figure out how packaging with pyproject.toml works—all of my existing projects use setup.py. The official tutorial from the Python Packaging Authority (PyPA) had everything I needed.

# 29th July 2022, 11:18 pm / packaging, python

Deno is a Browser for Code (via) One of the most interesting ideas in Deno is that code imports are loaded directly from URLs—which can themselves depend on other URL-based packages. On first encounter it feels wrong—obviously insecure. Deno contributor Kitson Kelly provides a deeper exploration of the idea, and explains how the combination of caching and lock files makes it no less secure than code installed from npm or PyPI.

# 29th May 2020, 2:36 am / packaging, deno

Why you should use `python -m pip` (via) Brett Cannon explains why he prefers “python -m pip install...” to “pip install...”—it ensures you always know exactly which Python interpreter environment you are installing packages for. He also makes the case for always installing into a virtual environment, created using “python -m venv”.

# 2nd November 2019, 4:41 pm / packaging, python, brett-cannon

Hynek Schlawack: Testing & Packaging (via) “How to ensure that your tests run code that you think they are running, and how to measure your coverage over multiple tox runs (in parallel!)”—Hynek makes a convincing argument for putting your packaged Python code in a src/ directory for ease of testing and coverage.

# 22nd May 2018, 10:12 pm / packaging, python, testing, hynek-schlawack

What to do when PyPI goes down. My deployment scripts tend to rely on PyPI these days (they install dependencies in to a virtualenv) which makes me distinctly uncomfortable. Jacob explains how to use the PyPI mirrors that are starting to come online, but that won’t help if the PyPI listing links to an externally hosted file which starts to 404, as happened with the python-openid package quite recently (now fixed). The comments on the post discuss workarounds, including hosting your own PyPI mirror or bundling tar.gz files of your dependencies with your project.

# 21st July 2010, 10:19 am / deployment, jacob-kaplan-moss, packaging, pip, pypi, python, setuptools, recovered

A history of Python packaging. A comprehensive history by Martijn Faassen, who argues that the existing set of tools tools works fine and has been working fine for several years.

# 10th November 2009, 8:48 pm / distutils, martijnfaassen, packaging, python, setuptools

Developing Django apps with zc.buildout. Jacob went ahead and actually documented one of Python’s myriad of packaging options.

# 16th April 2009, 9:50 am / jacob-kaplan-moss, packaging, python, zcbuildout

Dive into Python 3. Mark Pilgrim’s seminal work taught me Python nearly eight years ago. Now he’s updating it to cover Python 3. It’s just a table of contents at the moment, but the chapter on “Packaging Python libraries” has me very excited.

# 26th January 2009, 6:10 pm / diveintopython, mark-pilgrim, packaging, python, python3

A Few Corrections To “On Packaging”. From Ian Bicking.

# 14th December 2008, 11:07 pm / ian-bicking, james-bennett, packaging, python