Publishing portable Python bindings for C, C++ and Fortran codes

Pieter Pas, Panagiotis Patrinos

Space:  Next slide
Shift+Space:  Previous slide
←/→:  Previous/next chapter
Esc:  Overview  F11:  Full screen

Outline

  1. Motivation
  2. Python bindings
  3. Building and installing Python packages
  4. Example project
  5. Publishing Python packages
  6. Installing build dependencies
  7. Conclusion

Motivation

Why provide Python bindings?

  • Fast prototyping and iteration
    • Implement performance-critical routines in a native language
    • Load and process data in Python
    • Call performance-critical routines from Python
    • Plot results in Python
  • Leverage existing Python libraries and frameworks
  • Reach a wider audience
    • Python has become a lingua franca in many scientific and engineering fields
    • Easier for users to try out
    • Easier for reviewers to check research code
  • With modern tools, creating and distributing Python bindings
    is easier than ever

Writing Python bindings

Writing Python bindings: the Python C API

Writing Python bindings: the Python C API

  • Low-level
    • Manual parsing of function arguments
    • Manual type checking
    • Manual error handling
    • Manual memory management
  • Boilerplate
    • Method tables, module definitions ...
    • Reference counting
  • docs.python.org/3/extending/extending.html

Writing Python bindings: higher-level tools

Write bindings in C++:

Import the compiled module in Python:

Writing Python bindings: higher-level tools

  • pybind11 (github.com/pybind/pybind11)
    • High-level bindings ─ module.def("add", add, ...)
    • Automatic conversion and type checking of function arguments
    • Automatic conversion of return types to Python objects
    • Type casters for standard library types and Eigen matrices
    • Automatic memory management
    • Exceptions
    • Battle-tested and widely used
  • nanobind (github.com/wjakob/nanobind)
    • Modern successor to pybind11
    • Smaller and faster
    • Nice ndarray abstractions

Writing Python bindings: higher-level tools

  • SWIG (github.com/swig/swig)
    • Code generation based on interface file
    • Supports many target languages (Python, Matlab, R, Java, C#, ...)
  • ctypes
    • Load shared libraries at runtime
    • Use existing C API
    • Manual type conversions and memory management

Writing Python bindings: lifetime management

  • Avoid memory leaks and use-after-free bugs
  • Lifetime management in C++ first (RAII)
  • Let pybind11 or nanobind manage lifetimes for bindings


Example from the QPALM solver (github.com/kul-optec/QPALM)

Writing Python bindings: lifetime management

  • pybind11 return value policies

Building and installing Python packages

Building and installing Python packages

  • Metadata defined in pyproject.toml
    • Package name, version, author, description, license ...
    • Dependencies
    • Build system requirements
  • Build and install using pip install .
    • Builds a Wheel package as described by pyproject.toml
    • Installs the package into the current Python environment

Building and installing Python packages: pyproject.toml

Building and installing Python packages: Build backends

  • Build backends
    • Underlying code executed by Pip to build a package
    • Hooks with interfaces defined by PEP 517
  • Many choices
  • Modern build backends replace setuptools
    • No more setup.py, setup.cfg

Example project

Example project: directory structure

Example project: project metadata

pyproject.toml

Example project: C++ Python bindings

src/ext/add_module.cpp

Example project: CMake build system

CMakeLists.txt

Example project: Python modules

src/py_build_cmake_example_project/__init__.py

src/py_build_cmake_example_project/sub_package/__init__.py

src/py_build_cmake_example_project/sub_package/sub.py

Example project: Build and install

  • Build binary Wheel distribution: python -m build --wheel
  • Create source distribution: python -m build --sdist
  • Install package in the current environment: pip install .
  • Install package in developer/editable mode: pip install -e .

py-build-cmake

  • What does py-build-cmake actually do?
    • Implements the PEP 517 build backend API for interoperability with packaging tools like Pip
    • Processes metadata from pyproject.toml to determine how to build and package the project
    • Installs any C/C++ build dependencies (beta)
    • Invokes CMake to configure, build and install the project
    • Sets the necessary CMake hints to select the appropriate version of Python
    • Installs the Python source files, extension modules and other resources into a Wheel package
    • Provides hooks for editable installs (automatic re-building on import)

Publishing Python packages

Publishing Python packages to the Python Package Index (PyPI)

  • Python Package Index (PyPI)
    • Main repository for publishing and distributing Python packages
    • Hosts both source distributions and pre-compiled Wheel packages
    • Default repository for Pip and other package managers
  • Publishing to PyPI
    • Write a CI/CD workflow
      (e.g. GitHub actions)
    • Register the workflow with PyPI
      (Account > Publishing > Add a new pending publisher)
    • Upload packages from CI/CD workflow using trusted publishing (docs.pypi.org/trusted-publishers)

Building Wheel packages for most platforms

Building Wheel packages for most platforms

  • Need binary Wheel package for all combinations of Python versions, operating systems and architectures
  • Use cibuildwheel (github.com/pypa/cibuildwheel)
  • Leverage CI/CD matrix builds
  • Cache compilation across different Python versions (e.g. sccache)
  • Consider cross-compilation
    • Uniform workflow from Raspberry Pi to HPC clusters
    • Portable GLIBC and Linux version requirements
    • Recent compiler releases
    • No accidental dependencies on libraries that happen to be present on the build machine

Installing build dependencies

Installing build dependencies

CMakeLists.txt

Installing build dependencies

  • Where to locate dependencies?
    • Manual or system-wide installation
    • Installation as Python build requirement
    • Automated installation using cibuildwheel pre-build hook
    • Automated installation using a C++ package manager (e.g. Conan)

  • System libraries may not be portable  → prefer installation from source
  • Users may be building from an sdist  → prefer automated installation
  • Reproducibility  → prefer a package manager with lock files

Installing build dependencies

  • Experimental Conan integration in py-build-cmake
    • pyproject.toml
    conanfile.txt

Conclusion