Python Package Management with Rye

“I built various iterations of this over the last three years out of personal interest, but I was cautious about publishing anything out of fear of adding yet another tool into the mix. Right now I’m reaching out to other people in the packaging ecosystem to see if there is a way to work together towards something, and I’m trying to get a better public description out of how I wish packaging in the Python ecosystem could work to align people on a vision.

Dependency Hell

I started my Python package management journey years ago using pip, then more recently I embraced Anaconda and conda more fully (particularly with the “conda-forge” repository) to resolve complex dependencies along with system/binary dependencies. Recently, when attempting to update our team’s standard Python docker image with the latest versions of the packages we use, and include some new ones, it appears that relying on conda and conda-forge is untenable: I have been unable to resolve the appropriate set of versions for the scientific Python packages our team require for our work. I have moved back to pip for packages which are not provided in the default Anaconda repository. pip has and continues to make a number of improvements, and had no problem providing our extra dependencies.

Installing NetCDF Python Packages

I was trying to remember how I have installed netCDF4 and related libraries for Python, and what I need to do differently for Windows systems vs. the Linux systems I usually use.

On Linux, sometimes I use the system netCDF C libaries, but often I compile and install specific versions of HDF5 and netCDF4 from scratch. Here is how I have built netCDF for various Docker container images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Build HDF5
cd hdf5-x.y.x
./configure --prefix=/usr/local --enable-shared --enable-hl
make
make install
cd ..

# Built NetCDF4
cd netcdf-x.y.z
LDFLAGS=-L/usr/local/lib
CPPFLAGS=-I/usr/local/include
./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen
make
make install
cd ..

# NetCDF4 Fortran
cd netcdf-fortran-x.y.z
./configure --enable-shared --prefix=/usr/local
make
make install
cd ..

netcdf4-python #

I typically try to use pip to install Python libraries if I can. Use pip if you need to, but I am using conda and conda-forge as much as possible now, in fact by using conda, the above compilation steps are usually not necessary as far as I know. See below.