Package

9 May 2023

Python Package Management with Rye

“I built various iterations of this over the last three years out of personal interest, but I was cautious about publishing anything out of fear of adding yet another tool into the mix. Right now I’m reaching out to other people in the packaging ecosystem to see if there is a way to work together towards something, and I’m trying to get a better public description out of how I wish packaging in the Python ecosystem could work to align people on a vision.

3 Apr 2023

Reproducible Scientific Python Using Containers

Use Microsoft’s VSCode editor (code), Docker Containers, and other open-source tools for scientific Python software collaboration, development, and use on Linux and Windows. Securely connect offices, remote workers, storage resources, compute resources, and the cloud with Tailscale as a replacement for traditional VPN.

System Packages #

VSCode Editor #

Download Visual Studio Code from Microsoft.

For a Debian-based GNU/Linux distribution like Ubuntu or Pop OS, the .deb can be installed with sudo dpkg -i code_$version_amd64.deb.

29 Mar 2023

MDAL

MDAL Logo

The Mesh Data Abstraction Library (MDAL) out of OSGeo is a “translator library” for many common conventions found in meteorology and hydrology. The library supports data found in Grib and NetCDF encoded as NetCDF Climate and Forecast Metadata Conventions (CF) or Unstructured Grid Conventions (CF/UGRID) and represented as geospatial mesh data. In addition to supporting mature and well-defined spatial data encodings in self-describing files, the library also supports numerous model-specific formats including Telemac, HEC-RAS, and TUFLOW.

27 Mar 2023

Trivy

The many layers of software dependencies within containers can add extra security considerations. Luckily there are a number of tools like Trivy and Docker scan to scan Docker images for vulnerabilities.

[Trivy is a] Scanner for vulnerabilities in container images, file systems, and Git repositories, as well as for configuration issues and hard-coded secrets

Trivy can be installed with the following commands on Debian-based Linux distros, including WSL environments.

23 Aug 2018

Reproducing Conda Environments

This short summary is based on the Anaconda blog post here https://www.anaconda.com/moving-conda-environments/. The original blog post is a great high-level summary for the various methods in conda for reproducing environments.

  • OS and platform specific (pulls from repos)

    # On source environment:
    conda list --explicit > spec-list.txt
    
    # New conda environment:
    conda create --name new_env_name --file spec-list.txt
    
  • Different platforms and OS (pulls from repos, also includes pip installed packages)

    # On source environment:
    conda env export > env.yml
    
    # New conda environment:
    conda env create -f env.yml
    
  • Platform and OS specific, no internet on target

17 Mar 2018

Installing NetCDF Python Packages

I was trying to remember how I have installed netCDF4 and related libraries for Python, and what I need to do differently for Windows systems vs. the Linux systems I usually use.

On Linux, sometimes I use the system netCDF C libaries, but often I compile and install specific versions of HDF5 and netCDF4 from scratch. Here is how I have built netCDF for various Docker container images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Build HDF5
cd hdf5-x.y.x
./configure --prefix=/usr/local --enable-shared --enable-hl
make
make install
cd ..

# Built NetCDF4
cd netcdf-x.y.z
LDFLAGS=-L/usr/local/lib
CPPFLAGS=-I/usr/local/include
./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen
make
make install
cd ..

# NetCDF4 Fortran
cd netcdf-fortran-x.y.z
./configure --enable-shared --prefix=/usr/local
make
make install
cd ..

netcdf4-python #

I typically try to use pip to install Python libraries if I can. Use pip if you need to, but I am using conda and conda-forge as much as possible now, in fact by using conda, the above compilation steps are usually not necessary as far as I know. See below.