Installing the new Go-based Fabric LLM cli

Fabric’s installation instructions from its website make it appear that installing this new version rewritten in the Go language should be easier than it was with the original Python implementation.

Although pipx made things relatively easy before, I ran into trouble when my installed version of Go did not meet the requirements of the newer Fabric package.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Old Python-based implementation
# git clone https://github.com/danielmiessler/fabric.git && # Clone Fabric to your computer
# 	pushd fabric &&
# 	pipx install . &&
# 	exec zsh && fabric --help

# New go version
# Install Fabric directly from the repo

go install github.com/daniel/miessler/fabric@latest

My solution was to try out PKGX which was something that I had been looking for an excuse to try.

Save CT Images from *Scan of the Month*

I love seeing the monthly updates on Scan of the Month, a website that posts detailed CT scan renderings for selected themes. This website is a portfolio created by CT device company Lumafield.

Lumafield is a company that was started by a team of engineers who had firsthand experience with trying to solve hard problems and being let down by the tools on hand. Lumafield is also a company, that while in stealth-mode, decided the technology they were building was just too dang cool to keep quiet about.

NWS12 to NWS13

tool to convert OWI’s NWS12 formatted ascii files (fort.22{1,2}) to OWI’s NWS13 NetCDF4

Artistic Overlapping Timeseries Plot

Python Package Management with Rye

“I built various iterations of this over the last three years out of personal interest, but I was cautious about publishing anything out of fear of adding yet another tool into the mix. Right now I’m reaching out to other people in the packaging ecosystem to see if there is a way to work together towards something, and I’m trying to get a better public description out of how I wish packaging in the Python ecosystem could work to align people on a vision.

Reproducible Scientific Python Using Containers

Use Microsoft’s VSCode editor (code), Docker Containers, and other open-source tools for scientific Python software collaboration, development, and use on Linux and Windows. Securely connect offices, remote workers, storage resources, compute resources, and the cloud with Tailscale as a replacement for traditional VPN.

System Packages #

VSCode Editor #

Download Visual Studio Code from Microsoft.

For a Debian-based GNU/Linux distribution like Ubuntu or Pop OS, the .deb can be installed with sudo dpkg -i code_$version_amd64.deb.

Installing Messenger For Multi-node Task-Spooler

Messenger Server #

On a base apt-based distro install the Task-Spooler Messenger Server:

1
2
3
4
5

sudo apt install -y libboost-dev libboost-system-dev libboost-thread-dev libboost-filesystem-dev

pushd messenger/messenger-server
sudo ./install
popd

Configure the messenger-server ms_server by creating ~/.hosts_ports with a port for listening to client messages.

1

echo "xxx $THISHOSTPORT" >> ~/.hosts_ports

Start the server with the command ms_server.

MDAL

The Mesh Data Abstraction Library (MDAL) out of OSGeo is a “translator library” for many common conventions found in meteorology and hydrology. The library supports data found in Grib and NetCDF encoded as NetCDF Climate and Forecast Metadata Conventions (CF) or Unstructured Grid Conventions (CF/UGRID) and represented as geospatial mesh data. In addition to supporting mature and well-defined spatial data encodings in self-describing files, the library also supports numerous model-specific formats including Telemac, HEC-RAS, and TUFLOW.

Oilspill Visualizaton

Found another set of animations of particle model output from some past work. The image below is a snapshot during the early part of the output of a simulation of the Deepwater Horizon oilspill, animations of which were ultimately used in parts of the court cases.

Python’s visvis library used to render the results of an oilspill simulation:

Two Ways To Run Python *In* The Browser

JupyterLite #

WebAssembly (WASM) is changing the game with regards to Python running legitimately in the browser…

JupyterLite is a Jupyter Notebook/Lab and Ipython implementation based on Pyodide.

I caught wind of JupyterLite from this blog post.

Easily embed a console, a notebook, or a fully-fledged IDE on any web page.

In a previous blog post, we announced JupyterLite, a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels.

Another Example Visualization of LarvaMap Output

Found another old animation sample that was uploaded to YouTube. Again, Python’s visvis library used to render LarvaMap model output.

While visvis is not longer being actively developed, it is still an incredibly powerful and efficient opengl toolkit for Python.

Example 3D Visualization of Lagrangian Particle Behavior

Found an old animation while cleaning up my computer. Python’s visvis library used to render LarvaMap model output.

Projecting Rasters for Plotly Mapbox Plots

I like using Plotly to create interactive plots and maps straight from Python. I find it more intuitive than Bokeh’s API. When combined with Plotly’s sister library Dash, the pair can be very powerful in creating both interactive and dynamic web based tools and visualizations.

While Bokeh has some native Datashader integration, including in the mapping components, the Plotly Mapbox mapping backend relies on Mapbox’s own image overlay capability to display arbitrary Datashader outputs. Plotly has very helpful documentation and includes this use case as an example.

Dependency Hell

I started my Python package management journey years ago using pip, then more recently I embraced Anaconda and conda more fully (particularly with the “conda-forge” repository) to resolve complex dependencies along with system/binary dependencies. Recently, when attempting to update our team’s standard Python docker image with the latest versions of the packages we use, and include some new ones, it appears that relying on conda and conda-forge is untenable: I have been unable to resolve the appropriate set of versions for the scientific Python packages our team require for our work. I have moved back to pip for packages which are not provided in the default Anaconda repository. pip has and continues to make a number of improvements, and had no problem providing our extra dependencies.

Reproducing Conda Environments

This short summary is based on the Anaconda blog post here https://www.anaconda.com/moving-conda-environments/. The original blog post is a great high-level summary for the various methods in conda for reproducing environments.

• OS and platform specific (pulls from repos)

  # On source environment:
  conda list --explicit > spec-list.txt
  

  # New conda environment:
  conda create --name new_env_name --file spec-list.txt
  

• Different platforms and OS (pulls from repos, also includes pip installed packages)

  # On source environment:
  conda env export > env.yml
  

  # New conda environment:
  conda env create -f env.yml
  

• Platform and OS specific, no internet on target

Decompiling Python .pyc Files

Have you accidentally deleted an important Python source file or are looking to inspect the contents of a .pyc file that has been provided to you? Luckily .pyc files contain enough information to reproduce the corresponding .py file. NB: this only applies to Python2. You won’t get the original comments or the original formatting, and there may be a few tweaks you need to do for the new .py file to be completely valid–but this can be a savior for some unexpected loss of source files if the .pyc files still exist.

Installing NetCDF Python Packages

I was trying to remember how I have installed netCDF4 and related libraries for Python, and what I need to do differently for Windows systems vs. the Linux systems I usually use.

On Linux, sometimes I use the system netCDF C libaries, but often I compile and install specific versions of HDF5 and netCDF4 from scratch. Here is how I have built netCDF for various Docker container images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Build HDF5
cd hdf5-x.y.x
./configure --prefix=/usr/local --enable-shared --enable-hl
make
make install
cd ..

# Built NetCDF4
cd netcdf-x.y.z
LDFLAGS=-L/usr/local/lib
CPPFLAGS=-I/usr/local/include
./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen
make
make install
cd ..

# NetCDF4 Fortran
cd netcdf-fortran-x.y.z
./configure --enable-shared --prefix=/usr/local
make
make install
cd ..

netcdf4-python #

I typically try to use pip to install Python libraries if I can. Use pip if you need to, but I am using conda and conda-forge as much as possible now, in fact by using conda, the above compilation steps are usually not necessary as far as I know. See below.

sci-wms

https://slides.com/alexandercrosby/sci-wms

LarvaMap

LarvaMap is a distributed parallel lagrangian transport model for larvae that includes customizable biological behavior.

The idea behind LarvaMap was to make it easy for researchers everywhere to use sophisticated larval transport models to explore and test hypotheses about the early life of marine organisms.

LarvaMap integrates four components: an ocean circulation model, a larval behavior library, a python Lagrangian particle model, and a web-system for running the transport models.