Alias to get IP

List active network interfaces with addresses

Set Linux XDG Default App To A Flatpak

The one where we set the default system web browser to LibreWolf installed as a Flatpak

Install *timewarrior-dataframe* using *pipx* for pivot tables of *timewarrior* summaries

Things to know:

• Timewarrior (timew) is a command line tool for tracking time.
• Timewarrior-Dataframe is an extension processes timew export output using the Pandas Python module.

Install #

Timewarrior can be installed on recent Ubuntu releases from the apt repositories. It can also be found in a number of other distro’s packaging systems.

1
2

# Ubuntu (and/or Debian?)
sudo apt install timewarrior

pipx can be used to install the timewarrior-dataframe’s twdf command from it’s git repository.

Adding ZWave devices to my Home Assistant smart home

There are some quirks related to running standalone Home Assistant (HA) from a Docker container. I found that the route to enable support and add a device using ZWave is not well documented for this HA configuration, especially for someone new to and unfamiliar with ZWave.

Running ZWave JS server #

My biggest bugaboo was that…

Without a ZWave hub on the network, a standalone ZWave server is needed to mediate the ZWave network and communicate with HA. I was able to run an official zwavejs/zwave-js-ui container image, and pass my ZWave USB dongle device through to “/dev/zwave”.

Installing the new Go-based Fabric LLM cli

Fabric’s installation instructions from its website make it appear that installing this new version rewritten in the Go language should be easier than it was with the original Python implementation.

Although pipx made things relatively easy before, I ran into trouble when my installed version of Go did not meet the requirements of the newer Fabric package.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Old Python-based implementation
# git clone https://github.com/danielmiessler/fabric.git && # Clone Fabric to your computer
# 	pushd fabric &&
# 	pipx install . &&
# 	exec zsh && fabric --help

# New go version
# Install Fabric directly from the repo

go install github.com/daniel/miessler/fabric@latest

My solution was to try out PKGX which was something that I had been looking for an excuse to try.

"teethis" script is one of the most useful tools I've created

tee is a GNU tool to piggy back stout off of a pipe. The following example shows the basic pattern for using it, where the stout is printed to the terminal and written to a log file.

1

mycommand | tee mycommand.log

“teethis” script #

I frequently want to run shell scripts where I can see both the stdout and stderr immediately while preserving the output to refer to later. While the tee command to do this is relatively simple, it can be a pain to type over and over again, so I created the following teethis script.

`apropos` is an overlooked Linux command

Search for Linux command that does what you need #

Sometimes the challenge of using the Linux command line is knowing what commands you need to do certain things. The apropos command lets you search the available commands by matching key words.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

❯ apropos csv
# bench_wcsv (3tcl)    - bench::out::csv - Formatting benchmark results as CSV
# csv (3tcl)           - Procedures to handle CSV data.
# csv2rec (1)          - csv to rec converter
# rec2csv (1)          - rec to csv converter


❯ apropos environment 
# 0desktop (1)         - add programs to the desktop environment
# 30-systemd-environment-d-generator (8) - Load variables specified by environment.d
# byobu-janitor (1)    - script for cleaning and upgrading environment after upgrades
# byobu-reconnect-sockets (1) - Sourcable script that updates GPG_AGENT_INFO and DBUS_SESSION_BUS_ADDRESS in the environment
# check-language-support (1) - returns the list of missing packages in order to provide a complete language environment
# clearenv (3)         - clear the environment
# dbus-update-activation-environment (1) - update environment used for D-Bus session services
# Dpkg::Build::Env (3perl) - track build environment
# env (1)              - run a program in a modified environment
# environ (7)          - user environment
# pam_env.conf (5)     - the environment variables config files
# ...

The command checks all of the installed man pages and package descriptions, however there are circumstances where this could leave out some relevant commands.

Feynman on Responsibility

“You have no responsibility to live up to what other people think you ought to accomplish. I have no responsibility to be like they expect me to be. It’s their mistake, not my failing.”

Simon Pett About Being a Geek

Being a geek is all about being honest about what you enjoy and not being afraid to demonstrate that affection. It means never having to play it cool about how much you like something. It’s basically a license to proudly emote on a somewhat childish level rather than behave like a supposed adult. Being a geek is extremely liberating.

Toni Morrison on Empowerment

“ If you have some power, then your job is to empower somebody else”

Traditional Girl's Skipping Rhyme

I have a deadly nightshade, So twisted does it grow…

Sharing **vscode** extensions

Microsoft created a modular text editor (referred to as vscode or just code on the command line) that is easy to install and use across platforms including integration with Docker containers and WSL2.

Install #

Use this link and follow the instructions to install vscode for your platform/OS: https://code.visualstudio.com/download

Ubuntu #

For Ubuntu, choose either CLI/x64 or .deb/x64 options.

Basic usage #

• Icons along left-side: Open different editor environments or side-bars including extension browser, file explorer, and git information.
• Command palette: Most actions or options are available from either shortcut-key-combinations or the command palatte which is a fuzzy-matching command search accessible with CTL+SHIFT+p.

Extensions #

vscode provides a mechanism to install official and community extensions to tailor the editor for specific code/development tasks and support for enhanced features when editing a multitude of plain text formats.

The IndieWeb

According to the eponymous IndieWeb.org, “IndieWeb” is “a people-focused alternative to the ‘corporate web’.” In 2024, it’s not necessary to describe the cons associated with the on-going centralization and corporatization of the internet.

The primary tenets of this alternative to the current Web2.0 trends are:

1. Own your data.
2. Use and publish with focus on human visibility before machine-readability.
3. Make what you need.
4. Use what you make.
5. Document it, your stuff, the journey, whatever.
6. Open source.
7. UX/Design before plumbing.
8. Make it modular.
9. Design to last.
10. Plurality of components, approaches, and implementations.
11. Keep it weird, interesting, and fun!

In support of these tenets, the movement promotes small modular components over monolithic and monocultural stacks; creating “a community of independent and personal websites connected by open standards and based on the principles of: owning your domain and using it as your primary online identity, publishing on your own site first (optionally elsewhere), and owning your content.”

Run Large Language Models Locally With *ollama.ai's* Docker Image

ollama is an open-source framework for self-hosting large language models (LLM) similar to ChatGPT or Google’s Bard. The official Docker image makes it painless to host any number of the supported models.

Shell script to get an Ollama model running #

The example “run_ollama_container.sh” script(s) below take the LLM model name as an argument (assigned to $LLMMODEL), but the pull and run docker exec commands can be called additional times to start other models as well within the same running container.

Quick and dirty geographic longitude offset for GIS

Convert GIS data with Lat/Lon (EPSG:4326/WGS84) coordinates with longitude in the range 0-360 to -360-0 to cover -180-180 in a GIS envrionments map with this quick and dirty custom proj-style projection string.

The custom projection definition is +proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +pm=-360 +no_defs

This can be used as follows,

1. load a spatial dataset with Lons defined in the 0-360 range,
2. duplicate or load another layer with the same dataset,
3. set layer in 1. CRS to WGS84 or EPSG4326,
4. and set layer in 2. to a custom CRS using the projection definition above.

This process results in the dataset covering the -360 to 360 Longitude range (and implcitly -180 to 180 as well) between the two layers which can be used in a variety of different project coordinate reference systems or projections. Switching +pm=-360 to +pm=+360 will go the opposite way from (-180-180) Lons to (0-360) if necessary, because that parameter sets or modifies the default Prime Meridian Longitude in degrees-East.

NWS13: NetCDF Replacement for NWS12 Met Inputs and Application

2020 ADCIRC User’s Group Meeting Slides

NWS13: NetCDF Wind/Pressure Inputs for ADCIRC

2018 ADCIRC User’s Group Meeting Slides

Save CT Images from *Scan of the Month*

I love seeing the monthly updates on Scan of the Month, a website that posts detailed CT scan renderings for selected themes. This website is a portfolio created by CT device company Lumafield.

Lumafield is a company that was started by a team of engineers who had firsthand experience with trying to solve hard problems and being let down by the tools on hand. Lumafield is also a company, that while in stealth-mode, decided the technology they were building was just too dang cool to keep quiet about.

What is the NVIDIA Container Toolkit?

The NVIDIA Container Toolkit is a set of utilities and libraries provided by NVIDIA to enable containers to use and share NVIDIA GPU resources. The toolkit is container agnostic, supporting several common container runtimes including Docker and Podman.

It appears to present GPU/CUDA features to software running within containers on the host with the NVIDIA GPU. The supported platforms seem to be specifically GNU/Linux distributions (Debian, Ubuntu, Centos, Fedora, Open Suse, and Amazon’s Amazon Linux AMIs) on typical amd64, arm64/aarch64 hardware.

Manually customize bash terminal prompt look

The prompt text that bash interprets and uses is set with the PS1 environment variable and is usually specified in ~/.bashrc. I have been manually tweaking my bash prompt look and feel to make some of my work project use-case tasks easier. The combination of default style changes, and specific prompt elements added for active environment or sourced environments should make terminal-based project management a little easier.

A little searching allowed me to discover this website that allows you to interactively customize the prompt contents and style with common components. https://bash-prompt-generator.org/

Automatically backup optical discs on linux

Here is a short starting-point for developing a bash script that can be used to automate optical disc ripping. At least for music and application CDs as well as DVDs. Some other cli commands will need to be added to deal with bluerays or other formats.

abcde to rip CD music #

abcde is used to rip music CDs to various formats (my configuration is set to FLAC to preserve the raw quality but in a compressed format to save space.) This command will fail for DVDs.

Notifications in GNU/Linux

Installation #

1
2

notify-send
# Command 'notify-send' not found, but can be installed with:

Installation with apt #

1
2

# debian/ubuntu
sudo apt install libnotify-bin

Installation with dnf #

1
2

# redhat/fedora
sudo dnf install libnotify

Usage #

1
2

notify-send "this is the summary"
notify-send "this is the summary" "this is the body of the notification"

Send to remote machine over SSH #

1

ssh user@host 'DISPLAY=:0 notify-send "TEST MESSAGE."'

Styling with HTML #

HTML tags are supported for styling the notification messages, even including images, although the result is not styled in the case of notifications in the rofication setup used by default in the i3wm-based regolith-desktop DE.

*Magic-Wormhole* file transfer

Warp is a Rust software package that makes sending content between computers super easy using entryption and simple one-time word-based or QR-code authentication keys. Because of the underlying protocol, transfers will occur over local LAN or internet depending on where the machines involved are located. The software allows users to send and recieve files between Linux, OSX, Windows and smart phones.

I was interested in using Warp on the command line for simple low-setup file transfers in and out of my office, but it turns out that the underlying protocol that Warp leverages is also an open-source project and it already includes a CLI (magic-wormhole).

*ugrid* ADCIRC vectors in QGIS with MDAL

use nco cli to make compliant files

NWS12 to NWS13

tool to convert OWI’s NWS12 formatted ascii files (fort.22{1,2}) to OWI’s NWS13 NetCDF4

Who knew you can make an animated GIF with *convert*

Another thing that comes up infrequently so it is hard to remember! This method of creating an animation from a set of individual images as frames is one of the easiest I have ever seen. (Get out of here FFMPEG!)

ImageMagick’s convert animation example command #

1

convert -delay {centiseconds} -loop {loop-n-times} *.png output.gif

Here is an output made with convert from .png frames using a 25 centisecond frame advancement time.

Artistic Overlapping Timeseries Plot

An Update To Apptainer Use On TACC HPC

The University of Texas’ newest supercomputer Frontera currently requires the following modules loaded for Apptainer images/containers based on the mvapich infiniband (ib) base images–particularly for any containerized programs that intend to use MPI.

Researchers moving their Apptainer images between TACC HPC systems, or building container images specifically for individual systems, may need to adapt their sbatch scripts to accommodate differences between default environments and infrastructures on the various available clusters.

Python Package Management with Rye

“I built various iterations of this over the last three years out of personal interest, but I was cautious about publishing anything out of fear of adding yet another tool into the mix. Right now I’m reaching out to other people in the packaging ecosystem to see if there is a way to work together towards something, and I’m trying to get a better public description out of how I wish packaging in the Python ecosystem could work to align people on a vision.

Reproducible Scientific Python Using Containers

Use Microsoft’s VSCode editor (code), Docker Containers, and other open-source tools for scientific Python software collaboration, development, and use on Linux and Windows. Securely connect offices, remote workers, storage resources, compute resources, and the cloud with Tailscale as a replacement for traditional VPN.

System Packages #

VSCode Editor #

Download Visual Studio Code from Microsoft.

For a Debian-based GNU/Linux distribution like Ubuntu or Pop OS, the .deb can be installed with sudo dpkg -i code_$version_amd64.deb.

Installing Messenger For Multi-node Task-Spooler

Messenger Server #

On a base apt-based distro install the Task-Spooler Messenger Server:

1
2
3
4
5

sudo apt install -y libboost-dev libboost-system-dev libboost-thread-dev libboost-filesystem-dev

pushd messenger/messenger-server
sudo ./install
popd

Configure the messenger-server ms_server by creating ~/.hosts_ports with a port for listening to client messages.

1

echo "xxx $THISHOSTPORT" >> ~/.hosts_ports

Start the server with the command ms_server.

MDAL

The Mesh Data Abstraction Library (MDAL) out of OSGeo is a “translator library” for many common conventions found in meteorology and hydrology. The library supports data found in Grib and NetCDF encoded as NetCDF Climate and Forecast Metadata Conventions (CF) or Unstructured Grid Conventions (CF/UGRID) and represented as geospatial mesh data. In addition to supporting mature and well-defined spatial data encodings in self-describing files, the library also supports numerous model-specific formats including Telemac, HEC-RAS, and TUFLOW.

Trivy

The many layers of software dependencies within containers can add extra security considerations. Luckily there are a number of tools like Trivy and Docker scan to scan Docker images for vulnerabilities.

[Trivy is a] Scanner for vulnerabilities in container images, file systems, and Git repositories, as well as for configuration issues and hard-coded secrets

Trivy can be installed with the following commands on Debian-based Linux distros, including WSL environments.

Data Catalog From Filesystem

This previous post demonstrated a way to use generic website html source to create an inventory of web-accessible files using discovered URL links. A similar methodology can be applied to search through filesystem directory tree to establish a catalog for any matching filetypes.

Use glob to recursively globstar-match filepaths #

In this case I use Python’s builtin glob and regular expression modules to list files and match extensions in the names. I used the os.path collection of utility methods to pull directory names from the full paths, but a more modern way would probably to use the builtin pathlib. Pandas is the only non-builtin package used, which could be removed if a DataFrame is not the desired output.

Examples of accessible data inventories

Simple over Standards #

Frequently the biggest hurdle to sharing data is the pre-existing organization of files. While open standards based API access can smooth over the bumps inherent to locally stored heterogeneous collections of files, the amount of effort to setup an existing API webservice to share your data can be overwhelming. (Not to mention having to roll your own in an standards complant way.)

The NOAA example below is a great example of a simple solution that can be used as a template to provide easy programatic access to any dataset.

`diff` Also Compares Directories

Short post so that I can remember this everytime I need to do something similar!

Using diff on more than individual files #

The quick and dirty explaination is that the GNU/Linux diff command has an -r flag to recursively compare two folders. The command help indicates that it is shorthand for the full --recursive flag, which might be easier to remember.

1
2
3
4
5

diff --help

# ...
  -r, --recursive                 recursively compare any subdirectories found
# ...

Example #

In the following example, “Only in” shows that particular files are only found in one of the folders. By default, matching files are not shown. If a file can be found in both folders and the two versions differ, the normal diff output is provided along with the modified times. All together these details provide a good summary of what a user might want to know when comparing two directories.

Home Assistant (HA) Smart Home

Hardware Setup #

• 4x Old Dell Optiplex 755 (only one machine necessary)
• Wired ethernet switch to home wireless router and WAN
• SONOFF Zigbee USB Dongle (this one) on one of the machines (only necessary to interact with Zigbee devices)

System Setup #

Each machine has the following:

• Ubuntu Linux 20.04
• Docker (with Swarm Cluster, Docker and Swarm not necessary for HA)
• GlusterFS shared disk across the cluster, not necessary for HA

Home Assistant Docker Setup #

Start from Linux on command line as swarm service (from the machine that has the USB dongle if necessary):

fort.15 configuration for NWS13

required changes

Interpolation Implementation for NWS13

supporting moving wind and pressure inputs

NWS13

Flexible (NetCDF) Metocean Inputs for ADCIRC

DIY Cyber Deck

Bash Globstar `**`

In recent versions of bash, the ** expression can be used to indicate matches for a particular directory while including sub-directories. This can be a big help when dealing with some messy filesystem structures.

1
2

shopt -s globstar
data/$year/**/*.nc

This example would match .nc files in $year and the folders below $year. Depending on the shell configuration or if running within a script, shopt -s globstar may be necessary to enable the capability.

WSL DNS Issue

Out of the blue a problem resolving common domains popped up while I was using a WSL Linux image.

Some internet searching led me to this:

https://gist.github.com/coltenkrauter/608cfe02319ce60facd76373249b8ca6

The Github gist and the comments had me scared, but poking around in the Linux system running under WSL indicated that /etc/resolv.conf was symlinked to some non-existing file. This was probably something that WSL is doing under the hood based on its own configuration judging by the link.

Migrating to Hugo

What’s hugo and why am I switching from jekyll?

https://gohugo.io/ #

The world’s fastest framework for building websites

Hugo is one of the most popular open-source static site generators. With its amazing speed and flexibility, Hugo makes building websites fun again.

Automatic content formatting and views based on rules #

This is really what shines about Hugo–content driven presentation and views into site contents treating everything as data that just needs to be formatted.

Oilspill Visualizaton

Found another set of animations of particle model output from some past work. The image below is a snapshot during the early part of the output of a simulation of the Deepwater Horizon oilspill, animations of which were ultimately used in parts of the court cases.

Python’s visvis library used to render the results of an oilspill simulation:

Two Ways To Run Python *In* The Browser

JupyterLite #

WebAssembly (WASM) is changing the game with regards to Python running legitimately in the browser…

JupyterLite is a Jupyter Notebook/Lab and Ipython implementation based on Pyodide.

I caught wind of JupyterLite from this blog post.

Easily embed a console, a notebook, or a fully-fledged IDE on any web page.

In a previous blog post, we announced JupyterLite, a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels.

Build ADCIRC using CMake

Gone are the days of having to use config.guess and sed replcements to customize the cmplrflags.mk for a particular HPC environment.

Here is the process to build the ADCIRC hydrodynamic model using CMake from the command line on a GNU/Linux host. The CMake infrastructure has only been available in the last few releases of ADCIRC, and this example uses v55.00 from the compressed tar archive of the code. The options specified below enable serial adcirc, distributed parallel padcirc, coupled serial adcswan (ADCIRC+SWAN), and the parallel coupled padcswan model. Standalone parallel SWAN and unstructured SWAN models are also configured, along with the adcprep ADCIRC pre-processor and other utilities.

SWAN Wave Model Container with MPI

Coinciding with the Containers at TACC training taking place today via webinar, here is a method I have used successfully to build our version of SWAN v41.10 spectral wave model in order to run with MPI on the University of Texas’ Stampede2 cluster.

Build SWAN with Docker #

The following Dockerfile will build the parallel mpi implementation of SWAN, the single-core serial SWAN model, and the SWAN hcat concatenation utility. The containerized model should be able to be used for 2D non-stationary, 1D or stationary simulations.

USB Cables for Sony Alpha Cameras

Intermittent problems getting my PC to recognize my Sony a6000 and a5100 mirrorless cameras led me to some web searching for solutions that I thought would be either software, driver, system OS, or firmware related. I found the following instructions which shed some light on the issue:

1. Insert USB-A into computer and Micro-USB into camera; and power the camera on.

2. If the computer doesn’t recognize the camera, try another cable…

Reclaim Docker WSL Disk Space

I ran out of system SSD disk space on my Win10 PC at home and tracked a large share of the usage to a vhdx virtual disk image file in a Docker Desktop folder. Below is the summary of what I found.

Docker Desktop for Windows uses WSL to manage all your images and container files and keeps them in a private virtual hard drive (VHDX) called ext4.vhdx.

It’s usually in C:\Users\YOURNAME\AppData\Local\Docker\wsl\data and you can often reclaim some of the space if you’ve cleaned up (pruned your images, etc) with Optimize-Vhd under an administrator PowerShell shell/prompt.

Another Example Visualization of LarvaMap Output

Found another old animation sample that was uploaded to YouTube. Again, Python’s visvis library used to render LarvaMap model output.

While visvis is not longer being actively developed, it is still an incredibly powerful and efficient opengl toolkit for Python.

Example 3D Visualization of Lagrangian Particle Behavior

Found an old animation while cleaning up my computer. Python’s visvis library used to render LarvaMap model output.

Bad Hacks to Find Linux Drive Device from UUID

On linux, you can’t count on the devices listed under /dev/sd* or /dev/xvd*, to have the same names or order. With the UUID (which is how /etc/fstab usually specifies how to mount the root system device), the /dev device name can be determined using the blkid command and some hacky bash.

1
2

# For a system that uses /dev/sda, etc. use 0:8 string slice for blkid
MYDEVICE=$(blkid | grep ${MYUUID}) ; echo ${MYDEVICE:0:8}  # /dev/sdc

However, you might need to do something like start a Docker container with your main system device connected using docker run --device=..., and so the following also works in this specific case looking for the device mounted at the local machine’s /.

Quickly Set Up Guacamole Using Docker

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Thanks to HTML5, once Guacamole is installed on a server, all you need to access your desktops is a web browser.

The Guacamole manual can be found here, but because the project is broken into 2 or more peices, it can take a bit of poking around and toggling between sections to find the exact steps one needs to quickly get up and running from scratch using Docker.

Projecting Rasters for Plotly Mapbox Plots

I like using Plotly to create interactive plots and maps straight from Python. I find it more intuitive than Bokeh’s API. When combined with Plotly’s sister library Dash, the pair can be very powerful in creating both interactive and dynamic web based tools and visualizations.

While Bokeh has some native Datashader integration, including in the mapping components, the Plotly Mapbox mapping backend relies on Mapbox’s own image overlay capability to display arbitrary Datashader outputs. Plotly has very helpful documentation and includes this use case as an example.

Useful Docker Patterns (On GNU/Linux)

To easily copy files from a Docker image when you dont want to start a container, you can do the following:

1

docker cp $(docker create --rm $IMAGE):$FROM_PATH $TO_PATH

Sometimes it is useful to compare the output of a command between two Docker images, especially comparing different versions of a particular image:

1
2
3

docker run --rm -ti --entrypoint=bash $IMAGE1 -c "$CMD" > /tmp/output1.tmp
docker run --rm -ti --entrypoint=bash $IMAGE2 -c "$CMD" > /tmp/output2.tmp
diff -s /tmp/output1.tmp /tmp/output2.tmp

I have both these command patterns saved as executable Bash shell scripts on my system path as cp_docker and diff_docker_cmd respectively.

Dependency Hell

I started my Python package management journey years ago using pip, then more recently I embraced Anaconda and conda more fully (particularly with the “conda-forge” repository) to resolve complex dependencies along with system/binary dependencies. Recently, when attempting to update our team’s standard Python docker image with the latest versions of the packages we use, and include some new ones, it appears that relying on conda and conda-forge is untenable: I have been unable to resolve the appropriate set of versions for the scientific Python packages our team require for our work. I have moved back to pip for packages which are not provided in the default Anaconda repository. pip has and continues to make a number of improvements, and had no problem providing our extra dependencies.

Overlooked Tools: 'watch'

There many GNU/Linux command line tools that provide continuously updated statistics or information like iotop, nload, and top. However, it is often necessary to see the output of another simple command or complex chain of tools, updated in real-time.

watch is a Linux command that does just that, and by doing so, is incredibly useful for continuously monitoring the output of any Linux terminal command. Usage of the watch command is incredibly simple, as the following example monitoring the currently running tsp task in the task-spooler queue demonstrates.

Lean Philosophy

Given my experience and education, I have always been sort of philosophically aligned with the concepts in “Lean” management, production, and software development thought to have originated at Toyota in the 90’s. Using software as example, those are:

• Eliminate Waste

  If some activity could be bypassed or the result could be achieved without it, it is waste. Partially done coding eventually abandoned during the development process is waste. Extra features like paperwork and features not often used by customers are waste. Switching people between tasks is waste. Waiting for other activities, teams, processes is waste. (from wikipedia)

Singularity From Local Docker Image

One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo.

To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.

Autumn Leaves Are Free Money

“Leaves build soil, conserve water and best of all, control weeds. By using leaves as your compost and mulch you are restoring nutrients to the soil that leaves used in order to grow…They’re like a bank account that never runs dry.”

greenwichfreepress.com: Leaves Improve Gardens Autumn-atically

I always mow my leaves into my lawn. Keep your leaves, they are free money! I also think its crazy that people pay landscapers to cut their lawn short, remove the clippings and kill the beneficial “weeds” like clover, and then further pay to water and fertilize because the lawn is too short, the soil stays bad, and the grass becomes reliant on the overapplied nutrients. We should all be trying to minimize our turf grass anyway: Lawns Are an Ecological Disaster

Overlooked Tools: task-spooler

Sometimes sophisticated job queue and cluster systems are overkill on office or individual PCs, especially for long lists of batch processing jobs that use comparatively few resources.

Use task-spooler to queue commands on Linux and control the number of simultaneous jobs for an easy way to parallelize a list of batch jobs! This tool is a great complement to some other command line workhorses like at and batch.

Task-spooler is not usually installed by default on Linux systems. It can be built by scratch after getting the source from https://vicerveza.homeunix.net/~viric/soft/ts/ or on Ubuntu (maybe other Debian based OS’s) you can use apt:

Overlooked Tools: xclip

If you have ever had a need to move the output of a Linux console command to your clipboard, xclip is the solution!

To copy a file’s contents to the standard “ctl+c/v” clipboard buffer issue you can following this command:

1

xclip -sel clip your-file-path

The most useful feature is reading from a stdin pipe:

1

head my-file-path | xclip -sel clip

…and then your ready to paste into an email or Slack chat.

Singularity Container System for HPC

At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables.

Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.

Shortcut to Website in Ubuntu 18.04 Dock

Create and populate the following file to add a dock shortcut to a website in Ubuntu using Ars Technica as an example. This will also enable the shortcut in the Gnome Shell search.

~/.local/share/applications/MyWebsite.desktop:

[Desktop Entry]
Comment=Ars Technica
Terminal=false
Name=Ars Technica
Exec=firefox https://arstechnica.com/
Type=Application
Icon=applications-internet
NoDisplay=false

To use Google Chrome instead, replace firefox with google-chrome if it is installed.

[Desktop Entry]
Comment=Ars Technica
Terminal=false
Name=Ars Technica
Exec=google-chrome https://arstechnica.com/
Type=Application
Icon=applications-internet
NoDisplay=false

Running WRF Singularity Container on TACC Systems

So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity.

I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)

Reproducing Conda Environments

This short summary is based on the Anaconda blog post here https://www.anaconda.com/moving-conda-environments/. The original blog post is a great high-level summary for the various methods in conda for reproducing environments.

• OS and platform specific (pulls from repos)

  # On source environment:
  conda list --explicit > spec-list.txt
  

  # New conda environment:
  conda create --name new_env_name --file spec-list.txt
  

• Different platforms and OS (pulls from repos, also includes pip installed packages)

  # On source environment:
  conda env export > env.yml
  

  # New conda environment:
  conda env create -f env.yml
  

• Platform and OS specific, no internet on target

Decompiling Python .pyc Files

Have you accidentally deleted an important Python source file or are looking to inspect the contents of a .pyc file that has been provided to you? Luckily .pyc files contain enough information to reproduce the corresponding .py file. NB: this only applies to Python2. You won’t get the original comments or the original formatting, and there may be a few tweaks you need to do for the new .py file to be completely valid–but this can be a savior for some unexpected loss of source files if the .pyc files still exist.

Installing NetCDF Python Packages

I was trying to remember how I have installed netCDF4 and related libraries for Python, and what I need to do differently for Windows systems vs. the Linux systems I usually use.

On Linux, sometimes I use the system netCDF C libaries, but often I compile and install specific versions of HDF5 and netCDF4 from scratch. Here is how I have built netCDF for various Docker container images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Build HDF5
cd hdf5-x.y.x
./configure --prefix=/usr/local --enable-shared --enable-hl
make
make install
cd ..

# Built NetCDF4
cd netcdf-x.y.z
LDFLAGS=-L/usr/local/lib
CPPFLAGS=-I/usr/local/include
./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen
make
make install
cd ..

# NetCDF4 Fortran
cd netcdf-fortran-x.y.z
./configure --enable-shared --prefix=/usr/local
make
make install
cd ..

netcdf4-python #

I typically try to use pip to install Python libraries if I can. Use pip if you need to, but I am using conda and conda-forge as much as possible now, in fact by using conda, the above compilation steps are usually not necessary as far as I know. See below.

Using the Blockchain for Open-access Journals?

One thing that excites me about the current buzz around blockchain technology is its use for open science. I can’t speak to the feasibility, but it seems to me that a distributed ledger could be an ideal place to publish and provide open-access to scientific research papers and articles.

If including a way to store, deliver and update supporting data, a blockchain could deliver research products that link directly to the data and analysis–providing an unparalleled level of provenance and context for research and results. Citations and work building on similar pieces of data could be connected allowing for straighforward literature searches and discovery.

sci-wms

https://slides.com/alexandercrosby/sci-wms

ugrid

Conventions Documentation #

The conventions can be found here:

https://ugrid-conventions.github.io/ugrid-conventions/

Which is maintained though the following git repository:

https://github.com/ugrid-conventions/ugrid-conventions

Check Compliance #

pyugrid can be used to parse and validate CF-UGRID metadata. Its documentation is located here:

https://pyugrid.github.io/pyugrid/

“Be kind to people, be ruthless to systems.“

             -- Michael Brooks 

in python, parse an html table into a pandas dataframe

To parse an HTML table into a pandas dataframe, you can use the pandas library's read_html function. This function will search for table elements in the HTML document and attempt to parse them into a pandas dataframe.

1 Jan 0001

LarvaMap

LarvaMap is a distributed parallel lagrangian transport model for larvae that includes customizable biological behavior.

https://slides.com/alexandercrosby/larvamap

The idea behind LarvaMap was to make it easy for researchers everywhere to use sophisticated larval transport models to explore and test hypotheses about the early life of marine organisms.

LarvaMap integrates four components: an ocean circulation model, a larval behavior library, a python Lagrangian particle model, and a web-system for running the transport models.

1 Jan 0001

Overlooked Tools: 'at'

If you find you are using commands like sleep to run a one-off task at a specific time, then meet the GNU at command.

at will allow you to specify a time to run a specific command, and accepts a wide range of different formats including “now + 4 hours”, “noon”, and “next tuesday”. The command will read commands from stdin or using the -f to specifiy a file to be executed using the default /bin/sh.