*Magic-Wormhole* file transfer

Warp is a Rust software package that makes sending content between computers super easy using entryption and simple one-time word-based or QR-code authentication keys. Because of the underlying protocol, transfers will occur over local LAN or internet depending on where the machines involved are located. The software allows users to send and recieve files between Linux, OSX, Windows and smart phones.

I was interested in using Warp on the command line for simple low-setup file transfers in and out of my office, but it turns out that the underlying protocol that Warp leverages is also an open-source project and it already includes a CLI (magic-wormhole).

An Update To Apptainer Use On TACC HPC

The University of Texas’ newest supercomputer Frontera currently requires the following modules loaded for Apptainer images/containers based on the mvapich infiniband (ib) base images–particularly for any containerized programs that intend to use MPI.

Researchers moving their Apptainer images between TACC HPC systems, or building container images specifically for individual systems, may need to adapt their sbatch scripts to accommodate differences between default environments and infrastructures on the various available clusters.

Reproducible Scientific Python Using Containers

Use Microsoft’s VSCode editor (code), Docker Containers, and other open-source tools for scientific Python software collaboration, development, and use on Linux and Windows. Securely connect offices, remote workers, storage resources, compute resources, and the cloud with Tailscale as a replacement for traditional VPN.

System Packages #

VSCode Editor #

Download Visual Studio Code from Microsoft.

For a Debian-based GNU/Linux distribution like Ubuntu or Pop OS, the .deb can be installed with sudo dpkg -i code_$version_amd64.deb.

Installing Messenger For Multi-node Task-Spooler

Messenger Server #

On a base apt-based distro install the Task-Spooler Messenger Server:

1
2
3
4
5

sudo apt install -y libboost-dev libboost-system-dev libboost-thread-dev libboost-filesystem-dev

pushd messenger/messenger-server
sudo ./install
popd

Configure the messenger-server ms_server by creating ~/.hosts_ports with a port for listening to client messages.

1

echo "xxx $THISHOSTPORT" >> ~/.hosts_ports

Start the server with the command ms_server.

Build ADCIRC using CMake

Gone are the days of having to use config.guess and sed replcements to customize the cmplrflags.mk for a particular HPC environment.

Here is the process to build the ADCIRC hydrodynamic model using CMake from the command line on a GNU/Linux host. The CMake infrastructure has only been available in the last few releases of ADCIRC, and this example uses v55.00 from the compressed tar archive of the code. The options specified below enable serial adcirc, distributed parallel padcirc, coupled serial adcswan (ADCIRC+SWAN), and the parallel coupled padcswan model. Standalone parallel SWAN and unstructured SWAN models are also configured, along with the adcprep ADCIRC pre-processor and other utilities.

SWAN Wave Model Container with MPI

Coinciding with the Containers at TACC training taking place today via webinar, here is a method I have used successfully to build our version of SWAN v41.10 spectral wave model in order to run with MPI on the University of Texas’ Stampede2 cluster.

Build SWAN with Docker #

The following Dockerfile will build the parallel mpi implementation of SWAN, the single-core serial SWAN model, and the SWAN hcat concatenation utility. The containerized model should be able to be used for 2D non-stationary, 1D or stationary simulations.

Singularity From Local Docker Image

One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo.

To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.

Singularity Container System for HPC

At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables.

Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.

Running WRF Singularity Container on TACC Systems

So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity.

I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)

sci-wms

https://slides.com/alexandercrosby/sci-wms

ugrid

Conventions Documentation #

The conventions can be found here:

https://ugrid-conventions.github.io/ugrid-conventions/

Which is maintained though the following git repository:

https://github.com/ugrid-conventions/ugrid-conventions

Check Compliance #

pyugrid can be used to parse and validate CF-UGRID metadata. Its documentation is located here:

https://pyugrid.github.io/pyugrid/