*Magic-Wormhole* file transfer

Warp is a Rust software package that makes sending content between computers super easy using entryption and simple one-time word-based or QR-code authentication keys. Because of the underlying protocol, transfers will occur over local LAN or internet depending on where the machines involved are located. The software allows users to send and recieve files between Linux, OSX, Windows and smart phones.

I was interested in using Warp on the command line for simple low-setup file transfers in and out of my office, but it turns out that the underlying protocol that Warp leverages is also an open-source project and it already includes a CLI (magic-wormhole).

An Update To Apptainer Use On TACC HPC

The University of Texas’ newest supercomputer Frontera currently requires the following modules loaded for Apptainer images/containers based on the mvapich infiniband (ib) base images–particularly for any containerized programs that intend to use MPI.

Researchers moving their Apptainer images between TACC HPC systems, or building container images specifically for individual systems, may need to adapt their sbatch scripts to accommodate differences between default environments and infrastructures on the various available clusters.

SWAN Wave Model Container with MPI

Coinciding with the Containers at TACC training taking place today via webinar, here is a method I have used successfully to build our version of SWAN v41.10 spectral wave model in order to run with MPI on the University of Texas’ Stampede2 cluster.

Build SWAN with Docker #

The following Dockerfile will build the parallel mpi implementation of SWAN, the single-core serial SWAN model, and the SWAN hcat concatenation utility. The containerized model should be able to be used for 2D non-stationary, 1D or stationary simulations.

Singularity From Local Docker Image

One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo.

To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.

Singularity Container System for HPC

At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables.

Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.

Running WRF Singularity Container on TACC Systems

So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity.

I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)