Install *timewarrior-dataframe* using *pipx* for pivot tables of *timewarrior* summaries

Things to know:

• Timewarrior (timew) is a command line tool for tracking time.
• Timewarrior-Dataframe is an extension processes timew export output using the Pandas Python module.

Install #

Timewarrior can be installed on recent Ubuntu releases from the apt repositories. It can also be found in a number of other distro’s packaging systems.

1
2

# Ubuntu (and/or Debian?)
sudo apt install timewarrior

pipx can be used to install the timewarrior-dataframe’s twdf command from it’s git repository.

Run Large Language Models Locally With *ollama.ai's* Docker Image

ollama is an open-source framework for self-hosting large language models (LLM) similar to ChatGPT or Google’s Bard. The official Docker image makes it painless to host any number of the supported models.

Shell script to get an Ollama model running #

The example “run_ollama_container.sh” script(s) below take the LLM model name as an argument (assigned to $LLMMODEL), but the pull and run docker exec commands can be called additional times to start other models as well within the same running container.

What is the NVIDIA Container Toolkit?

The NVIDIA Container Toolkit is a set of utilities and libraries provided by NVIDIA to enable containers to use and share NVIDIA GPU resources. The toolkit is container agnostic, supporting several common container runtimes including Docker and Podman.

It appears to present GPU/CUDA features to software running within containers on the host with the NVIDIA GPU. The supported platforms seem to be specifically GNU/Linux distributions (Debian, Ubuntu, Centos, Fedora, Open Suse, and Amazon’s Amazon Linux AMIs) on typical amd64, arm64/aarch64 hardware.

*Magic-Wormhole* file transfer

Warp is a Rust software package that makes sending content between computers super easy using entryption and simple one-time word-based or QR-code authentication keys. Because of the underlying protocol, transfers will occur over local LAN or internet depending on where the machines involved are located. The software allows users to send and recieve files between Linux, OSX, Windows and smart phones.

I was interested in using Warp on the command line for simple low-setup file transfers in and out of my office, but it turns out that the underlying protocol that Warp leverages is also an open-source project and it already includes a CLI (magic-wormhole).

An Update To Apptainer Use On TACC HPC

The University of Texas’ newest supercomputer Frontera currently requires the following modules loaded for Apptainer images/containers based on the mvapich infiniband (ib) base images–particularly for any containerized programs that intend to use MPI.

Researchers moving their Apptainer images between TACC HPC systems, or building container images specifically for individual systems, may need to adapt their sbatch scripts to accommodate differences between default environments and infrastructures on the various available clusters.

Trivy

The many layers of software dependencies within containers can add extra security considerations. Luckily there are a number of tools like Trivy and Docker scan to scan Docker images for vulnerabilities.

[Trivy is a] Scanner for vulnerabilities in container images, file systems, and Git repositories, as well as for configuration issues and hard-coded secrets

Trivy can be installed with the following commands on Debian-based Linux distros, including WSL environments.

SWAN Wave Model Container with MPI

Coinciding with the Containers at TACC training taking place today via webinar, here is a method I have used successfully to build our version of SWAN v41.10 spectral wave model in order to run with MPI on the University of Texas’ Stampede2 cluster.

Build SWAN with Docker #

The following Dockerfile will build the parallel mpi implementation of SWAN, the single-core serial SWAN model, and the SWAN hcat concatenation utility. The containerized model should be able to be used for 2D non-stationary, 1D or stationary simulations.

Quickly Set Up Guacamole Using Docker

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Thanks to HTML5, once Guacamole is installed on a server, all you need to access your desktops is a web browser.

The Guacamole manual can be found here, but because the project is broken into 2 or more peices, it can take a bit of poking around and toggling between sections to find the exact steps one needs to quickly get up and running from scratch using Docker.

Useful Docker Patterns (On GNU/Linux)

To easily copy files from a Docker image when you dont want to start a container, you can do the following:

1

docker cp $(docker create --rm $IMAGE):$FROM_PATH $TO_PATH

Sometimes it is useful to compare the output of a command between two Docker images, especially comparing different versions of a particular image:

1
2
3

docker run --rm -ti --entrypoint=bash $IMAGE1 -c "$CMD" > /tmp/output1.tmp
docker run --rm -ti --entrypoint=bash $IMAGE2 -c "$CMD" > /tmp/output2.tmp
diff -s /tmp/output1.tmp /tmp/output2.tmp

I have both these command patterns saved as executable Bash shell scripts on my system path as cp_docker and diff_docker_cmd respectively.

Singularity From Local Docker Image

One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo.

To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.

Singularity Container System for HPC

At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables.

Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.

Running WRF Singularity Container on TACC Systems

So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity.

I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)