Overlooked Tools: 'watch'

There many GNU/Linux command line tools that provide continuously updated statistics or information like iotop, nload, and top. However, it is often necessary to see the output of another simple command or complex chain of tools, updated in real-time.

watch is a Linux command that does just that, and by doing so, is incredibly useful for continuously monitoring the output of any Linux terminal command. Usage of the watch command is incredibly simple, as the following example monitoring the currently running tsp task in the task-spooler queue demonstrates.

Singularity From Local Docker Image

One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo.

To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.

Overlooked Tools: task-spooler

Sometimes sophisticated job queue and cluster systems are overkill on office or individual PCs, especially for long lists of batch processing jobs that use comparatively few resources.

Use task-spooler to queue commands on Linux and control the number of simultaneous jobs for an easy way to parallelize a list of batch jobs! This tool is a great complement to some other command line workhorses like at and batch.

Task-spooler is not usually installed by default on Linux systems. It can be built by scratch after getting the source from https://vicerveza.homeunix.net/~viric/soft/ts/ or on Ubuntu (maybe other Debian based OS’s) you can use apt:

Overlooked Tools: xclip

If you have ever had a need to move the output of a Linux console command to your clipboard, xclip is the solution!

To copy a file’s contents to the standard “ctl+c/v” clipboard buffer issue you can following this command:

1

xclip -sel clip your-file-path

The most useful feature is reading from a stdin pipe:

1

head my-file-path | xclip -sel clip

…and then your ready to paste into an email or Slack chat.

Singularity Container System for HPC

At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables.

Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.

Shortcut to Website in Ubuntu 18.04 Dock

Create and populate the following file to add a dock shortcut to a website in Ubuntu using Ars Technica as an example. This will also enable the shortcut in the Gnome Shell search.

~/.local/share/applications/MyWebsite.desktop:

[Desktop Entry]
Comment=Ars Technica
Terminal=false
Name=Ars Technica
Exec=firefox https://arstechnica.com/
Type=Application
Icon=applications-internet
NoDisplay=false

To use Google Chrome instead, replace firefox with google-chrome if it is installed.

[Desktop Entry]
Comment=Ars Technica
Terminal=false
Name=Ars Technica
Exec=google-chrome https://arstechnica.com/
Type=Application
Icon=applications-internet
NoDisplay=false

Running WRF Singularity Container on TACC Systems

So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity.

I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)

Overlooked Tools: 'at'

If you find you are using commands like sleep to run a one-off task at a specific time, then meet the GNU at command.

at will allow you to specify a time to run a specific command, and accepts a wide range of different formats including “now + 4 hours”, “noon”, and “next tuesday”. The command will read commands from stdin or using the -f to specifiy a file to be executed using the default /bin/sh.