Overlooked Tools: 'watch'
There many GNU/Linux command line tools that provide continuously updated statistics or information like iotop, nload, and top. However, it is often necessary to see the output of another simple command or complex chain of tools, updated in real-time. watch is a Linux command that does just that, and by doing so, is incredibly useful for continuously monitoring the output of any Linux terminal command. Usage of the watch command is incredibly simple, as the following example monitoring the currently running tsp task in the task-spooler queue demonstrates.
February 10, 2021
Lean Philosophy
Given my experience and education, I have always been sort of philosophically aligned with the concepts in “Lean” management, production, and software development thought to have originated at Toyota in the 90’s. Using software as example, those are: Eliminate Waste If some activity could be bypassed or the result could be achieved without it, it is waste. Partially done coding eventually abandoned during the development process is waste. Extra features like paperwork and features not often used by customers are waste. Switching people between tasks is waste. Waiting for other activities, teams, processes is waste. (from wikipedia)
January 17, 2021
Singularity From Local Docker Image
One of the powerful features of Singularity is the ability to create Singularity images from Docker images, pulling directly from public or otherwise accessible Docker repositories. There are also some un- or minimally- documented ways to extend Singularity’s impressive capacity, like converting local Docker images that may not be available in a proper Docker image repo. To convert a local image without fetching from a remote repository you can use docker-daemon in the protocol as found here.
January 13, 2021
Autumn Leaves Are Free Money
'Leaves build soil, conserve water and best of all, control weeds. By using leaves as your compost and mulch you are restoring nutrients to the soil that leaves used in order to grow…They’re like a bank account that never runs dry.'
November 18, 2020
Overlooked Tools: task-spooler
Sometimes sophisticated job queue and cluster systems are overkill on office or individual PCs, especially for long lists of batch processing jobs that use comparatively few resources. Use task-spooler to queue commands on Linux and control the number of simultaneous jobs for an easy way to parallelize a list of batch jobs! This tool is a great complement to some other command line workhorses like at and batch. Task-spooler is not usually installed by default on Linux systems. It can be built by scratch after getting the source from https://vicerveza.homeunix.net/~viric/soft/ts/ or on Ubuntu (maybe other Debian based OS’s) you can use apt:
November 17, 2020
Overlooked Tools: xclip
If you have ever had a need to move the output of a Linux console command to your clipboard, xclip is the solution! To copy a file’s contents to the standard “ctl+c/v” clipboard buffer issue you can following this command: xclip -sel clip your-file-path The most useful feature is reading from a stdin pipe:
November 16, 2020
Singularity Container System for HPC
At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables. Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.
September 23, 2019
Shortcut to Website in Ubuntu 18.04 Dock
Create and populate the following file to add a dock shortcut to a website in Ubuntu using Ars Technica as an example. This will also enable the shortcut in the Gnome Shell search. ~/.local/share/applications/MyWebsite.desktop: [Desktop Entry] Comment=Ars Technica Terminal=false Name=Ars Technica Exec=firefox https://arstechnica.com/ Type=Application Icon=applications-internet NoDisplay=false To use Google Chrome instead, replace firefox with google-chrome if it is installed.
September 2, 2019
Running WRF Singularity Container on TACC Systems
So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity. I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)
March 18, 2019
Decompiling Python .pyc Files
Have you accidentally deleted an important Python source file or are looking to inspect the contents of a .pyc file that has been provided to you? Luckily .pyc files contain enough information to reproduce the corresponding .py file. NB: this only applies to Python2. You won’t get the original comments or the original formatting, and there may be a few tweaks you need to do for the new .py file to be completely valid–but this can be a savior for some unexpected loss of source files if the .pyc files still exist.
June 28, 2018