Singularity Container System for HPC
At work I had a task to port the WRFv4 atmospheric model that we currently run in the office, from a Docker container, up to the Lonestar5 (LS5) supercomputer at the Texas Advanced Computing Center (TACC). My first impulse was to simply replicate the Dockerfile steps that we use in a simple bash script to natively build the required WRF executables. Since I always forget the module load commands, I opened up the LS5 documentation. In doing so, I noticed that a recent update system update provides support for “Singularity” containers. Somehow I missed the rising popularity and support for Singularity, but was intrigued because it attempts to solve several of the problems we deal with running parallel numerical models (MPI, GPU and queue/batch systems) with Docker containers. And so I went down the rabbit hole of researching and installing Singularity with the hopes of replicating our Docker-like container workflows in a way that will run on LS5.
September 23, 2019
Shortcut to Website in Ubuntu 18.04 Dock
Create and populate the following file to add a dock shortcut to a website in Ubuntu using Ars Technica as an example. This will also enable the shortcut in the Gnome Shell search. ~/.local/share/applications/MyWebsite.desktop: [Desktop Entry] Comment=Ars Technica Terminal=false Name=Ars Technica Exec=firefox https://arstechnica.com/ Type=Application Icon=applications-internet NoDisplay=false To use Google Chrome instead, replace firefox with google-chrome if it is installed.
September 2, 2019
Running WRF Singularity Container on TACC Systems
So prior the the recent TACC training on containers I was trying to get our WRF Docker images to run up on the TACC HPC infrastructure using Singularity. I checked the version of Singularity on the Lonestar5 cluster and it was behind the most recent versions a bit, but luckily there are Docker images for Singularity so you can play with whatever version might be required. (The version on Lonestar5 may already be updated by now too, recent version on Stampede2 was 3.4.3.)
March 18, 2019
Reproducing Conda Environments
This short summary is based on the Anaconda blog post here https://www.anaconda.com/moving-conda-environments/. The original blog post is a great high-level summary for the various methods in conda for reproducing environments. OS and platform specific (pulls from repos) # On source environment: conda list --explicit > spec-list.txt # New conda environment: conda create --name new_env_name --file spec-list.txt Different platforms and OS (pulls from repos, also includes pip installed packages)
August 23, 2018
Decompiling Python .pyc Files
Have you accidentally deleted an important Python source file or are looking to inspect the contents of a .pyc file that has been provided to you? Luckily .pyc files contain enough information to reproduce the corresponding .py file. NB: this only applies to Python2. You won’t get the original comments or the original formatting, and there may be a few tweaks you need to do for the new .py file to be completely valid–but this can be a savior for some unexpected loss of source files if the .pyc files still exist.
June 28, 2018
Installing NetCDF Python Packages
I was trying to remember how I have installed netCDF4 and related libraries for Python, and what I need to do differently for Windows systems vs. the Linux systems I usually use. On Linux, sometimes I use the system netCDF C libaries, but often I compile and install specific versions of HDF5 and netCDF4 from scratch. Here is how I have built netCDF for various Docker container images. # Build HDF5 cd hdf5-x.y.x ./configure --prefix=/usr/local --enable-shared --enable-hl make make install cd .. # Built NetCDF4 cd netcdf-x.y.z LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include ./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen make make install cd .. # NetCDF4 Fortran cd netcdf-fortran-x.y.z ./configure --enable-shared --prefix=/usr/local make make install cd .. netcdf4-python I typically try to use pip to install Python libraries if I can. Use pip if you need to, but I am using conda and conda-forge as much as possible now, in fact by using conda, the above compilation steps are usually not necessary as far as I know. See below.
March 17, 2018
Using the Blockchain for Open-access Journals?
One thing that excites me about the current buzz around blockchain technology is its use for open science. I can’t speak to the feasibility, but it seems to me that a distributed ledger could be an ideal place to publish and provide open-access to scientific research papers and articles. If including a way to store, deliver and update supporting data, a blockchain could deliver research products that link directly to the data and analysis–providing an unparalleled level of provenance and context for research and results. Citations and work building on similar pieces of data could be connected allowing for straighforward literature searches and discovery.
March 10, 2018