Save CT Images from *Scan of the Month*

I love seeing the monthly updates on Scan of the Month, a website that posts detailed CT scan renderings for selected themes. This website is a portfolio created by CT device company Lumafield.

Lumafield is a company that was started by a team of engineers who had firsthand experience with trying to solve hard problems and being let down by the tools on hand. Lumafield is also a company, that while in stealth-mode, decided the technology they were building was just too dang cool to keep quiet about.

NWS12 to NWS13

tool to convert OWI’s NWS12 formatted ascii files (fort.22{1,2}) to OWI’s NWS13 NetCDF4

Artistic Overlapping Timeseries Plot

MDAL

The Mesh Data Abstraction Library (MDAL) out of OSGeo is a “translator library” for many common conventions found in meteorology and hydrology. The library supports data found in Grib and NetCDF encoded as NetCDF Climate and Forecast Metadata Conventions (CF) or Unstructured Grid Conventions (CF/UGRID) and represented as geospatial mesh data. In addition to supporting mature and well-defined spatial data encodings in self-describing files, the library also supports numerous model-specific formats including Telemac, HEC-RAS, and TUFLOW.

Data Catalog From Filesystem

This previous post demonstrated a way to use generic website html source to create an inventory of web-accessible files using discovered URL links. A similar methodology can be applied to search through filesystem directory tree to establish a catalog for any matching filetypes.

Use glob to recursively globstar-match filepaths #

In this case I use Python’s builtin glob and regular expression modules to list files and match extensions in the names. I used the os.path collection of utility methods to pull directory names from the full paths, but a more modern way would probably to use the builtin pathlib. Pandas is the only non-builtin package used, which could be removed if a DataFrame is not the desired output.

Examples of accessible data inventories

Simple over Standards #

Frequently the biggest hurdle to sharing data is the pre-existing organization of files. While open standards based API access can smooth over the bumps inherent to locally stored heterogeneous collections of files, the amount of effort to setup an existing API webservice to share your data can be overwhelming. (Not to mention having to roll your own in an standards complant way.)

The NOAA example below is a great example of a simple solution that can be used as a template to provide easy programatic access to any dataset.

fort.15 configuration for NWS13

required changes

Interpolation Implementation for NWS13

supporting moving wind and pressure inputs

NWS13

Flexible (NetCDF) Metocean Inputs for ADCIRC

Using the Blockchain for Open-access Journals?

One thing that excites me about the current buzz around blockchain technology is its use for open science. I can’t speak to the feasibility, but it seems to me that a distributed ledger could be an ideal place to publish and provide open-access to scientific research papers and articles.

If including a way to store, deliver and update supporting data, a blockchain could deliver research products that link directly to the data and analysis–providing an unparalleled level of provenance and context for research and results. Citations and work building on similar pieces of data could be connected allowing for straighforward literature searches and discovery.

ugrid

Conventions Documentation #

The conventions can be found here:

https://ugrid-conventions.github.io/ugrid-conventions/

Which is maintained though the following git repository:

https://github.com/ugrid-conventions/ugrid-conventions

Check Compliance #

pyugrid can be used to parse and validate CF-UGRID metadata. Its documentation is located here:

https://pyugrid.github.io/pyugrid/