Simple over Standards #

Frequently the biggest hurdle to sharing data is the pre-existing organization of files. While open standards based API access can smooth over the bumps inherent to locally stored heterogeneous collections of files, the amount of effort to setup an existing API webservice to share your data can be overwhelming. (Not to mention having to roll your own in an standards complant way.)

The NOAA example below is a great example of a simple solution that can be used as a template to provide easy programatic access to any dataset.

HTML as Catalog #

The “pretty” part of the web that we interact with daily is based on HTML files, which can serve as data catalogs in themselves if leveraged appropriately by smart organizations. Needing to download a set of files that listed as links on a webpage, I often use the following hacky method scrape a static webpage for links with specific extensions.

The static HTML scrape hack for a set of webpages I often need to check the the links point to re,al files and the content paths sometimes change without warning. Nine times out of ten this method gets the job done.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import pandas as pd
import bs4
import requests
from urllib.parse import urlparse

ext = {"json", "pdf", "png", "gif", "jpg", "tcw", "txt", "xml", "kml", "kmz", "zip"}

catalog = []
url = ["https://www.navcen.uscg.gov/north-american-ice-service-products",
       "https://www.metoc.navy.mil/jtwc/jtwc.html",
       "https://www.metoc.navy.mil/jtwc/rss/jtwc.rss"]
for mount in url:
    r = requests.get(mount)
    domain = mount.split(":")[0] + "//" + urlparse(mount).netloc
    links = bs4.BeautifulSoup(r.text, features="lxml").find_all(name="a")
    for link in links:
        if "href" in link.attrs:
            if link.attrs["href"] is not None:
                iext = link.attrs["href"].split(".")[-1]
                if iext in ext:
                    linkpath = link.attrs["href"]
                    catalog.append((link.text, domain, f"{domain}{linkpath}", iext,))
catalog = pd.DataFrame(catalog, columns=["Description", "Folder", "Fullpath", "Extension",])
if catalog.shape[0] > 1:
    catalog["Pattern"] = catalog.Fullpath
    catalog["LowerPath"] = catalog.Fullpath.str.lower()
    catalog["LowerDescription"] = catalog.Description.str.lower()

catalog.to_csv("temp.csv", index=False)

temp.csv output: #

Martin County DEM Data #

My first impulse was to scrape this site for the DEM tile .img files given that they all are provided as url links on the page.

However, in this case they have saved me some time and the website already provides a list of the publically available tile files, and ancillary files like a tile index, and metadata.

To download all the .img files, all I have to do is the following from a terminal:

1

curl https://chs.coast.noaa.gov/htdata/raster2/elevation/FL_Martin_County_DEM_2016_6254/urllist6254.txt | grep ".img" | parallel wget {}

So simple, but so useful for folks who would not be keen to try to write code to parse html and extract the relevent links and possible to construct with minimum effort.

Next time I provide files for a client I think I’ll provide a link to my web accessible file list and the sample download command right in my email.