Reallywow

At ADS we are looking at new ways to index and provide full text searching for the Astronomy and Physics literature we manage to obtain, either through scanning + OCR of historical content, or from digital material provided by some publishers. Two options we’re looking at are Apache Solr and CDS-Invenio. But that’s not what this post is about.

While parsing and indexing a pile of about 42k articles from the past dozen or so years of the ApJ, AJ, ApJL and ApJS, formatted in the NLM XML schema, I noticed that many of the articles contained external links to various things, most interestingly, astronomical datasets.* My first thought was, “hmm, I wonder what’s at the other end of all those links…,” followed closely by, “hey, crawling those links would make a nice dataset to load into that nifty new Freebase Gridworks tool I heard about the other day.” So that’s what I did.

Out of 13652 articles there were 33600 total links which fell into three categories: http urls (28555), dataset links (938) and supplement links (4107). Dataset links consist of an identifier that looks something like ADS/Sa.CXO#obs/927. To get the goods you have to feed that id to a resolver which, assuming a valid identifier, will redirect you to the real location of the dataset. Supplement links took a bit more head-scratching as their values consisted of just a relative file name, like datafile3.txt or 69491.figures.html. We figured out that the solution was to append the filename to the publisher’s URL for the article, e.g., article and dataset or article and figures.

The ultimate objective was to load the results of crawling these links into Gridworks, but that means getting the data into csv or tsv form. Rather than have the crawl script output straight to csv, I stash the results in a MongoDB instance. Here’s an example of one of the resulting json documents in Mongo:

From there it was easy to dump what I needed to csv and load into Gridworks. I’m not going to get into how totally awesome the Gridworks software is, except to say you should watch the demo videos.

I can’t post the entire Gridworks project, but here’s some screencaps, a column list and some of the more interesting facets.

Initial data load plus some derived columns

Column list:

• Id of the MongoDB doc
• Id of the solr doc
• ADS bibcode identifier of the article
• Publication year – derived from the bibcode
• DOI
• xpath expression of the <ext-link> element
• parent tag – the containing element type
• link context – the containing element’s serialized xml contents
• link type – one of url, dataset or supplement
• anchor text – the text contents of the <ext-link>
• full text source file
• journal
• full text source – publisher
• extlink id – either the url or the dataset id or the supplement filename
• domain – derived from the url
• status – http status returned when requesting the resource
• content-type – content-type header returned in the response
• mimetype – derived from the content-type response header
• location – the final url of the resource following any redirects
• content length
• response headers – list of all the header attribute names return in the response (just to see what other interesting stuff might be there)

Still to be determined how many of the url links point to some kind of data

Knowing the container could help parsing out something about the semantics of the link

~70% 200's was more than I expected. Of course 200 doesn't mean it actually found something interesting.

would have hoped for fewer text/html

All the gcn.gsfc.nasa.gov hits look like observation reports, like this one, which I think is a good thing

Finally a thanks to Sean Hannan who worked out a hack to a bit of the Gridworks javascript that automatically turns any cell values beginning with “http://” or “https://” into active links. The nice thing about that was it let me turn the column containing the MongoDB id into a link to a little web.py script that dumps a JSON representation of the document.

* NLM allows for links to external resources using either <ext-link> or <supplementary-material> elements.