Reallywow

At ADS we are looking at new ways to index and provide full text searching for the Astronomy and Physics literature we manage to obtain, either through scanning + OCR of historical content, or from digital material provided by some publishers. Two options we’re looking at are Apache Solr and CDS-Invenio. But that’s not what this post is about.

While parsing and indexing a pile of about 42k articles from the past dozen or so years of the ApJ, AJ, ApJL and ApJS, formatted in the NLM XML schema, I noticed that many of the articles contained external links to various things, most interestingly, astronomical datasets.* My first thought was, “hmm, I wonder what’s at the other end of all those links…,” followed closely by, “hey, crawling those links would make a nice dataset to load into that nifty new Freebase Gridworks tool I heard about the other day.” So that’s what I did.

Out of 13652 articles there were 33600 total links which fell into three categories: http urls (28555), dataset links (938) and supplement links (4107). Dataset links consist of an identifier that looks something like ADS/Sa.CXO#obs/927. To get the goods you have to feed that id to a resolver which, assuming a valid identifier, will redirect you to the real location of the dataset. Supplement links took a bit more head-scratching as their values consisted of just a relative file name, like datafile3.txt or 69491.figures.html. We figured out that the solution was to append the filename to the publisher’s URL for the article, e.g., article and dataset or article and figures.

The ultimate objective was to load the results of crawling these links into Gridworks, but that means getting the data into csv or tsv form. Rather than have the crawl script output straight to csv, I stash the results in a MongoDB instance. Here’s an example of one of the resulting json documents in Mongo:

From there it was easy to dump what I needed to csv and load into Gridworks. I’m not going to get into how totally awesome the Gridworks software is, except to say you should watch the demo videos.

I can’t post the entire Gridworks project, but here’s some screencaps, a column list and some of the more interesting facets.

Initial data load plus some derived columns

Column list:

• Id of the MongoDB doc
• Id of the solr doc
• ADS bibcode identifier of the article
• Publication year – derived from the bibcode
• DOI
• xpath expression of the <ext-link> element
• parent tag – the containing element type
• link context – the containing element’s serialized xml contents
• link type – one of url, dataset or supplement
• anchor text – the text contents of the <ext-link>
• full text source file
• journal
• full text source – publisher
• extlink id – either the url or the dataset id or the supplement filename
• domain – derived from the url
• status – http status returned when requesting the resource
• content-type – content-type header returned in the response
• mimetype – derived from the content-type response header
• location – the final url of the resource following any redirects
• content length
• response headers – list of all the header attribute names return in the response (just to see what other interesting stuff might be there)

Still to be determined how many of the url links point to some kind of data

Knowing the container could help parsing out something about the semantics of the link

~70% 200's was more than I expected. Of course 200 doesn't mean it actually found something interesting.

would have hoped for fewer text/html

All the gcn.gsfc.nasa.gov hits look like observation reports, like this one, which I think is a good thing

Finally a thanks to Sean Hannan who worked out a hack to a bit of the Gridworks javascript that automatically turns any cell values beginning with “http://” or “https://” into active links. The nice thing about that was it let me turn the column containing the MongoDB id into a link to a little web.py script that dumps a JSON representation of the document.

* NLM allows for links to external resources using either <ext-link> or <supplementary-material> elements.

Here’s what I know: you can embed a set of <meta/> tags containing citation metadata in your HTML to help Google Scholar to index your content. We’ve been doing it at ADS for quite a while. I’m not certain if the impetus came directly from Google, or, more likely, we got the idea from a CrossTech blog post by Tony Hammond that describes the technique.

For example, if you execute  curl -s http://adsabs.harvard.edu/abs/1977NuPhB.126..298A | grep meta you should see:

Since first implementation we’ve had some back-and-forth with Abhishek Jain at Google Scholar to ensure we’re making use of the full set of fields that Google Scholar looks for.*

Dan Chudnov, David Bucknum & Ed Summers at the LoC recently expressed interest in also embedding these tags. In the absence of official reference from the Google Scholar folks, I figured it would be a good thing to post here.

• citation_language
• citation_doi
• citation_abstract_html_url
• citation_title
• citation_authors
• citation_issn
• citation_date
• citation_journal_title
• citation_volume
• citation_firstpage
• citation_lastpage
• citation_publisher
• citation_issue
• citation_pdf_url
• citation_pmid
• citation_keywords (multiple instances OK)
• citation_conference
• citation_dissertation_name
• citation_dissertation_institution
• citation_patent_number
• citation_patent_country
• citation_technical_report_number
• citation_technical_report_institution

I had to cull this list via a visual scan of a long, forwarded e-mail thread. So, like I tried to insinuate above, it sure would be great if Google Scholar would publish an official reference to this schema somewhere.

* all instances of the term “we” should really be read as “my boss, Alberto”.

I thought I’d share the interview outline I’ve been using to conduct some low effort contextual inquiry sessions with ADS users.

thumbnail links to google doc

Classic contextual inquiry, in which the researcher sits with or shadows a person in the context of the subject’s own working environment, is often conducted in 3+ hour sessions, frequently with all manner of video capturing equipment. My goal is cut that time down to 30 minutes, partly because this whole user research thing is supposed to be a part-time endeavor, and also because the majority of ADS users are PhD’s, and we all know just how valuable their time is.

So far I’ve only managed to conduct four of these interviews (with two more scheduled). Would love to get a total of 10. Since I don’t have access to video equipment I simply mash out typewritten, poorly spelled notes as fast I can. The notes have a stream of consciousness flavor, but the early indications are that the information gathered will be valuable.

Example notes:

refers to bibcode as "indexing thing". "not any use to me."
wrote a perl script that rewrites the bibcode into something understandabl
other strategies for searching for particular star: entering star name into abstract search or title search.
finds one article using abstract search.
mentions that he doesn't know boolean sytnax by memory
to find more tries going to simbad and finds alternate names for the star