At ADS we are looking at new ways to index and provide full text searching for the Astronomy and Physics literature we manage to obtain, either through scanning + OCR of historical content, or from digital material provided by some publishers. Two options we’re looking at are Apache Solr and CDS-Invenio. But that’s not what this post is about.

While parsing and indexing a pile of about 42k articles from the past dozen or so years of the ApJ, AJ, ApJL and ApJS, formatted in the NLM XML schema, I noticed that many of the articles contained external links to various things, most interestingly, astronomical datasets.* My first thought was, “hmm, I wonder what’s at the other end of all those links…,” followed closely by, “hey, crawling those links would make a nice dataset to load into that nifty new Freebase Gridworks tool I heard about the other day.” So that’s what I did.

Out of 13652 articles there were 33600 total links which fell into three categories: http urls (28555), dataset links (938) and supplement links (4107). Dataset links consist of an identifier that looks something like ADS/Sa.CXO#obs/927. To get the goods you have to feed that id to a resolver which, assuming a valid identifier, will redirect you to the real location of the dataset. Supplement links took a bit more head-scratching as their values consisted of just a relative file name, like datafile3.txt or 69491.figures.html. We figured out that the solution was to append the filename to the publisher’s URL for the article, e.g., article and dataset or article and figures.

The ultimate objective was to load the results of crawling these links into Gridworks, but that means getting the data into csv or tsv form. Rather than have the crawl script output straight to csv, I stash the results in a MongoDB instance. Here’s an example of one of the resulting json documents in Mongo:

From there it was easy to dump what I needed to csv and load into Gridworks. I’m not going to get into how totally awesome the Gridworks software is, except to say you should watch the demo videos.

I can’t post the entire Gridworks project, but here’s some screencaps, a column list and some of the more interesting facets.

Initial data load plus some derived columns

Column list:

• Id of the MongoDB doc
• Id of the solr doc
• ADS bibcode identifier of the article
• Publication year – derived from the bibcode
• DOI
• xpath expression of the <ext-link> element
• parent tag – the containing element type
• link context – the containing element’s serialized xml contents
• link type – one of url, dataset or supplement
• anchor text – the text contents of the <ext-link>
• full text source file
• journal
• full text source – publisher
• extlink id – either the url or the dataset id or the supplement filename
• domain – derived from the url
• status – http status returned when requesting the resource
• content-type – content-type header returned in the response
• mimetype – derived from the content-type response header
• location – the final url of the resource following any redirects
• content length
• response headers – list of all the header attribute names return in the response (just to see what other interesting stuff might be there)

Still to be determined how many of the url links point to some kind of data

Knowing the container could help parsing out something about the semantics of the link

~70% 200's was more than I expected. Of course 200 doesn't mean it actually found something interesting.

would have hoped for fewer text/html

All the gcn.gsfc.nasa.gov hits look like observation reports, like this one, which I think is a good thing

Finally a thanks to Sean Hannan who worked out a hack to a bit of the Gridworks javascript that automatically turns any cell values beginning with “http://” or “https://” into active links. The nice thing about that was it let me turn the column containing the MongoDB id into a link to a little web.py script that dumps a JSON representation of the document.

* NLM allows for links to external resources using either <ext-link> or <supplementary-material> elements.

Here’s what I know: you can embed a set of <meta/> tags containing citation metadata in your HTML to help Google Scholar to index your content. We’ve been doing it at ADS for quite a while. I’m not certain if the impetus came directly from Google, or, more likely, we got the idea from a CrossTech blog post by Tony Hammond that describes the technique.

For example, if you execute  curl -s http://adsabs.harvard.edu/abs/1977NuPhB.126..298A | grep meta you should see:

Since first implementation we’ve had some back-and-forth with Abhishek Jain at Google Scholar to ensure we’re making use of the full set of fields that Google Scholar looks for.*

Dan Chudnov, David Bucknum & Ed Summers at the LoC recently expressed interest in also embedding these tags. In the absence of official reference from the Google Scholar folks, I figured it would be a good thing to post here.

• citation_language
• citation_doi
• citation_abstract_html_url
• citation_title
• citation_authors
• citation_issn
• citation_date
• citation_journal_title
• citation_volume
• citation_firstpage
• citation_lastpage
• citation_publisher
• citation_issue
• citation_pdf_url
• citation_pmid
• citation_keywords (multiple instances OK)
• citation_conference
• citation_dissertation_name
• citation_dissertation_institution
• citation_patent_number
• citation_patent_country
• citation_technical_report_number
• citation_technical_report_institution

I had to cull this list via a visual scan of a long, forwarded e-mail thread. So, like I tried to insinuate above, it sure would be great if Google Scholar would publish an official reference to this schema somewhere.

* all instances of the term “we” should really be read as “my boss, Alberto”.

I thought I’d share the interview outline I’ve been using to conduct some low effort contextual inquiry sessions with ADS users.

thumbnail links to google doc

Classic contextual inquiry, in which the researcher sits with or shadows a person in the context of the subject’s own working environment, is often conducted in 3+ hour sessions, frequently with all manner of video capturing equipment. My goal is cut that time down to 30 minutes, partly because this whole user research thing is supposed to be a part-time endeavor, and also because the majority of ADS users are PhD’s, and we all know just how valuable their time is.

So far I’ve only managed to conduct four of these interviews (with two more scheduled). Would love to get a total of 10. Since I don’t have access to video equipment I simply mash out typewritten, poorly spelled notes as fast I can. The notes have a stream of consciousness flavor, but the early indications are that the information gathered will be valuable.

Example notes:

refers to bibcode as "indexing thing". "not any use to me."
wrote a perl script that rewrites the bibcode into something understandabl
other strategies for searching for particular star: entering star name into abstract search or title search.
finds one article using abstract search.
mentions that he doesn't know boolean sytnax by memory
to find more tries going to simbad and finds alternate names for the star

Like the rest of the planet it seems, I’ve been consumed lately by the show Mad Men. Jennifer and I are still catching up via Netflix. It really is one of those truly great and remarkable shows that comes along too rarely.

[Warning: Insignificant spoiler in the next couple of sentences]

About midway through the second season we came to the somewhat infamous picnic scene.  The Draper’s have taken the new Cadillac for a spin in the countryside. They relax and recline on a blanket. The grass is green, the breeze is mild, they talk about how rich they are. All is good. Then it’s time to pack it up and head home. We see Don stand up, stretch, smile, and chuck his empty beer can into the idyllic landscape as if he was tossing a baseball to his son. Betty pinches two corners of the  blanket and gives it a lift & shake, distributing the paper plates, napkins and other picnic detritus across the grass. The trash begins to lightly flutter and drift down the slope. In 1963 the happy family piles into the car and motors away. Meanwhile, in 2009, we sit on the couch, jaws agape at this stunning spectacle of thoughtless littering.

It’s suprisingly shocking. There’s the shock of seeing it, and then there’s the shock at being so shocked. Every bone in your body wants to be repulsed, but the relativist mindset makes it difficult to fault the characters. As the writer points out in the DVD commentary, Iron Eyes Cody didn’t come along until 1971.

The scene also in a way briefly cracks open that narrative fourth wall in that it’s clear the writer/director is blatantly highlighting these banal actions to serve up a very in-your-face cultural anachronism. It’s only been 40 years, but wow have the dominant cultural attitudes about the environment changed.

We’ve since watched a few more episodes, but that scene still sticks with me, and lately it’s got me imagining someone sitting on their couch in 2049–or hovering in their Anti-Grav Lounger, or whatever–and passing judgment on our present day actions. It’s interesting to think what might be the contemporary equivalents of folks in the early 60s treating the planet like giant trash receptacle.

I’m guessing they’ll look back in horror at us actually throwing things–anything!–away in a garbage can rather than somehow recycling or composting.

You mean the water from the shower just drains away into the sewer?!?

They have apples in a Boston supermarket that were grown in New Zealand? Insanity!

I mean, can you imagine!?

Can’t get to IRC because port 6667 is blocked on your local network? Here are some instructions for how to create a SSH tunnel using  PuTTY, and then connect to freenode (or any other IRC server) with Pidgin using the tunnel as a SOCKS5 proxy. You can most likely s/Pidgin/your IRC client of choice/, but the screenshots below will show the Pidgin config dialogs. These instructions also assume that SSH, port 22, is not also blocked. Woe be to you if that is the case.

My original source for how to do this was this post, which describes the same trick but for FireFox.

Step 1: create a new PuTTY session configuration. In this case I using the login me@ssh.example.com and calling the session irc-7777. I usually name the session based on the local port number I’m going to forward.

create and save a new PuTTY session

Step 2: go to the Connection -> SSH -> Tunnels node of the session config. In the Source port field enter “7777″ (or some other port number). In the radio button section below that select Dynamic and Auto. Click the Add button. You should see “D7777″ appear in the list of forwarded ports.

Configure the ssh tunnel

Tunnel D7777 appears in the list

Step 3: go back to the man session config node and save the session again. Then open the PuTTY session by clicking the Open button. A normal looking PuTTY terminal window should open. This session is your tunnel so you should probably leave it be. i.e., don’t use it for doing stuff in the shell and as a tunnel (although I don’t really know what consequences that would lead to). If the fact that this tunnel takes up space in your TaskBar (it does me) check out PuTTY-Tray.

You've probably never seen one of these before

Step 4: Configure your Pidgin IRC account to use the tunnel as a SOCKS5 proxy. Go to Accounts -> Manage Accounts. Highlight your IRC protocol account and click Modify (or create one by clicking Add). Go to the Advanced tab of the config dialog. In the Proxy Options section select SOCKS5 as the proxy type, enter “localhost” as the Host and “7777″ (or whatever port you used) as the Port.

Specify your local tunnel as the SOCKS5 proxy

Save and that’s it. You should be able to connect to IRC server through Pidgin.

On a *nix machine (or using Cygwin, I suppose) this is, of course, much simpler. You can replace the PuTTY steps with a single ssh command: ssh -D localhost:7777 me@ssh.example.com

Leading up to the Linked Data pre-conf at cod4lib09 there were several irc discussions around just how to structure the day and what could we do to give attendees the best shot at having an “ah-ha moment”. One idea was to create a simple application that would demonstrate the potential of linked data while also being participatory. I think it took Ed Summers less than 24 hours to hack together the first iteration of the cod4lib2009 attendees foaf crawler/gallery thingy. He can usually be counted on for such feats of overnight engineering.

By the day of the pre-conf we already had a gallery of 20+ foaf profiles. I spent part of the afternoon that day trying to guide several folks through the process of creating a foaf file and getting linked into our new corner of the web of data (with varying degrees of success). Along the way several people added enhancements, bugs were identified and sometimes fixed, much was learned re: linked data, vocabs, hash fragments vs. 303s, etc. Jonathan Brinley and Michael Klein even wrote a companion Supybot plugin for our irc bot, zoia. It was fun.

Since then my mind has come back to it now and then, mulling over how simple it would be to push some of the variables into a configuration file and make the crawler + gallery re-usable for other events, for example, the soon-to-be-happening BarCampBoston 4. This turned out to be relatively easy, and pretty quickly I had an demo attendees page for bcb4. Some fellow bcb4 goers were nice enough to participate by them asserting their attendence in their FOAF files and me pretending to know them. I’ve decided not to try leading any sessions on linked data or foaf at bcb4, mostly because it’s my first time and I want to soak up what other folks are into, but maybe it’ll make for a good ice-breaker or something.

Here’s my problem though: I’m starting to second guess myself as to the level of general usefulness of this thing–which, btw, I have christened gtfo, aka “gitfo”, aka “get the foaf out”–outside of the context of a tutorial/demo. I mean, could it really even function as an actual conference event attendee gallery? There’s the whole issue of folks needing to foaf:knows each other. I feel like there’s the seed of something cool in there, but it may need a substantial rethink and refactor to get beyond being the equivalent of the pet store shopping cart of linked data apps.

I have a few ideas… like allowing the criteria for inclusion to be configurable as something other than attendance at a specified event… or allowing the crawler to traverse a network of people connected via DOAP assertions rather than foaf:knows… or maybe allowing for galleries generated dynamically based on user input rather than a passive crawler, e.g., 1) choose seed node, 2) choose link relationship, 3) choose inclusion criteria, 4) generate gallery.

Anyway, if you’re interested in contributing ideas and or code, I’ve pushed my fork of the original app to Github: http://github.com/lbjay/gtfo/tree/master.

I was playing around with the idea of parallelizing something the other day to eke out some performance. Unfortunately, I’ve gotten a bit rusty since writing some MPI code for a parallel computing course a few years back. I got stuck on what should be the simple part of dividing up my input across the threads.

The goal is to divvy things up into continguous blocks of roughly equal size. i.e., if the size of your input is 38 (n) and you start four threads (p) you don’t want to give the first three threads chunks of 12 and the last thread gets 2. You want slices of 10, 9, 10 and 9.

So I flailed away with loops and the POSIX::floor for little awhile and came pretty close to what I remembered. I had to finally drag out my textbook (and translate from the C Macros) to get it right.

#!/usr/bin/perl

# Block Data Decomposition:
# Divide array n into p contiguous blocks of roughly equal size

use POSIX qw(floor);
use strict;

sub block_start {
    my ($i, $p, $n) = @_;
    return floor(($i * $n) / $p);
}

sub block_end {
    my ($i, $p, $n) = @_;
    return (block_start($i + 1, $p, $n) - 1);
}

my @input = get_input();
my $n = scalar @input;
my $p = 4;

for my $i (0..$p-1)
{
    my $start = block_start($i, $p, $n);
    my $end = block_end($i, $p, $n);
    my @range = @input[$start..$end];
    do_something(\@range);
}

The idea is that

sends a slice of input off for processing by one of your threads. A pretty useful algorithm when doing this sort of thing. Certainly not rocket science. Which is why we should all be happy I’m not a rocket scientist.

To cut to the chase, I extracted the hCal events from the 2009 conference schedule and fed them into a Simile Timeline. I then linked each event to the corresponding slice of my IRC client log. If you want to take a look it’s here.

I don’t remember what initially sent me there, but my introduction to Code4Lib was through the IRC channel. I’ve been logged in there off and on ever since. It keeps me informed and entertained and, yes, occasionally distracted. I’ve since attended all four of the yearly conferences, met and meatspace-friended a good percentage of the #code4lib regulars, contributed a patch here and there to a couple of projects, and helped organize a one-day regional gathering. I guess you could say at this point that I’m pretty fond of the whole thing.

This is why I take it somewhat personally when the annual hand-wringing debate begins over the perceived “cliquishness” of the community. There was much fuss this year–an awkward amount, even, IMO–over 1st-timers vs. old-timers. I make it a point to try and sit with people I don’t know during the lunches and shake a few hands. Lots of folks do the same for dinner.  Basically, IMO, if you feel like an outcast n00b, YOU’RE NOT TRYING HARD ENOUGH.

There is, however–and maybe I have a bit of “I haz a straw man. Let me show u it” going on here, but anyway–an aspect of the “code4lib is just a big fat secretive, juvenille high school-ish in crowd” argument where I think we majorly fail, and that is the non-open nature of the backchannel.

I met a lot of awesome new people over the past few days attending the 4th code4lib conference in Providence, RI, Jon Phipps of the NSDL MetaData Registry. I was a little suprised to read he didn’t enjoy the program, but that’s cool. I give him big props for calling it as he sees it. The part that got under my skin, because he’s totally right, was his mention of “the hugely active IRC back channel of ongoing commentary (which really should be displayed where everyone including the presenters can read it)”. This simply rang true to me.

Let me first say what I’m not saying: I do not think it’s rude and unfair that a bunch of us (100+, depending if you count those not physically present at the conf) are carrying on a parallel conversation while the presenters we have invited are getting up on stage and sharing projects and ideas that they care deeply about and have slaved over. This is the nature of our beast. To paraphrase something BillDeuber said in channel yesterday, is the channel and extension of the conf, or vice versa? I think the latter.

But I do think it’s rude and unfair that we are carrying on an un-open and inaccessible parallel conversation while the presenters we have invited are getting up on stage and sharing projects and ideas that they care deeply about and have slaved over.

The funny thing is, this reasoning is not what first prompted me to put my chat log up on the web. I did it because Corey Harper asked if I’d email him the section from when he was presenting at the linked data preconf and I figured others might like the same courtesy. I also thought Timeline would be a cool project to experiment with. I’ve since had a few conversations about whether or not it’s fair to the people in-channel who maybe didn’t realize what they were saying was going to be published later on. But if someone is accusing you of being cliquish and secretive, how is the proper response not to be more open and transparent? To say, “Here, take a look. See for yourself.”

This is probably making it seem like there must be some really juicy shit going on in the back channel, but that’s exactly the thing: there’s really not. It never gets any snarkier than your typical MST3K episode. And would anyone argue that Joel, Mike, Crow and Tom Servo didn’t really, deep down, love those old, bad movies they were forced to watch?

I’ve been on the lookout for a client that would somehow unify my streams from both Twitter & Identi.ca. There’s a few clients that will support one-or-the-other (sometimes via hacking the source) but I’ve found none that scratch my itch of have both services presented in the same window. My taskbar is precious real-estate.

These instructions reflect my setup on an Ubuntu vhost which I ssh into via PuTTY.

Required:

• Twitter account
• Identi.ca account
• TTYtter, a perl command-line client for accessing Twitter-compatible APIs
• Gnu Screen

Step 1, download TTYtter, make it executable and put it somewhere on your $PATH.

Step 2, create a file at ~/.ttytterrc1 with the following contents, including your identi.ca login:

  url=http://identi.ca/api/statuses/friends_timeline.json
  rurl=http://identi.ca/api/statuses/replies.json
  uurl=http://identi.ca/api/statuses/user_timeline
  wurl=http://identi.ca/api/users/show
  update=http://identi.ca/api/statuses/update.json
  dmurl=http://identi.ca/api/direct_messages.json
  frurl=http://identi.ca/api/friendships/exists.json
  user=[username]:[pass]

Step 3, create another file at ~/.ttytterc2 with just your twitter login:

  user=[username]:[pass]

Step 4, create a dedicated screen session with the command screen -S ttytter

Step 5, split the screen horizontally using the screen command Ctrl+a S

Step 6, start your Identi.ca session in the top window with the command ttytter.pl -rc=1

Step 7, switch to the bottom pane with the screen command Ctrl+a TAB

Step 8, create a new window in the bottom pane with the screen command Ctrl+a c

Step 9, start the Twitter session in the bottom pane with the command ttytter.pl -rc=2

Example screenshot

Only set this up 2 hours ago but already I can tell this is going to work for me long term. Beats having to do hard resets due to some combination of Twhirl and/or my video drivers. One improvement I’d like to investigate is some kind of color highlighting of the <username>’s to improve readability.

Update: January 13, 2009 at 11:30

So I quickly realized that the above setup has a major malfunction, namely that it doesn’t allow preserving of the split window so it’s not possible to detach and reattach to the ttytter screen session. So there’s an extra trick necessary to do this:

Step 3.5, create an “outer” screen session that wraps the ttytter session with the command screen -e^Ee -S outer.

The -e^Ee option binds the escape key for that session to Ctrl+e instead of the default Ctrl+a. This is a common trick for embedding screen sessions within sessions. With this extra step I can now detach from the outer session using Ctrl+e d and reattach with the inner split-screen session presevered using screen -r outer.

Also, I learned that to colorize the output you can use the -ansi option to ttytter.

Interesting conversation this month. (This is the stuff I listen to on my commute.) I was particularly intrigued by the 10 or so minutes spent discussing the need for a method of embedding identifiers and the location of a web service into HTML. Send the identifier to the service and get back the metadata. This is the exact use case of unAPI.

I was all set to get to work and give them a brrring! on the cluephone, a.k.a., comment on the post. But before I got around to it Ed Summers pointed out on irc that you can achieve the same thing using just <link> elements and/or HTTP Link: headers. In other words, why separate the identifiers from the service URI.

I like how simple unAPI is to implement. Since your metadata service’s base url doesn’t change you don’t have to worry about coordinating attributes of elements that need to appear in both your <head> and page <body>. This is a non-issue for lots of folks, but I bet it’s not so simple if your using WordPress or Drupal for your CMS.

As for the Glue extension thingie, I’ll try it out before passing judgement. But it did strike me funny that they’re not using RDF for anything. Also, and maybe I’m imagining it, but in the 10-minute wrapup at the end of the podcast I think Tom Heath basically takes a some veiled jabs at the Glue guys for being SemWeb poseurs.