Jay Luker

Starting MongoDB on CentOS with NUMA disabled

Wed, 01 Aug 2012 00:00:00 +0000

This may not be the best way to do this but it works for me. I got fed up with seeing the following message in the logs every time MongoDB was restarted.*

Wed Aug 1 12:06:39 [initandlisten] ** WARNING: You are running on a NUMA machine.
Wed Aug 1 12:06:39 [initandlisten] * We suggest launching mongod like this to avoid performance problems:
Wed Aug 1 12:06:39 [initandlisten] * numactl –interleave=all mongod [other options]

I’m using a pretty boilerplate init.d script for mongo so I figured it would be simple to update the start command to use numactl. What I discovered is that my init script uses a builtin bash function called daemon to start the mongo process. daemon allows for a –user option. Unfortunately, numactl does not. Neither is it possible to execute a bash function using numactl. “TO THE GOOGLES!”

Hmm, everything I see recommends wrapping the numactl command around another command called start-stop-daemon. OK, but CentOS doesn’t have a start-stop-daemon command. Argh.

Finally I resorted to digging into the daemon function to see what it was doing and came up with this:

numactl --interleave=all runuser -s /bin/bash $MONGO_USER -c "$mongod $MONGO_OPTS"

It works. Moving on now.

["Quinoa Tabbouleh Salad w/ Preserved Lemon"]

Mon, 28 Jun 2010 00:00:00 +0000

I say “salad” as this doesn’t have near the ratio of parsley to grains as a standard tabbouleh would have. Still, the ingredient list matches up. So far I’ve only made it using the Red Quinoa available from Trader Joe’s. Gives it a nice, hearty color.

As for the preserved lemons, this is something I’ve gotten into ever since a friend with a too-small kitchen unloaded a large mason jar of them on me. I’ve since made my own following this guide. It’s really remarkably easy and they give lots of dishes a remarkably tasty flavor. Jennifer loves them.

Exploring Astronomy Dataset Links with GridWorks

Thu, 27 May 2010 00:00:00 +0000

At ADS we are looking at new ways to index and provide full text searching for the Astronomy and Physics literature we manage to obtain, either through scanning + OCR of historical content, or from digital material provided by some publishers. Two options we’re looking at are Apache Solr and CDS-Invenio. But that’s not what this post is about.

While parsing and indexing a pile of about 42k articles from the past dozen or so years of the ApJ, AJ, ApJL and ApJS, formatted in the NLM XML schema, I noticed that many of the articles contained external links to various things, most interestingly, astronomical datasets.* My first thought was, “hmm, I wonder what’s at the other end of all those links…,” followed closely by, “hey, crawling those links would make a nice dataset to load into that nifty new Freebase Gridworks tool I heard about the other day.” So that’s what I did.

Out of 13652 articles there were 33600 total links which fell into three categories: http urls (28555), dataset links (938) and supplement links (4107). Dataset links consist of an identifier that looks something like ADS/Sa.CXO#obs/927. To get the goods you have to feed that id to a resolver which, assuming a valid identifier, will redirect you to the real location of the dataset. Supplement links took a bit more head-scratching as their values consisted of just a relative file name, like datafile3.txt or 69491.figures.html. We figured out that the solution was to append the filename to the publisher’s URL for the article, e.g., article and dataset or article and figures.

The ultimate objective was to load the results of crawling these links into Gridworks, but that means getting the data into csv or tsv form. Rather than have the crawl script output straight to csv, I stash the results in a MongoDB instance. Here’s an example of one of the resulting json documents in Mongo:

{u'_id': ObjectId('4bfc3737a1f714263b000012'),
u'anchor_text': u'http://astronomy.swin.edu.au/staff/dforbes/glob.html',
u'bibcode': u'2001ApJ...556L..83F',
u'content': u'\n\nDuncan A. Forbes, Swinburne University, Globular Clusters\n\n\n

Globular Cluster Research

\n\nI am interested in various aspects of Extragalactic Globular\n Cluster research. In particular the formation and evolution\n of Globular Cluster Systems and their host galaxies. \n
\n\n

GLOBULAR CLUSTER PHOTOMETRY DATABASE

\n\n

GLOBULAR CLUSTER SPECTRAL DATABASE

\n\n\n

GLOBULAR CLUSTER REVIEW PAPERS

\n\n\n

SAGES PROJECT

\n\n

\n\t HREF="http://www.physics.mcmaster.ca/resources/fs3_resources.html"> HARRIS DATABASE\n

\n\n\n\n

\n\n \n',
u'context': u'

The combined sample data are available at http://astronomy.swin.edu.au/staff/dforbes/glob.html.

\n',
u'doi': u'10.1086/323006',
u'ft_source': u'/proj/ads/articles/sources/AAS/ApJL/2001/556/2/323006/323006.xml',
u'link_id': u'http://astronomy.swin.edu.au/staff/dforbes/glob.html',
u'link_type': u'UrlLink',
u'response': {u'accept-ranges': u'bytes',
u'content-length': u'781',
u'content-location': u'http://astronomy.swin.edu.au/~dforbes/glob.html',
u'content-type': u'text/html; charset=UTF-8',
u'date': u'Tue, 25 May 2010 10:14:07 GMT',
u'server': u'Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/0.9.8e-fips-rhel5',
u'status': u'200'},
u'solr_id': u'31908',
u'url': u'http://astronomy.swin.edu.au/staff/dforbes/glob.html',
u'xpath': u'/html/article/body/sec[5]/fn-group/fn/p/ext-link'}

From there it was easy to dump what I needed to csv and load into Gridworks. I’m not going to get into how totally awesome the Gridworks software is, except to say you should watch the demo videos.

I can’t post the entire Gridworks project, but here’s some screencaps, a column list and some of the more interesting facets.

Initial data load plus some derived columns

Column list:

Id of the MongoDB doc
Id of the solr doc
ADS bibcode identifier of the article
Publication year – derived from the bibcode
DOI
xpath expression of the element
parent tag – the containing element type
link context – the containing element’s serialized xml contents
link type – one of url, dataset or supplement
anchor text – the text contents of the
full text source file
journal
full text source – publisher
extlink id – either the url or the dataset id or the supplement filename
domain – derived from the url
status – http status returned when requesting the resource
content-type – content-type header returned in the response
mimetype – derived from the content-type response header
location – the final url of the resource following any redirects
content length
response headers – list of all the header attribute names return in the response (just to see what other interesting stuff might be there)

<p class="wp-caption-text">
  Still to be determined how many of the url links point to some kind of data
</p>

<p class="wp-caption-text">
  Knowing the container could help parsing out something about the semantics of the link
</p>

  <p class="wp-caption-text">
    ~70% 200's was more than I expected. Of course 200 doesn't mean it actually found something interesting.
  </p>
</div>

<div id="attachment_151" style="width: 283px" class="wp-caption aligncenter">
  <a href="http://blog.reallywow.com/static/uploads/2010/05/gridworks_contenttype.png"><img class="size-full wp-image-151" title="gridworks_contenttype" src="http://blog.reallywow.com/static/uploads/2010/05/gridworks_contenttype.png" alt="" width="273" height="473" /></a>

  <p class="wp-caption-text">
    would have hoped for fewer text/html
  </p>
</div>

<div id="attachment_152" style="width: 285px" class="wp-caption aligncenter">
  <a href="http://blog.reallywow.com/static/uploads/2010/05/gridworks_domain.png"><img class="size-full wp-image-152  " title="gridworks_domain" src="http://blog.reallywow.com/static/uploads/2010/05/gridworks_domain.png" alt="" width="275" height="321" /></a>

  <p class="wp-caption-text">
    All the gcn.gsfc.nasa.gov hits look like observation reports, like this one, which I think is a good thing
  </p>
</div>

<p style="text-align: left;">
  Finally a thanks to <a href="http://dysinterested.com/">Sean Hannan</a> who worked out <a href="http://gist.github.com/414927">a hack</a> to a bit of the Gridworks javascript that automatically turns any cell values beginning with &#8220;http://&#8221; or &#8220;https://&#8221; into active links. The nice thing about that was it let me turn the column containing the MongoDB id into a link to a little <a href="http://webpy.org">web.py</a> script that dumps a JSON representation of the document.
</p>

<p>
  * NLM allows for links to external resources using either <a href="http://dtd.nlm.nih.gov/articleauthoring/tag-library/2.3/n-ju50.html"><ext-link></a> or <a href="http://dtd.nlm.nih.gov/articleauthoring/tag-library/2.3/n-2hw0.html"><supplementary-material></a> elements.
</p>

Embedding citation metadata in the ADS HTML

Mon, 01 Mar 2010 00:00:00 +0000

Here’s what I know: you can embed a set of tags containing citation metadata in your HTML to help Google Scholar to index your content. We’ve been doing it at ADS for quite a while. I’m not certain if the impetus came directly from Google, or, more likely, we got the idea from a CrossTech blog post by Tony Hammond that describes the technique.

For example, if you execute  curl -s http://adsabs.harvard.edu/abs/1977NuPhB.126..298A | grep meta you should see:

...
<meta name="citation_language" content="en" />
<meta name="citation_doi" content="10.1016/0550-3213(77)90384-4" />
<meta name="citation_abstract_html_url" content="http://adsabs.harvard.edu/abs/1977NuPhB.126..298A" />
<meta name="citation_title" content="Asymptotic freedom in parton language" />
<meta name="citation_authors" content="Altarelli, G.; Parisi, G." />
<meta name="citation_issn" content="0550-3213" />
<meta name="citation_date" content="08/1977" />
<meta name="citation_journal_title" content="Nuclear Physics B" />
<meta name="citation_volume" content="126" />
<meta name="citation_firstpage" content="298" />
<meta name="citation_lastpage" content="318" />
...

Since first implementation we’ve had some back-and-forth with Abhishek Jain at Google Scholar to ensure we’re making use of the full set of fields that Google Scholar looks for.*

Dan Chudnov, David Bucknum & Ed Summers at the LoC recently expressed interest in also embedding these tags. In the absence of official reference from the Google Scholar folks, I figured it would be a good thing to post here.

citation_language
citation_doi
citation_abstract_html_url
citation_title
citation_authors
citation_issn
citation_date
citation_journal_title
citation_volume
citation_firstpage
citation_lastpage
citation_publisher
citation_issue
citation_pdf_url
citation_pmid
citation_keywords (multiple instances OK)
citation_conference
citation_dissertation_name
citation_dissertation_institution
citation_patent_number
citation_patent_country
citation_technical_report_number
citation_technical_report_institution

I had to cull this list via a visual scan of a long, forwarded e-mail thread. So, like I tried to insinuate above, it sure would be great if Google Scholar would publish an official reference to this schema somewhere.

all instances of the term “we” should really be read as “my boss, Alberto”.

Contextual Inquiry on the Cheap

Thu, 31 Dec 2009 00:00:00 +0000

I thought I’d share the interview outline I’ve been using to conduct some low effort contextual inquiry sessions with ADS users.

thumbnail links to google doc

Classic contextual inquiry, in which the researcher sits with or shadows a person in the context of the subject’s own working environment, is often conducted in 3+ hour sessions, frequently with all manner of video capturing equipment. My goal is cut that time down to 30 minutes, partly because this whole user research thing is supposed to be a part-time endeavor, and also because the majority of ADS users are PhD’s, and we all know just how valuable their time is.

So far I’ve only managed to conduct four of these interviews (with two more scheduled). Would love to get a total of 10. Since I don’t have access to video equipment I simply mash out typewritten, poorly spelled notes as fast I can. The notes have a stream of consciousness flavor, but the early indications are that the information gathered will be valuable.

Example notes:

refers to bibcode as "indexing thing". "not any use to me."
wrote a perl script that rewrites the bibcode into something understandabl
other strategies for searching for particular star: entering star name into abstract search or title search.
finds one article using abstract search.
mentions that he doesn't know boolean sytnax by memory
to find more tries going to simbad and finds alternate names for the star

Mad Anachronisms

Wed, 19 Aug 2009 00:00:00 +0000

Like the rest of the planet it seems, I’ve been consumed lately by the show Mad Men. Jennifer and I are still catching up via Netflix. It really is one of those truly great and remarkable shows that comes along too rarely.

[Warning: Insignificant spoiler in the next couple of sentences]

About midway through the second season we came to the somewhat infamous picnic scene. **** The Draper’s have taken the new Cadillac for a spin in the countryside. They relax and recline on a blanket. The grass is green, the breeze is mild, they talk about how rich they are. All is good. Then it’s time to pack it up and head home. We see Don stand up, stretch, smile, and chuck his empty beer can into the idyllic landscape as if he was tossing a baseball to his son. Betty pinches two corners of the blanket and gives it a lift & shake, distributing the paper plates, napkins and other picnic detritus across the grass. The trash begins to lightly flutter and drift down the slope. In 1963 the happy family piles into the car and motors away. Meanwhile, in 2009, we sit on the couch, jaws agape at this stunning spectacle of thoughtless littering.

It’s suprisingly shocking. There’s the shock of seeing it, and then there’s the shock at being so shocked. Every bone in your body wants to be repulsed, but the relativist mindset makes it difficult to fault the characters. As the writer points out in the DVD commentary, Iron Eyes Cody didn’t come along until 1971.

The scene also in a way briefly cracks open that narrative fourth wall in that it’s clear the writer/director is blatantly highlighting these banal actions to serve up a very in-your-face cultural anachronism. It’s only been 40 years, but wow have the dominant cultural attitudes about the environment changed.

We’ve since watched a few more episodes, but that scene still sticks with me, and lately it’s got me imagining someone sitting on their couch in 2049–or hovering in their Anti-Grav Lounger, or whatever–and passing judgment on our present day actions. It’s interesting to think what might be the contemporary equivalents of folks in the early 60s treating the planet like giant trash receptacle.

I’m guessing they’ll look back in horror at us actually throwing things–anything!–away in a garbage can rather than somehow recycling or composting.

You mean the water from the shower just drains away into the sewer?!?

They have apples in a Boston supermarket that were grown in New Zealand? Insanity!

I mean, can you imagine!?

Chimichurri Potato Salad

Sun, 16 Aug 2009 00:00:00 +0000

The quality of the potatoes from our farm share this year has been outstanding, especially the waxy, creamy red potatoes. Perfect for potato salads. We prefer vinegar based salads vs. mayonnaise, so I generally use a combination of mustard, vinegar, oil, dill, etc. Yesterday I had a beauty of an anaheim chile on hand and some fresh parsley, so I decided to try tossing the potatoes with a chimichurri sauce, which is a kind of Argentinian green sauce usually served with grilled meat. It was simply the best potato salad I’ve ever eaten.

Chimichurri Potato Salad

2 lbs red potatoes
1 pint green beans
2 cups packed parsley leaves
1/3 cup olive oil
1/4 cup red wine vinegar
1 anaheim chile, diced (optionally seeded)
1 T minced garlic
1/4 cup minced shallot
salt & pepper to taste

Scrub the potatoes clean (or peel if you must), cut into 3/4 inch chunks, and steam for about 15 minutes or until easily pierced with a sharp knife.

Trim the green beans and break into 1 inch pieces. Blanch the beans for a few minutes. If you’re clever you can re-use the boiling water from steaming the potatoes. When done shock them in some ice water to stop the cooking and then drain.

Add the remaining ingredients to a food processor and pulse several times until the parsley is broken down and the sauce gets a good, saucey consistency. More than salsa, less than pesto.

Pour the sauce over the still-warm potatoes and the beans. Mix. Chill. Devour.

The Clam Chowder Criteria

Mon, 20 Jul 2009 00:00:00 +0000

The following set of criteria is what jjzach & I use for the purposes of clam chowder quality evaluation. This system was devised in partnership with our friends, Kelly & Erika, on the back of a napkin about a year ago.

Thicknocity – the viscosity/thickness of the chowder. Is it brothy? Creamy?
Aroma – does it give off an appealing scent?
Clam Content – This can hopefully be gauged by a good stir. Quality of the clams would also fall under this criterion.
Temperature – When served, it can be too hot as well as too cold
Flavor Balance – aka, predominant flavor. Is there one? Does it drown out everything else?
Seasoning – aka, “Zip”
*Potato Quality *– chunk size, peel on or off (this could be good or bad), mushy or firm
*Absence of Grit *– clams live in sandy muck. Proper cleaning is required. Some tasters find a modicum of grit acceptable and see it as a sign of homemade-ness.
*X-Factor *– the unexpected!
*Digestibility *– how do you feel afterwards? How about 45 minutes into the ride home in the backseat?

These are mostly subjective measurements, of course. Some folks, for example, like their chowder thick and creamy and might rate highly a variation in which their spoon stands up in the cup. Needless to say, others would give that a very low Thicknocity score.

Potato Quality

Honey Garlic Chicken

Mon, 29 Jun 2009 00:00:00 +0000

Something about the smallish, red “bunching” onions that came in our farm share said, “cook me in the crockpot with some chicken”, so I improvised a sweet, garlicky dish that we ended up taking over to some friends’ for a “potluck”.

Honey Garlic Chicken

I like my chicken legs cooked long and slow so pretty much all the fat melts and the meat falls away from the bone. This goes great over boiled brown rice.

1 T cracked green peppercorns
1 t cinnamon
1 t salt
4 skin-on chicken legs (thigh + drumstick)
1 T peanut oil
1 cup orange juice
1/4 cup soy sauce or tamari
3 T honey
8-10 cloves of garlic, peeled
8-10 bunching onions or 16-20 pearl onions
2 T chopped thai or sweet basil

Mix the salt, pepper and cinnamon. Rub the mixture between the skin and meat of the chicken legs.

Heat the oil over medium-high until just beginning to smoke. Sear the legs, skin side down first, for 3-4 minutes each side. Transfer legs to the crockpot, skin side up.

Add orange juice to the sauté pan and scrape up any browned bits. Add soy sauce and honey and stir until combined. Pour sauce over chicken.

Spread onions and garlic evenly atop and around the legs. Cover and cook on low for four hours.

Transfer chicken to a serving dish and cover or tent with foil to keep warm. Strain sauce back into the sauté pan, reserving onions and garlic, and boil until reduced by half. Add onions and garlic to serving dish. Pour reduced sauce over everything, sprinkle with the basil and serve.

CSA Spanikopita

Mon, 22 Jun 2009 00:00:00 +0000

One of the shareholders in our CSA contributed this recipe to the newsletter. I haven’t made it yet, but wanted to save it somewhere outside of my inbox. Our shares have included large quantities of greens, so I’m always looking for ways to use them up.

Spanikopita w/ Assorted Greens

This is a slightly non-traditional spanikopita pie made with assorted greens rather than just spinach.

2 lbs greens (any kind you have mixed together – spinach, collard greens, swiss chard)
1/2 pound feta cheese, crumbled
2 eggs
several sheets of phyllo dough
2 Tbs butter, melted
1/8 tsp nutmeg

Saute the greens in a Tbs of water, covered, until wilted. Mix them with 2 eggs, feta cheese, and the nutmeg.

Butter a square baking dish (roughly 8×8) and put down a layer of phyllo dough. Lightly butter each layer of phyllo – if you are using standard thin phyllo put down about 5 layers on the bottom. Put in the green and feta mixture and then put several layers of phyllo on for the top “crust”, lightly buttering each layer as you go.

Bake at 375 for about 30 – 45 minutes or until phyllo dough is golden brown.