Mesos, Omega, Borg: A Survey

May 27, 2015

Google recently unveiled one of their crown jewels of system infrastructure: Borg, their cluster scheduler. This prompted me to re-read the Mesos and Omega papers, which deal with the same topic. I thought it'd be interested to do a compare and contrast of these systems. Mesos gets credit for the groundbreaking idea of two-level scheduling, Omega improved upon this with an analogy from databases, and Borg can sort of be seen as the culmination of all these ideas.

Transparent encryption in HDFS

May 27, 2015

I went on a little European roadshow last month, presenting my recent work on transparent encryption in HDFS at Hadoop Summit Brussels and Strata Hadoop World London. I'll also be giving the same talk this fall at Strata Hadoop World NYC, which will possibly be the biggest audience I've ever spoken in front of.

Slides: pptx

Video: Hadoop Summit Brussels (youtube)

If you have access to O'Reilly, there should be a higher quality video available there.

Bucket list: Catch a fish and eat it

December 30, 2014

I checked off one of my bucket list items yesterday: catching a fish, cleaning it, and eating it.

Paper review: Facebook f4

October 29, 2014

It's been a while since I did one of these! I did a previous review of Facebook Haystack, which was designed as an online blob storage system. f4 is a sister system that works in conjunction with Haystack, and is intended for storage of warm rather than hot blobs. As is usual for Facebook, they came up with a system that is both eminently practical and tailored for their exact use case.

This paper, "f4: Facebook's Warm BLOB Storage System" by Muralidhar et al., was published at OSDI '14.

<< Older posts