Windows Azure Storage

February 4, 2016

What makes this paper special is that (I believe) it is the only published paper about a production cloud blobstore. The 500-pound gorilla in this space is Amazon S3, but I find Windows Azure Storage (WAS) the more interesting system since it provides strong consistency, additional features like append, and serves as the backend for not just WAS Blobs, but also WAS Tables (structured data access) and WAS Queues (message delivery). It also occupies a different design point than hash-partitioned blobstores like Swift and Rados.

This paper, "Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency" by Calder et al., was published at SOSP '11.

Mesos, Omega, Borg: A Survey

May 27, 2015

Google recently unveiled one of their crown jewels of system infrastructure: Borg, their cluster scheduler. This prompted me to re-read the Mesos and Omega papers, which deal with the same topic. I thought it'd be interested to do a compare and contrast of these systems. Mesos gets credit for the groundbreaking idea of two-level scheduling, Omega improved upon this with an analogy from databases, and Borg can sort of be seen as the culmination of all these ideas.

Transparent encryption in HDFS

May 27, 2015

I went on a little European roadshow last month, presenting my recent work on transparent encryption in HDFS at Hadoop Summit Brussels and Strata Hadoop World London. I'll also be giving the same talk this fall at Strata Hadoop World NYC, which will possibly be the biggest audience I've ever spoken in front of.

Slides: pptx

Video: Hadoop Summit Brussels (youtube)

If you have access to O'Reilly, there should be a higher quality video available there.

Bucket list: Catch a fish and eat it

December 30, 2014

I checked off one of my bucket list items yesterday: catching a fish, cleaning it, and eating it.

<< Older posts