Bicycle touring post-mortem

San Francisco to San Diego was my first multi-day tour, and overall I’m very happy with how it went. I’ve done plenty of overnight bike tours to Half Moon Bay or Samuel P. Taylor, and I carried basically the same kit on the multi-day tour.

Here’s a breakdown of what went well and what I might do differently on my next tour. I’m really eager to do the northern section of this route (Seattle or Portland to San Francisco), perhaps next year.

Continue reading →

Posted by andrew in Travel, 0 comments

Riding the 101: Bicycle Touring Mega-Update

I spent about two weeks of my sabbatical riding from San Francisco to San Diego (Jul 31- Aug 15). I got in the habit of posting end-of-day recaps to Facebook and Strava as I went, which really helped me reflect on what happened. I’m reposting all of them here as a mega-post.

Total distance: 622.9 miles

Total climbing: 24436 feet

Total riding time: 55.85 hours

Sunsets on the beach: whenever possible

Continue reading →

Posted by andrew in Travel, 0 comments

Blog refresh: WordPress

I’ve come full-circle. My very first websites circa-2005 were built with a CMS (Joomla or WordPress). I started messing with custom themes and plugins (which is how I really learned to code) then drank deeply of the semantic web koolaid and started hand-coded everything in XHTML, CSS, and PHP. I migrated to a static site generator seeking simplicity and reduced hosting costs, and now, umbrant.com is again powered by a full-fledged CMS: WordPress.

Continue reading →

Posted by andrew, 1 comment

The Next Generation of Apache Hadoop

Apache Hadoop turned ten this year. To celebrate, Karthik and I gave a talk at USENIX ATC ’16 about open problems to solve in Hadoop’s second decade. This was an opportunity to revisit our academic roots and get a new crop of graduate students interested in the real distributed systems problems we’re trying to solve in industry.

This is a huge topic and we only had a 25 minute talk slot, so we were pitching problems rather than solutions. However, we did have some ideas in our back pocket, and the hallway track and birds-of-a-feather we hosted afterwards led to a lot of good discussion.

Karthik and I split up the content thematically, which worked really well. I covered scalability, meaning sharded filesystems and federated resource management. Karthik addressed scheduling (unifying batch jobs and long-running services) and utilization (overprovisioning, preemption, isolation).

I’m hoping to give this talk again in longer form, since I’m proud of the content.

Slides: pptx

USENIX site with PDF slides and audio

Posted by andrew in Talks, 0 comments

Distributed testing

I gave a presentation titled Happier Developers and Happier Software through Distributed Testing at Apache Big Data 2016, which detailed how our distributed unit testing framework has decreased the runtime of Apache Hadoop’s unit test suite by 60x from 8.5 hours to about 8 minutes, and the substantial productivity improvements that are possible when developers can easily run and interact with the test suite.

The infrastructure is general enough to accommodate any software project. We wrote frontends for both C++/gtest and Java/Maven.

This effort started as a Cloudera hackathon project that Todd Lipcon and I worked on two years ago, and I’m very glad we got it across the line. Furthermore, it’s also open-source, and we’d love to see it rolled out to more projects.

Slides: pptx

Source-code: cloudera/dist_test

Posted by andrew in Talks, 0 comments

Windows Azure Storage

What makes this paper special is that it is one of the only published papers about a production cloud blobstore. The 800-pound gorilla in this space is Amazon S3, but I find Windows Azure Storage (WAS) the more interesting system since it provides strong consistency, additional features like append, and serves as the backend for not just WAS Blobs, but also WAS Tables (structured data access) and WAS Queues (message delivery). It also occupies a different design point than hash-partitioned blobstores like Swift and Rados.

This paper, “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency” by Calder et al., was published at SOSP ’11.

Continue reading →

Posted by andrew in Reviews, 0 comments

Transparent encryption in HDFS

I went on a little European roadshow last month, presenting my recent work on transparent encryption in HDFS at Hadoop Summit Brussels and Strata Hadoop World London. I’ll also be giving the same talk this fall at Strata Hadoop World NYC, which will possibly be the biggest audience I’ve ever spoken in front of.

Slides: pptx

Video: Hadoop Summit Brussels (youtube)

If you have access to O’Reilly, there should be a higher quality video available there.

Posted by andrew in Talks, 0 comments

Mesos, Omega, Borg: A Survey

Google recently unveiled one of their crown jewels of system infrastructure: Borg, their cluster scheduler. This prompted me to re-read the Mesos and Omega papers, which deal with the same topic. I thought it’d be interested to do a compare and contrast of these systems. Mesos gets credit for the groundbreaking idea of two-level scheduling, Omega improved upon this with an analogy from databases, and Borg can sort of be seen as the culmination of all these ideas.

Continue reading →

Posted by andrew in Reviews, 0 comments

Bucket list: Catch a fish and eat it

I checked off one of my bucket list items yesterday: catching a fish, cleaning it, and eating it.

This was the last day of a family vacation in Port St. Lucie in Florida. My original plans to go deep sea fishing fell through, so I went to the surprisingly well-stocked local Walmart to pick up some freshwater gear. I was lucky enough to nab a healthy-looking 15″ largemouth bass with a silver Mepps spinner from the lake behind our timeshare.

Continue reading →

Posted by andrew in Personal, 0 comments

Paper review: Facebook f4

It’s been a while since I did one of these! I did a previous review of Facebook Haystack, which was designed as an online blob storage system. f4 is a sister system that works in conjunction with Haystack, and is intended for storage of warm rather than hot blobs. As is usual for Facebook, they came up with a system that is both eminently practical and tailored for their exact use case.

This paper, “f4: Facebook’s Warm BLOB Storage System” by Muralidhar et al., was published at OSDI ’14.

Continue reading →

Posted by andrew in Reviews, 0 comments