EBS Pre-warming Matters

This topic is inspired by the above tweet.  EBS volumes (Amazon's equivalent to a SAN as a Service) have different performance characteristics before and after their first reads and writes.  The effects aren't clearly documented by Amazon, so I figured it was worth sharing my two key findings:

1) Reading every block of a snapshot is a Good Thing.  This is partially because of the Amazon documented '5-50% decrease in IOPS' on first read; but my primary driver for doing this is to cut off the high-latency tail.  I see a small number of blocks that take more than 10,000ms on first-read whenever I start using cold restored volumes.  It's much better to fix those with a simple 'cat device > /dev/null' than it is to let the application trip over them at some random future moment.

2) Writing to every block of a new volume is a mixed bag.  It provided a small but measurable win on my synthetic benchmarks, but it didn't make a material difference to any of my actual applications.  Unfortunately, it does increase EBS snapshot costs, so writing was a loser for me.