This blog is a follow up on our previous blog “Snapshots for HDFS”
In June we had posted an early prototype of snapshots that allowed us to experiment with a few ideas in HDFS-2802. Since then we have added more details to the design document and made significant progress on a brand new implementation (over 40 subtasks in HDFS-2802).
Some of the highlights of this new design include:
An initial implementation of snapshots with some tests is already completed. We are now working on the improvements, some new tools for snapshots and adding more tests.
The major work-in-progress items are:
Meetup at Hortonworks: rallying the community
Recently we also held a Meetup at Hortonworks office, where over 30 folks attended to discuss the design and some of the features in great detail. A wide range of topics were discussed from Snapshot usage by HBase, administration aspects of snapshots, overhead of creating and maintaining snapshots, and lower level details such as length of files that are open-for-writing. We had representation from HDFS developers, HBase developers and engineers with deep experience in managing hadoop and other storage systems. We thank the community for the valuable discussion and feedback on the feature requirements and the open questions.