Reliability Analysis of Deduplicated and Erasure-Coded Storage

    Space efficiency and data reliability are two primary concerns for
modern storage systems.  Chunk-based deduplication, which breaks up data
objects into single-instance chunks that can be shared across objects,
is an effective method for saving storage space.  However, deduplication
affects data reliability because an object's constituent chunks are
often spread across a large number of disks, potentially decreasing the
object's reliability.  Therefore, an important problem in deduplicated
storage is how to achieve space saving yet maintain each object's
original reliability.  In this paper, we present initial results on the
reliability analysis of HP-KVS, a deduplicated key-value store that
allows each object to specify its own reliability level, and that uses
software erasure coding for data reliability.  The combination of
deduplication and erasure coding gives rise to several interesting
research problems.  We show how to compare the reliability of erasure
codes with different parameters, and we show how to analyze the
reliability of a big data object given its constituent parts'
reliabilities.  We also outline several research challenges in designing
large scale reliable deduplication systems.