dm vdo: implement the delta index
authorMatthew Sakai <msakai@redhat.com>
Fri, 17 Nov 2023 00:52:58 +0000 (19:52 -0500)
committerMike Snitzer <snitzer@kernel.org>
Tue, 20 Feb 2024 18:43:14 +0000 (13:43 -0500)
commit024512e7326a27311de5158d5396eb39bbe030bc
tree7d7ca418cd2c77a8b4cfe80359a5c2d904ed17a6
parentb46d79bdb82aa12bf1101c1a38f0233cefe94cb1
dm vdo: implement the delta index

The delta index is a space and memory efficient alternative to a hashtable.
Instead of storing the entire key for each entry, the entries are sorted by
key and only the difference between adjacent keys (the delta) is stored.
If the keys are evenly distributed, the size of the deltas follows an
exponential distribution, and the deltas can use a Huffman code to take up
even less space.

This structure allows the index to use many fewer bytes per entry than a
traditional hash table, but it is slightly more expensive to look up
entries, because a request must read and sum every entry in a list of
deltas in order to find a given record. The delta index reduces this lookup
cost by splitting its key space into many sub-lists, each starting at a
fixed key value, so that each individual list is short.

Co-developed-by: J. corwin Coburn <corwin@hurlbutnet.net>
Signed-off-by: J. corwin Coburn <corwin@hurlbutnet.net>
Co-developed-by: Michael Sclafani <dm-devel@lists.linux.dev>
Signed-off-by: Michael Sclafani <dm-devel@lists.linux.dev>
Co-developed-by: Thomas Jaskiewicz <tom@jaskiewicz.us>
Signed-off-by: Thomas Jaskiewicz <tom@jaskiewicz.us>
Signed-off-by: Matthew Sakai <msakai@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
drivers/md/dm-vdo/delta-index.c [new file with mode: 0644]
drivers/md/dm-vdo/delta-index.h [new file with mode: 0644]
drivers/md/dm-vdo/hash-utils.h [new file with mode: 0644]