I'm working on an opensource project implementing deduplication. (For a link to the project, see the two hyperlinks below)
Performance of the project currently is quite okay, but degrades as more blocks get written to disk.
This is due to the HashManager. For each block written, the hashmanager stores a Hash-BlockId pair. For the deduplication process, a list of block identifiers is needed which have a given hash. (hash used is Crc32)
For the interface of the HashManager, see [the source](https://github.com/terwoord/overdrive/blob/master/Sources/Core/IHashManager.cs).

The current [implementation](https://github.com/terwoord/overdrive/blob/master/Sources/Core/Implementations/SimpleHashManager.cs) of the interface stores the lists in 256 files (crc & 0xFF), and loads a complete list into memory. When another list is needed, the previous list is saved and the next one is loaded. Besides the fact that this could cause memory exhaustion, this means degrading performance.

What good options are there to overcome the issue?

(Just to clearify: blocks are checked completely to see if they match before deduplicating)