Firstly, detecting whether a file entering the pipeline has been seen before is critical to reducing the computational strain on the overall pipeline. To avoid reprocessing of duplicate data, this generator will store a hash of the generator and each input in the DBgen Log Database discussed above. This enables DBgen to automatically detect whether it has seen an extracted input before and, if so, it can skip reprocessing the duplicate row. While this may seem trivial for a path to a file, this duplicate detection generalizes to complex inputs that are aggregated and processed from many entities within the schema.