1. A data processing method comprising: collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records, by one or more processors, for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index; and storing, in a repository, the identified non-duplicate data records and maintaining, in the repository, a single copy of the identified duplicate data records in the first set of data records, wherein a first data record in the stream of data records is uniquely identifiable by way of a first identifier that comprises a first part and a second part, such that the first identifier is utilized to find whether or not the first data record has a duplicate in the repository.