1. An apparatus for creating indices and loading key-value pairs for NoSQL databases, the apparatus comprising: a processor; and one or more stored sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of: creating a plurality of attributes that correspond to a plurality of records in a NoSQL database, wherein each attribute of the plurality of attributes comprises data from a corresponding plurality of record fields; creating an index based on the plurality of attributes; loading, in a memory, a plurality of attributes that correspond to a subset of the index as keys in a key-value pair and a plurality of identifiers that correspond to a plurality of records that correspond to the plurality of attributes as values in the key-value pair; sorting, in the memory, the plurality of attributes that correspond to the subset of the index; and identifying, in the memory, any duplicate attributes from the sorted plurality of attributes, wherein any identifiers that correspond to the any duplicate attributes also identify records in the NoSQL database to be evaluated as to whether the identified records are duplicates.