Finding or confirming absence of corrupted data in MongoDB Background

I have a directory full of .gz files that were produced by mongodump on a mongod running MongoDB 3.4.x, which I am writing into a different instance running MongoDB 3.4.2 using mongorestore ... --numInsertionWorkersPerCollection 8 --drop --nsFrom ... --nsTo .... All but one are getting loaded properly. They all take between 1 and 3 hours to load the data, and then another 2-4 hours to rebuild indexes. The problematic file hangs before the step below, and I usually kill the process externally after about a day. I've reproduced the scenario 4 or 5 times. 

finished restoring database.collection (N documents) 


During this time the mongorestore process is still visible from the shell, but does not have a corresponding entry in db.currentOp(), and the size of the database is not increasing.

These errors were returned right before the log line "restoring indexes for collection database.collection from metadata":

2017-03-13T09:30:04.218-0400    demux namespaceHeader: {database collection true 726142452659775357}
2017-03-13T09:30:04.218-0400    demux checksum for namespace database.collection is correct (726142452659775357), 54599167880 bytes
2017-03-13T09:30:04.218-0400    demux finishing (err:corruption found in archive; I/O error reading length or terminator ( gzip: invalid header ))


I have found a couple reports of mongorestore hanging (e.g.), and some documentation of issues with /s in namespaces in older versions (e.g.), but I haven't found a description of quite my situation.

Question

Now I find myself with some indication that a file I have loaded may have been corrupt, or at least had a corrupt header. Issuing realistic queries to the collection in question and its indices returns results that look reasonable. How can I definitively detect or rule out "corruption" of this type? What indeed does "corrupt" mean in this context? Can I expect to find it in the database, or only within the zip file?

Update: At @mustaccio's suggestion I ran gzip -t on the problematic file and it reported "trailing garbage". Is this sufficient to pin the problem reported by MongoDB on a bad input and stop looking for corruption in the database?
demux finishing (err:corruption found in archive; I/O error reading length or terminator ( gzip: invalid header ))


This error message refers to processing of the gzip archive rather than the BSON data within. Your test with gzip -t implies that the contents are likely OK and the error may be due to ignorable padding at the end of the archive (see: gzip complains with trailing garbage ignored).


  All but one are getting loaded properly. That one reproducibly hangs (indefinitely)


If the problem is easily reproducible, I would raise a bug report in the MongoDB Jira TOOLS project including more details such as the specific versions of mongodump and mongorestore and command line parameters used as well as how long you waited. It seems reasonable for mongorestore to have some sort of timeout or error handling for this case rather than hanging indefinitely; I also wouldn't expect mongodump to create files that the same version of mongorestore cannot handle.


  Now I find myself with some indication that a file I have loaded may have been corrupt, or at least had a corrupt header. Issuing realistic queries to the collection in question and its indices returns results that look reasonable.
    How can I definitively detect or rule out "corruption" of this type?  What indeed does "corrupt" mean in this context? Can I expect to find it in the database, or only within the zip file?


It's unlikely that you would be able to restore corrupt BSON documents into MongoDB, but there may be some nuances depending on how/where data is corrupted. In most cases invalid data will trigger an exception/assertion when the server attempts to read the BSON, but valid BSON does not necessarily mean that the field contents are as expected.

To be entirely certain you would have to compare the document counts or checksums of the restored data against the original data (assuming you still have a copy other than the gzip archive).