Docker ADD always invalidates cache
Check the image history of your two images with docker image history. Pay careful attention to the sha256 checksum of the files being added on the ADD line. Docker performs a checksum on the files, including some of the metadata on these files, so a single byte change or a permission update will result in a cache miss.

Once you've checked this, if it is not your issue, please update the question with the pull command, the build command and full output, and the history command on both images.

Side note: you likely want to use COPY instead of ADD unless you are pulling remote files or extracting a tar file into a directory.