Craft assets indexing wants to remove files that still exist
This has happened to me in a couple scenarios:


When I am working across multiple environments and forgot to sync assets properly. (i.e. I pulled down the database snapshot from production, but not the newer assets that would be referenced in its Assets index).
When one environment uses a case-sensitive file system (e.g. a unix host), and one uses a case-insensitive file system (e.g. OSX). In this case, syncs and Git pulls may fail to differentiate between files with similar names, and both Git and Assets may bugger up.


Those are the first two things I'd double-check.