Running 'composer update' replaces module folders
Is this normal behavior for composer?


Yes


  Is there some way to specify that I only want to replace what's going into the vendor directory?


Sure, remove the reference to the module(s) from composer.json. Obviously at that point you’ll need to manage any updates for them yourself.

A better resolution would be to install Composer patches, and patch the modules automatically as they’re being installed/updated.