Any way to customize default nomask used in file_scan_directory() to allow Node.js modules?
As far as Drupal/PHP are concerned, no, there's no option other than either changing the default arguments to the function, or finding all calls to that function that don't include the 2nd arg, and updating things there. Either way you're likely to have to patch a core method or two.

NPM's convention is to use a folder called node_modules. They don't want to change that (https://github.com/npm/npm/issues/2734, https://github.com/npm/npm/issues/775), but there is a prefix config option which may be suitable for your needs.