Attach behavior only to the file upload button
You can simply be more specific with your jQuery selector. For example, you could theme_file_managed_file to give your upload button it's own class and then change your jQuery('input.form-submit', context) line to include that class in the selector.

jQuery('input.my-upload-class', context)