How to make 'Views' consider contextual filter value to be 'all' automatically?
I don't know if it clashes with your configuration, but usually you use the option Display all results for the specified field under When the filter value is NOT available. 



There is also the option under When the filter value IS available or a default is provided > Specify validation criteria: 

Action to take if filter value does not validate

where you can choose to Display all results for the specified field. Maybe you can combine it with some Validation conditions to achieve what you want.