Why does changing "Open with" also change a files "Modification date"?
Some experimentation using the xattr command shows this:


Label information is stored in the com.apple.FinderInfo extended attribute, which every file should have by default.
Custom application associations are stored in the com.apple.ResourceFork attribute, which not every file has.
Updating the FinderInfo attribute does not cause the modification date to be modified.
Creating/updating the ResourceFork causes the modification date to be modified.


The resource fork historically had a special place in Mac file systems. I'm not sure if the com.apple.ResourceFork extended attribute is the same as a resource fork, if the latter was superseded by the former or how else they are connected.

If there's a technical difference between an extended attribute and a resource fork, it may be "necessary" for a file to be modified when the resource fork is updated, but not when the extended attribute is written.
It may be that the (legacy) API for updating resource forks updates the file modification date in the process, while the extended attribute API doesn't. The latter possibly in error, possibly because of different philosophies.

Vague information, but it might give a little more insight. In the end though you're right. Some meta information modification causes the modification date to be updated, others doesn't. Whether there's a reason behind this or if it's just an oversight… only a few guys at Apple may know. :)