what limits the maximum voltage of this optocoupler (6N137)?
The original 6N137 (designed by HP over fourty years ago) is specified only for 5 V (± 10 %), but actually regulates its supply:


  
  (source: Optoelectronics Applications Manual)


Vishay's 6N137 does not copy this circuit (it has hysteresis, and the output transistor is a MOSFET), so it is not known whether it has a regulator. But the use of a MOSFET points to a possible failure more, which is described (for similar transistors in completely different chips) in TI's AHC Designer's Guide:


  Input voltages greater than 7 V must be avoided to preclude damage to the gate oxide of the input stage. This damage is not necessarily permanent, but will adversely affect the expected lifetime of the circuit. The gate oxide of AHC devices is only 200 Å thick. An input voltage of 7 V corresponds to a field strength over the gate oxide of 350 kV/cm. Although breakdown of the oxide is expected only at input voltages above 10 V, electrons tunnel increasingly into the gate oxide at field strengths greater than 350 kV/cm, influencing characteristics of the transistors and causing failure.


So it is possible that the 12 V has already done some damage to your chip.

In any case, the presence of the regulator shows that the circuit is likely to be sensitive to changes in the supply voltage, so it is quite possible that the chip runs outside the guaranteed electrical characteristics.