Substituting Arduino for ESP8266 in a (Yet Another!) Garage Door Controller
The H11AA1M has a minimum CTR (Current Transfer Ratio) of 20%. That means if you have 20mA going through the emitter (LED) then the detector (transistor) may only be able to switch 20mA * 20% = 4mA.

You measured 50mA 'trigger' current when the garage door opener input was shorted, so you should be using an optocoupler that will switch that much current. ESP8266 GPIOs are only guaranteed to supply 12mA maximum, so your optocoupler should have a CTR of at least 50mA/12mA = 4.2.

So why does it work after removing R2? Your particular ESP8266 might manage to drive 20mA into a 100&ohm; resistor and infrared LED, and your individual optocoupler unit might achieve a CTR of >1.5 at 20mA. However the fact that it can't handle an extra 100&ohm; on the output suggests that it is barely working. It may fail in colder or hotter weather, or when the optocoupler transistor overheats because it is out of saturation.     

For reliability you should increase the value of R1 to draw no more than 12mA from the ESP8266's GPIO, and change the optocoupler to one with higher CTR, eg. 4N33 or H11B3.