what will happen if an optocoupler output is connected to AC?
The datasheet (I recommend linking one in the question, because it can differ slightly between manufacturers!) specifies an emitter-collector breakdown voltage of 7V. This means that if the emitter is over 7V higher than the collector, which would happen during the negative portion of your AC signal, it will not function correctly, probably breaking it.