How should trigger the 5V relay with an ESP8266? Using PC817 or transistor? What are the pros and cons?
The first circuit:


is used when the relay VCC is higher than the driver (in your case ESP8266)
provides electrical decoupling protection


The second circuit:


is good when the relay voltage is the same as the driver's
has lower cost than previous circuit
you might have problems making the transistor go into saturation when used with higher voltage on the relay