How does OpenVPN work? I'm confused with the technologies that OpenVPN uses. The official site mentions it uses SSL/TLS in-place of IKEv1/IKEv2, and uses IPSec ESP to secure the data, and yet that it isn't compatible with IPSec.

OpenVPN's security model is based on using SSL/TLS for session authentication and the IPSec ESP protocol for secure tunnel transport over UDP. OpenVPN supports the X509 PKI (public key infrastructure) for session authentication, the TLS protocol for key exchange, the OpenSSL cipher-independent EVP interface for encrypting tunnel data, and the HMAC-SHA1 algorithm for authenticating tunnel data.
Use all of the encryption, authentication, and certification features of the OpenSSL library to protect your private network traffic as it transits the internet
OpenVPN is not compatible with IPSec, IKE, PPTP, or L2TP.

Source: https://openvpn.net/index.php/component/content/article/55.html
Also wikipedia info:

It uses a custom security protocol[9] that utilizes SSL/TLS for key exchange
OpenVPN uses the OpenSSL library to provide encryption of both the data and control channels

Source: https://en.wikipedia.org/wiki/OpenVPN
OpenVPN uses two different protocols. For the standard and most common settings, a TLS link called the control channel is used to exchange a symmetric key. This symmetric key is then used to encrypt a second protocol, the data channel, which uses the OpenVPN protocol itself. The protocol is not identical to IPSec, but it is based off of IPSec ESP, a protocol for secure transport of data over the UDP protocol, which plain TLS cannot do (at the time the protocol was created, DTLS, which is TLS over UDP, did not yet exist). Note that this means that you may be using multiple ciphers for encryption. You may find the control channel using AES to exchange a key that will be used to encrypt the data channel with Blowfish.

Using the control channel with TLS is not the only way to key the data channel. You can also configure OpenVPN to use preshared static keys which will directly be used to derive the symmetric key for data channel encryption. This loses forward secrecy, but gets around the need to use the control channel to exchange a key using TLS.