Why use UART to communicate between Arduino and ESP Wi-Fi module?
UART offers two way communication with just 2 wires. 

I2C is one way communication for 2 wires. It can be made two way by implementing a custom notify mechanism where slave sends a signal whenever it has some data to send using a GPIO connection. This requires another pin and hence 3 wires for two way communication as compared to two in UART. Another alternative is polling as pointed out in comments below. Personally I prefer interrupt driven flow over polling. However if you are OK with polling, 2 wires can indeed do the job.

SPI requires a minimum of 4 wires. (Maybe 3 wires in case of single slave as Elliot commented)

Given the limited number of pins available on esp modules, I guess community might have chosen to use UART as the primary data transfer mechanism. Plus UART is simpler to implement than other counterparts. (This line could be opinion based so please pardon me if you tend to disagree)

It's possible to use esp modules as a stand alone MCU for general projects. You can use port expanders in case more GPIOs are needed. I used a similar architecture in one of my projects. I used esp-arduino for developing the firmware inside arduino IDE itself.