MQTT bridging with Arduino and ESP8266 - What software should I use? Sorry for the vagueness in my question, I am just looking for ideas to point me in the correct direction please...

I wish to connect many (read: 20+) different sensors to a local wifi network and have them all datalogged and controlled by a single server on the network. Pretty standard stuff for the home automation people etc. Some sensors are connected to esp8266 boards directly, others communicate via serial and will employ an 8266 to provide wifi.

I am having trouble deciding which direction to go or which software / firmware to combine since its changing rapidly and a fair number of tutorials are around but all have different approaches and end goals. 

This must be entirely contained offline - I have a raspberry pi, a wireless router, and the various sensors (mostly arduino based).

Now from research it would seem I need to setup a serial to MQTT 'bridge' - and have the esps 'talk' to the pi by MQTT. And that the pi should be running 'mosquitto' if I wish the MQTT to be bi-directional (i.e. I can receive serial data AND send serial commands. I have never dealt with MQTT so I am trying to learn as I go.

I have been working for a time with 'ESP-Link' software and had success using it as a simple serial to tcp-ip bridge and using virtual serial ports to communicate with the attached device. However using virtual serial ports on the server is clunky, unreliable and not what I wish to do - does anyone know if it can be setup so I can send 'raw' serial messages over TCP to the server for use? And from the server to control the connected devices? Or if there is an easier way? I cannot modify all of the serial devices I wish to connect, so just want to get the data out/in and have it manipulated by the server where necessary.

In essence I wish to end up with several sensors all connected to the router, and logged / controlled through a web interface hosted on the PI, all 'offline' and contained. Eventually I would like the server to be able to recognize what devices it can 'see' automatically, however at this moment I simply wish to gain insight into how I can most easily achieve this? Or even if I am completely mistaken in my approach?

Thanks
MQTT is perfectly suited to this application. Here is an overview of a possible configuration. You should be able to find tutorials about each piece of the system. 

On the Raspberry Pi:


Setup a Mosquitto server that can receive messages published by the ESP devices.
Write and run a MQTT client on the RPi that subscribes to the messages published by the ESP devices. This client would then store the data in some sort of permanent data store (database, flat files, whatever).
Write and run on the RPi a web app that reads and presents the data from the data store.


On the ESP devices:


Connect to the same wireless network as the RPi (or set up the RPi as an access point and have the devices connect to it)
Read data from the sensors and publish it to the MQTT server on the RPi.


If you want to push data to the ESP devices -- have them subscribe to certain messages (topics) on the MQTT server and then publish messages in those topics on the server.  Your web app could do this. 

All of this can be done on an isolated network.