What are the formulae for calculating the thermal heating effect of back radiation in greenhouse gases?
As you have mentioned already, there are several physical phenomena which are responsible for heat and energy transfer. I'll just list them again for completeness, although you already addressed them in your question.


Conduction
Convection
Latent heat of vaporization
Radiation


To get to your question: Radiation is the transport of energy by means of electromagnetic waves emitted from a body of a absoulte temperature uneqal zero Kelvin.

What does that mean? Radiation does not rely on contact or a transporting fluid as do conduction and convection. Instead it is equivalent to the fourth power of its absolute Temperature. This is what we know as "Black Body Radiation".

The equation for this is:

$$ E = \sigma * T^4$$

where σ is the Stefan-Boltzman Constant and T is the absolute Temperature in Kelvin. This law is obtained by integrating Planck's law over the solid angle and wavelength, in order to determinate the energy which is transmitted not only by one particular wavelength, but the whole spectrum and in all directions. Combined with Kirchhoffs Law, which says, that the absorbtivity of an object equals its emissivity, you can calculate the radiated Energy of any object or atmospheric layer.
The emissivity (ε) of a body is a value between zero and one with one being the perfect black body. For most things you can find values in the web. It also explains why aluminum foil is such a good insulator.

$$ E = ε * \sigma * T^4$$

To put all of this in an example:
Everything around you radiates heat. Beginning with the most obvious, the sun. The sun's surface has an approximate temperature of 6000K. This means it emits radiation of the following spectrum:



As you can see, the maximum of the energy of the sun is radiated in the part which we know as visible light. Also a big part of the radiated energy is absorbed in the atmosphere (the difference between yellow and red graphs). Responsible for these are particularly Ozone (O3), Water (H2O) and (CO2) which are also our most important greenhouse gases. 

What happens, is that the radiated energy of a particular wavelength "hits" the molecules and excites them into a state of higher kinetic energy. The average random kinetic energy of a substance is what we know as the temperature of this substance. Hence, also the atmosphere acts like a black body and radiates energy 
back to space but also back to earth. 

Unfortunately I can't really say much about the second part of your question. I don't know about the original experiments. But I suggest a read about Gustav Kirchhoff, who introduced the idea in 1860 which then has been proceeded by Josef Stefan in 1879 and Ludwig Boltzman (1884). Also Planck's works from 1900 could be helpful.

I hope I was able to point you in the right direction with your studies.