Powering GSM Module from 2 x AA NiMH or 3 x AA alkaline batteries
The three alkaline AA cells will probably not work, due to the ESR of the batteries as you surmised.

While a GSM module draws 2A, it is only for a brief period, around 575 µS every 4.6 ms.

An AA alkaline battery can output 2A for a brief period, but because of the internal resistance of 80 mΩ you will lose almost 1/2 volt (.08Ω * 3 cells * 2A) while it is outputting the 2A.

This means you will have a maximum voltage available of 4.5V - 0.5V - 0.5V = 3.5v

where the first 0.5V is the forward voltage of the diode, and the second is the drop due to the ESR of the batteries.  This leaves no headroom at all for discharge of the batteries. 

Even with a Schottky diode with a slightly less V\$_F\$, like the MBRS410LT3 (0.25V at 2A), that doesn't help very much.

You could try adding add a a large tantalum capacitor in front of the cell module, but that also won't help that much since it will have a similar ESR.

===========================================================

So it looks like the boost IC you chose, TPS61230, is a better option, since you can keep the output voltage of the boost IC constant.  However as you point out, V\$_{IN}\$ minimum is a little high.  The LTC3425 has about identical specifications, but has a V\$_{IN}\$ minimum of 0.5V.

If you want to use AA NiMH batteries, you should be able to use either two or three cells since the fully charged voltage per NiMH cell is 1.2V, and the total will always be below the maximum of the cell module.  Be sure to use an IC specifically designed to charge NiMH.   The LTC4060 can charge from 1 to 4 NiMH calls in series.  

Now you can run the output voltage closer to 4.6v.  I would add a tantalum capacitor (or capacitors) of at least 200 µF and as low ESR as possible to smooth out the transient from the 2A draw.  If you have trouble finding tantalums with a 5V rating and low ESR, you could lower the output of the boost circuit to 3.7V and use one or two 4V capacitors like this one.  Turns out 3.7V is what cell modules are really designed to work at, since that is the nominal voltage of a single-cell Li Ion battery (which can go up to 4.2V or so when fully charged).

===========================================================

Another option would be to go with with 4 alkaline batteries (4*1.65V or 9.90V new, according to a comment to the OP, and 4*1.5V, or 6V nominal), and use a buck-boost regulator that can handle 10V of input like the ISL85403.   That way they could drop to 0.78V apiece (2.5V / 4 + 0.16V) and still operate (where the 0.16V is the drop due to ESR).  (You probably can't really run with battery voltages quite this low, since if I remember correctly the voltage of a alkaline battery drops rapidly once the cells go below about one volt.)

With four cells, you will lose a total 0.64V due to the ESR of each battery, but if you keep the output of the buck-boost at 4.6v, that should not be a problem.  I would still add a tantalum capacitor (or capacitors) of at least 200 µF and as low ESR as possible to smooth out the transient from the 2A draw. Once again if you have trouble finding tantalums with a 5V rating and low ESR, you could lower the output of the boost circuit to 3.7V and use one or two 4V capacitors.