Powering up Arduino + GSM with LiPo rider with a LiPo battery
The key issue is that your GSM shield has its own on-board regulator (the huge 5-pin IC in the top right-hand corner of your photo) and requires VIN > 5V to provide a properly regulated 4.1V to the GSM module.  Additionally, the Arduino itself requires Vin > 6V for proper operation.

An easy way to meet the above two requirements is to use two LiPo cells in series to supply 7.2V to the system's PWRIN ("9V") jack, as shown below:



I'm not sure if your version of LiPoRider can be used to charge two cells in series. If not, find a charge control board that does.

You could also bust a few moves to improve battery life of your Arduino board.

As a side note, a GSM module that was switched off takes around 30s to connect to a base station - don't forget that in your battery life estimates.

Safety note: If you circumvent the PWRIN jack by connecting your LiPos directly to VIN and then plug an external power supply into to the PWRIN jack, you could start a LiPo fire.

Therefore, be safe and connect the LiPos to the Power jack. (or use VIN but put epoxy in the power jack/unsolder D1 to make the jack unusable) Connecting the LiPos to the PWRIN jack additionally gives you reverse battery protection due to diode D1 (see Arduino snippet below), but on the down side D1 drops a whole 1V while the GSM module transmits.


Arduino Power Supply Schematic