How to power a device with a LIPO battery correctly
I have a little device which is normally powered by 5V 2A. But the manufacturer says it is compatible up to 16V 2A and I tested it with a powerbank which outputs 12V 2A.
Will I be fine with simply connecting the battery to the device and it will self regulate the Ampere?

The maximum output voltage of a LiPo is 4.2V/cell or 12.6V for 3 cells in series when fully charged.  IF your device really will work safely on 13V it will be OK using a 3S LiPo.
You could use 2 x LiPo = 6V when fully discharged and 8.4V when fully charged.
IF the device draws 2A at 5V it will probably draw somewhat less at higher voltages as it probably has an internal switching regulator.
I at voltage V is probably ~~= 5/V x I_at_5V x k
Where k is probably in the range 1.1 - 1.2