How can I make my home internet router work with a battery?
Get a UPS
No, not that UPS. An uninterruptible power supply, also known as a battery backup. This is a box that takes normal AC (e.g., 110-120V in the US) and uses it to charge a battery and power your devices. If power goes out (technically: voltage out of range) then it uses an inverter to power your devices from the battery. All automatically.
You typically want to connect:

Cable or DSL or FIOS modem
Router
Computer if you are using a desktop computer
Any other small but important devices - e.g., if you have an Ethernet switch then connect it so if you have a laptop or other device on that switch (instead of WiFi) it will continue to have a wired internet connection

Sizes and prices vary, primarily based on the size of the battery which, combined with the size of the load, determines how long it will run when AC is out.