DIY Two Routers on 18650
The nominal voltage of these batteries is 3.6 or 3.7 V, and four of them in series should be enough for a buck converter to provide 12V to the router. The charging circuit needs to be balancing. You can buy cheap balancing BMS charger circuit boards of variable quality on Ebay. If the BMS doesn't have a low voltage protection feature, you'll have to implement that separately.
You manage an increased current draw by paralleling cells. Instead of four cells in series, you can have groups of two, three, four, as many as you like cells paralleled, and then put four such groups in series. The buck converter needs to be scaled up accordingly, naturally.
If both devices pull 3.5A, and you want them to last for 2 hours, you need 3.5A * 2h = 7Ah of capacity. Parallel cells until you reach this number.
Yes, if you use Lithium cells, you need a BMS. Especially if you want your system to charge the cells as well as discharging them.
No. The loads will be in parallel, and each load will only draw as much current as it needs at any given time (which is, by the way, probably lower than the rated current most of the time).
Yes, you will need some sort of BMS if you want to use Lithium cells, especially when you want to use multiple cells in series.


As others have mention in the comments, you will have an easier time using a deep cycle 12V lead acid battery instead of Lithium. The battery management is much easier with this kind of battery. Using Lithium cells is certainly doable, but if weight or space is not a constraint, it might turn out to be unneccesarily complicated.