Calculating the inner diameter of a sleeve for wrapping a bundle cables in a house
I would not attempt bundling the cables before installing them in the sleeves / conduits / raceways, it will be counterproductive.  

Electrical codes have guidelines for conduit fill.  For example in the US, the National Electrical Code has a base rule for power wiring that you can fill 40% of the cross sectional area of a conduit or raceway.  (I am oversimplifying, there's more to these rules, but not important here.)  

The 40% guideline doesn't really apply as a rule to a protective sleeve for ethernet cables, but it is still a good guideline for you to use.  You can cheat a little, especially if the conduit is more or less straight.  It can get hard to pull more cable than this through conduit, and you should not tug very hard on ethernet cables.  

Incidentally, be aware that there are limits on how many bends you can have in a conduit / raceway because as you accumulate bends it gets very hard to pull cable through.  Do not cheat on this part - in fact try to have less bends than the rules permit.  The NEC allows 360 degrees of total bend (for example, four 90 degree bends, eight 45 degree bends, etc.) but for communications, it's better to limit it to 270 degrees total bend.  

You are using wire gauges for your cables but those are irrelevant, what matters is the outer diameter of the cable jackets.  

The cross sectional fill is easy enough to calculate, but you can find lots of conduit fill charts and calculators online that will do the math for you.  The code / algorithm snippets in the question are the kind of logic embedded in those calculators.