Choosing Parent Chain, Complexity of Side Chains
You have already reduced the problem to two possible parent chains:
A: 5-(2-methylpropyl)-6-(3-methyloctan-4-yl)dodecane
or
B: 5-(butan-2-yl)-6-(2-methyloctan-4-yl)dodecane

In order to identify the preferred name, we have to read further in the current version of Nomenclature of Organic Chemistry – IUPAC Recommendations and Preferred Names 2013 (Blue Book). In particular, the following potentially relevant rules are actually not helpful in this case:


P-45.2.1 maximum number of substituents cited as prefixes
P-45.2.2 the lower locant set for substituent groups cited as prefixes
P-45.2.3 the lower locant set for substituent groups in order of citation


However, we finally find:


  P-45.5.1 The preferred IUPAC name is the name that is earlier in alphanumerical order (see P-14.5). Alphabetic letters are considered first in the order that they appear in the name; all Roman letters are considered before any italic letters, unless the latter are used as locants or are a part of a compound or composite locant, for example, ‘N’ and ‘4a’. Then, if still there is a choice, numerical locants are considered in the order of their appearance in the name.


Therefore, the preferred name is 5-(butan-2-yl)-6-(2-methyloctan-4-yl)dodecane since ‘butan-yl’ is earlier alphabetically than ‘methylpropyl’.