How can you determine what set of boxes will maximize nesting?
Let $\le_B$ denote the inclusion relation between boxes, i.e. $b_i\le_B b_j$ if we can put the box $b_i$ inside $b_j$. Obviously $\le_B$ is a partial ordering of $B=\left\{b_1,...,b_n\right\}$. 

Your question translates to finding the maximal linearly ordered subset of $B$ relative to $\le_B$. A linearly ordered subset is a path in the acyclic directed graph $G$ representing $\le_B$, i.e. $V_G=B$ and $(b_i,b_j)\in E_G\iff b_i\le_B b_j$. We can now reduce your problem into finding the longest path in the DAG $G$. This can be done in linear time with topological sorting. Overall, the time consuming part is constructing $G$, which requires $O(n^2)$ time.