How can you determine what set of boxes will maximize nesting? I'm trying to find a dynamic solution to the nesting boxes problem.

You're basically given a set of "boxes" which all have different dimensions. The goal is to find the maximum set of boxes that can be nested inside of each other.

So more formally,

Given set B = {b_1, b_2, . . . b_n} where each b_i contains a box with a width, height and depth.

Find the set of boxes that will allow you to nest as many as possible together.
Let $\le_B$ denote the inclusion relation between boxes, i.e. $b_i\le_B b_j$ if we can put the box $b_i$ inside $b_j$. Obviously $\le_B$ is a partial ordering of $B=\left\{b_1,...,b_n\right\}$. 

Your question translates to finding the maximal linearly ordered subset of $B$ relative to $\le_B$. A linearly ordered subset is a path in the acyclic directed graph $G$ representing $\le_B$, i.e. $V_G=B$ and $(b_i,b_j)\in E_G\iff b_i\le_B b_j$. We can now reduce your problem into finding the longest path in the DAG $G$. This can be done in linear time with topological sorting. Overall, the time consuming part is constructing $G$, which requires $O(n^2)$ time.