Maximum Stacking Height Problem
While I don't know of any specific algorithms for this problem, you could approach this in a pretty efficient method by breaking it down into separate parts. 

I would start by finding the rotation for each individual shape that provides a maximum height while maintaining a valid balancing orientation (is: not on a point like with the triangle).  If a shape has multiple equal heights, i would go with the configuration that gives the greatest surface area on top of it. Once you have this, you can then figure out how to best stack each object in a manor that is able to be balanced.