"Flow layouts" inside a GUI -- how do I come up with a good algorithm?
I've encountered variations of this problem several times doing GUI layout work. Your question about "What should I optimize?" is key to answering your question. Until you've decided that you really can't create a better algorithm.

More than likely you'd like to consider the size-ratios. Each child will have likely a minimum size, possibly a maximum in each dimension, and a preferred ratio, or range of acceptable ratios. You can use this basic data to make a guess as to what the layout should be. Now, for the resulting layout of each child you need a way to produce the preference score.

Now you simply have an optimization problem: maximize the total score of all the children. You can also include global scoring attributes, like the parent size (to cover your second problem).  Some layouts you can also reject if they produce invalid results (parent too wide, child not acceptable).

Depending on how many children you have, and how optimal your layout needs to be, will determine how good your algorithm needs to be. For a very simple greedy approach you can simply do line-based. For example, just consider two lines at a time, the current line, and the next line. You should be able to come up with a list of items that easily have enough space on this line, and some that maybe don't. Then just try a few combinations of those trailing components (this line or next) and find a maximum score.