Determining the correct chain length (52/36 with 12/29 11S cassette combination)
From the picture, it's clear you have a "short cage" rear derailleur. That could be a big contributor to the problem.

Rear derailleurs are rated for a certain maximum "capacity" that they can handle, which is the difference between the smallest total number of teeth and largest total. In your case, your system goes from 12+36=48 teeth to 29+52=81. This requires a capacity of 33T. Suppose your chain is almost snug when shifted into the 29 + 52 combination, so that the derailleur's tension pulley is almost horizontal, and then you shift into 12 + 36, the derailleur's tension pulley has to then pick up 33 teeth of slack.

A 33T range is about right at the edge of the capacity of a typical short cage derailleur, which is probably why you're having trouble setting things up to be okay at both extremes. It would be more comfortable with a middle cage length unit.