How deep do neural networks need to be?
The only way to know is to try it and see.  There is no general theory that allows us to reliably predict the outcome of such experiments.

That said, I question the motivation.  You say that it feels "a bit heavy", but it's not entirely clear to me what that concretely means or why it would matter.  Perhaps it is worth mentioning that even with dozens of layers, neural networks can be surprisingly fast.  Remember the warning about premature optimization: don't let a general fear or assumption that the simple solution might be slow tie you up in knots and lead you to overly complex solutions.  Instead, in this case I suggest you try the simplest solution first, and measure how well it performs.  Only if you have evidence that it fails in some concrete, measurable way would you then consider more complex approaches.  This will help you avoid unnecessarily complicating your life.

Finally, remember that there are other approaches to computer vision other than deep neural networks.  People were doing object detection before neural networks became popular, and there are other methods that might be effective for detecting simple geometric shapes (e.g., morphological operators, Hough transform, shape descriptors, and more).  But try the simple things first, and measure.