Structured layout in Drupal
First of all, I'm a big Panels fan, so I definitely think you should be using it, but I'm not sure it's the best for the task you are trying to achieve.

Some options, in what I feel is descending order of usefulness to you.


Since it sounds like it's necessary for you to mix several fields into a single render element, you could implement a Ctools "content type". How does one create a new ctools plugin should be helpful here. This easily allows you any custom logic in rendering the fields on a per entity basis. You should still separete the logic from the rendering, by declaring your own template. See Correct way to render a modules output through a template?
If the output you describe above completely dominates the page, then perhaps it would make sense to move the rendering up a level. A panels "layout"-plugin could be useful instead. The plugin link from above is still the source for more information.
If it makes sense to work with individual fields, which I doubt in your case, a custom field formatter could be what you are looking for. See How do I make the output of field X look like Y?
This one is mostly speculcation, I haven't attempted it, but if you are working with a Field collection, it being an entity, perhaps you can use Display suite to control it's output, and then have Panels render it inline. Or you could skip DS and directly template the FC entity.