Modeling on round surfaces without deform
Your best bet (depending on your workflow) is to work on the cylinder's round surface 'unwrapped' then wrap it around with a curve and modifier after doing any changes to the geometery.

Looking at the model you've created, you might also have luck with boolean modelling. The concepts of boolean modelling are using many modifiers - boolean, bevel, subsurf. To get this look, create the cylinder, apply a bevel to create the top and bottom edge bevels, apply a subsurf to smooth the surface, then use a boolean to cut out the rectangular piece. Once that is done, you could apply another bevel (limiting the influence based on angle) to create a bevel on the cut-out edges.

By doing this, you retain editability of every step and can go in and make changes as required.