Can Expansion Combining "Break" the Game?
In my opinion you can combine any of the expansions (except possible the Catapult) together to make a decent new game.  In fact we often combine different parts of different expansions if we want to avoid having too many rules.  We might play with the inns, pigs and mayors for instance.

Also we sometimes want a longer game but without too many extra rules.  For example, by playing with Traders and Builders we get a lot of extra tile shapes, which can be very satisfying if you like to fill in holes, but we might agree before the game that we won't use the resources associated with those tiles.

One expansion which we never use alone is the Princess and Dragon.  We like to play with the dragon on a large board, as this helps to encourage more long-term strategies.  For example on a small board (with the dragon) you are unlikely to risk making a large city, or placing a farmer early.  If the board is bigger then the dragon is less likely to come near your meeples, so there is a bit more margin for prospective play.

In your example it sounds like you are using a sensible number of rules.  If you're not a fan of small cities then you could maybe try playing without the siege tiles?  However at this stage it becomes a question of personal taste.