My revised paper was accepted surprisingly quickly; should I worry?
Unless you have some reason to believe your revised paper has some serious additional problems, I don't think you should worry.  It may not be best practices for referees and editors to operate this way (just looking to see whether the changes requested in the first report have been adequately completed).  Ideally, the referee should check over carefully that none of the changes have introduced new problems, but that isn't always done.

The referee and editor may have felt that the revisions were not really that big a deal.  You may have been overthinking things when you made the changes; they might have intended some considerably less sweeping editing.  Or the referee may have seriously underestimated how much additional work and explication would be required to fix the problems the referee pointed out.  The referee might even have told the editors that another round of refereeing would not actually be necessary; if you made the changes, then the paper would be ready for publication.  (For several of the journals I have refereed for, this is one of the options in the recommendation menu of the reviewing software; however, I also know, as an author, that this part of the review is not shared with the authors.)