What happens if I let my building permit expire?
Right off the bat, you won't get your power turned on.  Because the power company won't turn on power until the panel flip is signed off.

Since you have already done the rewire, I would pull a permit for the rewire and then get that signed off on.  I personally think you took your life in your hands by doing the rewire without a permit for that.  

The inspector suggesting additional work does not have the effect of pulling a permit for that work.  

It sounds like the inspector is ever pressuring you to increase the scope of the project.  I agree, this is like one of those lottery/Nigerian Prince scams, everything you do, the inspector will suggest another thing to add to the scope of work.  With the Nierian Prince scam, they tend to bump your next payment by about 2.5 times your last payment, and that sounds like this.  Going from a panel flip to blowing out all your drywall is a huge leap.  

The only way out is to use the words "no", "not funded", "not this year", "outside project scope" and "I don't have a permit for that".