Is it OK that the frame of my house doesn't line up with the slab?
No, it's not ok. For one thing, by the time the city inspector looks at the fully framed building, it will be a little too late to fix it. Secondly, the builder's attitude seems very questionable. Either his framing crew or the concrete sub messed up. At the very least their job is not done in a workmanlike manner. They shouldn't wait and hope things 'slip' by the city inspector. A good contractor with personal integrity just wouldn't allow something like this to happen in the first place, IMHO. Or if this was truly not a big deal, he would take time to explain why this is ok.