New bottom bracket, asymmetric cranks
Put the spacers back in and see how it feels. If it feels better and your gears a shifting well enough for you, leave them in.

As @Daniel says, chainline is not as important as some believe. In a perfect world, you get the chain line right and never, never, ever, cross the chain. But just like the movie, sometimes you just have to cross the chain, and hell does not freeze over. 

New MTB's are going to 1x11 (and 1x12), and the cassettes are wider than 9' or 10's. If chain line was so critical, these 1x11 would never work. 

If your chainline is badly out, you may have problems getting shifting to work as smoothly as it should, your chain may wear a little quicker than it would, and the little cogs on the derailleur may wear out a little sooner. Notice the word may, and its a a very small if it does, and very debatable.....