Are passengers really not allowed to change seats before take-off due to weight and balance?
The crew gave you the correct reason, but it would had taken a co-ordinated effort from the majority of the passengers to upset the aircraft's balance.

But in smaller aircraft even a single person moving around can shift the center of gravity outside its safe limits. The de Havilland company once produced such an aircraft, the DH86 Express. Wikipedia reports:


  On 2 October 1935 Holyman's VH-URT Loina was also lost in Bass Strait,
  again with no survivors. This time a significant amount of wreckage
  was recovered from the sea and from beaches on Flinders Island.
  Investigation of the wreckage revealed a section of charred carpet on
  a piece of cabin flooring from just ahead of the lavatory door. It was
  thought possible that a small fire from a dropped cigarette had led to
  someone running aft suddenly to stamp it out – the sort of sudden
  change in weight distribution that could set up a fatal loss of
  directional control while the aircraft was on a low-speed landing
  approach.


A dramatic example what a shifting load can do to the controllability of an aircraft was National Airlines Flight 102, a Boeing 747-400 freighter conversion which took off from Bagram on April 29, 2013. The most likely cause was insufficiently secured payload which shifted back when the aircraft pitched up during take-off. A single passenger will not be as heavy as a MRAP, but several of them moving over a significant part of the cabin length in combination with a position of the center of gravity close to the limit could well upset the flight.


CG at the forward limit, passengers moving to the front: The aircraft will need higher speed and much more elevator deflection for lift-off. Especially in combination with an engine failure late in the take-off run, a crash is a distinct possibility.
CG at the rear limit, passengers moving to the back: The static stability and, in manually controlled aircraft, the stick forces are reduced. Flying the aircraft needs more attention, and in extreme cases the aircraft will become longitudinally unstable.


In any case, the pilots need to adjust trim.