If all passengers moved to the back, front or one side, would they crash the plane?
The issue at hand is known as weight and balance. Most of the larger airliners do not have much of a problem with a large shift in Passenger weight but shifting cargo has indeed caused accidents (See https://en.wikipedia.org/wiki/National_Airlines_Flight_102) which was what happened when a large amount of cargo, not properly secured, fell towards the back of the aircraft during takeoff causing the B747 to crash.

In general, the smaller the aircraft, the more sensitive it is to Moment and Arm changes in its weight and balance. If there is too much weight towards the rear (known as Aft-CG) then the pilot may not have enough down-elevator authority to keep the aircraft from pitching up even further.

A forward-CG problem may also happen where there is too much weight towards the front of the aircraft which may cause the pilot to run out of UP elevator authority during the landing flare, causing a nose-first landing.

Both fore and aft Center-of-gravity issues have caused airline crashes in the past although usually not from passengers but rather shifting cargo not properly secured.

I am not aware of any sideways loading issues that could not be corrected with aileron (assuming both wings are attached). Flying with more fuel in one wing than the other can be tiresome but is usually safe.