Richardson was born in Edinburgh, the son of Margaret (née Drummond) and John Richardson.
After National Service in the Army (part of which he spent as an announcer and drama director with the British Forces Broadcasting Service) he obtained a place at the College of Dramatic Arts in Glasgow.