 Ruffell was born in Doncaster, but moved south at an early age.He played for Essex Road School, Manor Park, Fullers, Chadwell Heath United, Manor Park Albion, East Ham and Wall End United, before West Ham signed him from the works team of the Ilford Electricity Board in March 1920.He made his League debut against Bury in December 1921.