Casey was born in the Georgetown section of Wilkes Barre, Pennsylvania in 1875 - the oldest son of an Irish immigrant family.
In 1883 at the age of eight, Casey was a breaker boy in Plymouth, Pennsylvania, responsible for breaking the slag off of coal fragments as they were excavated from the mine.