They started in 1863 as `` Stoke Ramblers '' which makes them the second oldest Football League club in the world . Founded in 1863 , Stoke are the oldest club in the Premier League , and are considered to be the second oldest professional football club in the world , after Notts County .