The law says it is illegal to drive a vehicle in an unsafe manner, and while both the proverbial jury remains out as to whether wearing a pair of jandals or being barefoot could constitute "unsafe", if the police suspect it contributed to any resulting incident, then you could well be charged.