Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?
Have a look at this stackoverflow answer, it seems to be caused by that the fact that your labels need to be zero indexed as the argmax function also return the index based on a zero indexed array.