Do I need to standardize my one hot encoded labels?
NO, you do not standardize labels
The purpose of standardization is to bring features with disparate ranges into a standard range. When the data is not standardized, features with large numerical values will tend to have a larger influence (weight) than those that are smaller numerically.

Consider the Automobile Data Set (https://archive.ics.uci.edu/ml/datasets/automobile)  from UC Irvine. Among the features, it has height, lenght, width, weight, # cylinders, along with a number of other label and numerical features. Height ranges from from 47.8 to 59.8 and weight ranges from 1488 to 4066. You'll want to standardize them, but not label features such as body-style or engine-type.