1. An apparatus, comprising: one or more processors; and one or more non-transitory computer-readable storage media having instructions stored thereupon which are executable by the one or more processors and which, when executed, cause the apparatus to: determine a number of out-of-vocabulary words in input text segments; generate an estimated difficulty feature score of a supervised machine learning model in translating the input text segments based, at least in part, on the number of out-of-vocabulary words; modify a misclassification cost associated with the supervised machine learning model, stored in a memory, to offset an imbalance between a plurality of classes of training data utilized to train a machine translation quality classifier to classify a quality of machine translated text segments, the training data comprising one or more feature scores including the estimated difficulty feature score; modify a loss function associated with the supervised machine learning model stored in the memory to penalize a misclassification of a lower-quality text segment as a higher-quality text segment more greatly than a misclassification of a higher-quality text segment as a lower-quality text segment; train the machine translation quality classifier utilizing the supervised machine learning model based, at least in part on the misclassification cost and the loss function; cause the machine translation quality classifier to be deployed to a computer in a service provider network; and utilize the machine translation quality classifier is utilized to classify a quality of translated segments received from a machine translation system operating in the service provider network into one of the plurality of classes in real time.