17. A non-transitory computer-readable storage media having instructions stored thereupon which are executable by one or more processors and which, when executed, cause the one or more processors to: generate an estimated difficulty feature score of a supervised machine learning model in translating input text segments based, at least in part, on a number of out-of-vocabulary words in the input text segments; train a machine translation quality classifier stored in a memory to classify a quality of the translated segments utilizing the supervised machine learning model configured with a misclassification cost configured to offset an imbalance between a plurality of classes of training data, the training data comprising feature scores associated with machine translated segments in a target language, the feature scores including the estimated difficulty feature score; and a loss function configured to penalize a misclassification of a lower-quality translated segment as a higher-quality translated segment more greatly than a misclassification of a higher-quality translated segment as a lower-quality translated segment; and utilize the machine translation quality classifier at a computer operating in a service provider network to classify the quality of the translated segments generated by the machine translation system into the plurality of classes.
17. A non-transitory computer-readable storage media having instructions stored thereupon which are executable by one or more processors and which, when executed, cause the one or more processors to: generate an estimated difficulty feature score of a supervised machine learning model in translating input text segments based, at least in part, on a number of out-of-vocabulary words in the input text segments; train a machine translation quality classifier stored in a memory to classify a quality of the translated segments utilizing the supervised machine learning model configured with a misclassification cost configured to offset an imbalance between a plurality of classes of training data, the training data comprising feature scores associated with machine translated segments in a target language, the feature scores including the estimated difficulty feature score; and a loss function configured to penalize a misclassification of a lower-quality translated segment as a higher-quality translated segment more greatly than a misclassification of a higher-quality translated segment as a lower-quality translated segment; and utilize the machine translation quality classifier at a computer operating in a service provider network to classify the quality of the translated segments generated by the machine translation system into the plurality of classes. 21. The non-transitory computer-readable storage media of claim 17 , having further instructions stored thereupon to retrain the machine translation quality classifier in conjunction with the retraining of a statistical machine translation system.