training deep leaning models the true positive rate measures the fraction of true positives with respect to the actual positivesand the true negative rate measures the fraction of true negatives with respect to the actual negatives (see figure - the roc curve plots the true positive rate on the axis and the false positive rate on the axis (see figure - the area under the curve (aucis used to summarize the roc curve in many casesstandard metrics like accuracyprecisionrecalletc do not allow us to truly capture model performance for the business use case at hand in such casesmetrics appropriate to the business use case need to be formulatedkeeping in mind the nature of the problemthe class imbalanceand the misclassification costs for instancein our running example of product categorizationwe may choose to not use predictions with low confidence and have them categorized manually there is cost associated with having examples manually categorizedand there is different cost associated with showing wrong products in the wrong category on an ecommerce site the cost of misclassifying popular product is also different (typically higherfrom the cost of misclassifying rarely bought product in such casewe might choose to use only the high confidence predictions from the model possible choice of metrics to use would be the number of examples misclassified (with high confidenceand the coverage (the number of examples covered with high confidenceone may also factor in the misclassification cost in this setting by taking weighted average of the two (appropriate weights may be chosen based on the misclassification costs metric definition is critical step of the model-building process in an industry setting practitioners should deeply analyze the business domainto understand the misclassification costsand the datato understand the class distributionsand design performance metrics accordingly badly defined metric can lead project down an incorrect path