1. A non-transitory computer program product for detecting tie plate anomalies in a tie plate image, the computer program product comprising: a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to: localize a tie plate region in an image from a set of tie plate images; divide the tie plate region into a set of local units; extract features from each local unit in the set of local units; perform a clustering over all local units; assign a class label to each local unit, the class label indicates a semantic content of the local unit; determine the weight of each local unit across all images in the set of images; and categorize the tie plate image as anomalous based on the weight assigned to the local units.