1. In a digital medium environment for distractor removal in an image, a method implemented by at least one computing device, the method comprising: forming, by the at least one computing device, a plurality of segments from the image; calculating, by the at least one computing device, a score for each of the plurality of segments that is indicative of a relative likelihood that a respective said segment is considered a distractor within the image, the calculating based on at least one distractor model trained using machine learning; classifying, by the at least one computing device, the plurality of segments as comprising salient objects that are less likely to be considered important to a user and salient objects that are more likely to be considered important to the user based on the calculated scores for the respective segments; outputting, by the at least one computing device, the classifications for the plurality of segments; and performing, by the at least one computing device, a modification to at least one segment of the plurality of segments based on the classifications of the plurality of segments.