1. A system, comprising: one or more processors; one or more non-transitory computer-readable storage mediums including instructions configured to cause the one or more processors to perform operations including: generating a filtering rule for characterizing activity data into two groups; accessing a database including activity data to obtain training data, wherein the training data includes past activity data representing unauthorized activity; partitioning the training data using the filtering rule, wherein partitioning includes filtering different portions of the past activity data into two partitions corresponding to the two groups; analyzing the partitions, wherein analyzing includes providing the partitions to a model, and wherein the model generates training classifications for the past activity data; identifying an accurate set of classifications for the training data, wherein the accurate set of classifications provides known classifications for the past activity data; comparing the known classifications for the past activity data with the training classifications for the past activity data to evaluate performance of the model, wherein the model performs better for a first of the two partitions and worse for a second of the two partitions; assigning a first of the two groups as a retained group and a second of the two groups as a rejected group based on the performance of the model, wherein the first of the two groups corresponds to the first of the two partitions; accessing a first database to obtain authorized activity data that represents activity involving a first service provided to a user, wherein the first service is provided by a service provider; accessing a second database to obtain new activity data that represents activity involving a second service provided to the user, wherein the second service is provided by the service provider, and wherein the second service is different from the first service; generating an initial classification for the new activity data, wherein the initial classification identifies the new activity data as including authorized activity and potentially unauthorized activity data; filtering the potentially unauthorized activity data using the filtering rule to identify retained activity data, wherein the filtering rule characterizes at least a portion of the potentially unauthorized activity data within the retained group; filtering the authorized activity data involving the first service to identify supplemental activity data for use in classifying the retained activity data, wherein the supplemental activity data corresponds to least a portion of the authorized activity data involving the first service that is filtered into the retained group using the filtering rule; analyzing the retained activity data and the supplemental activity data to determine an updated classification for the retained activity data, wherein analyzing includes classifying the retained activity data as authorized activity or classifying the retained activity data as unauthorized activity; deactivating an unnecessary security measure for the authorized activity classified from the retained activity data selecting a new security measure for the unauthorized activity classified from the retained activity data; and activating the new security measure for the authorized activity classified from the retained activity data.