1. A computing device that is configured to test a virtualized service in a service provider environment, comprising: a processor; a memory in electronic communication with the processor; instructions stored in the memory, the instructions being executable by the processor to: access a pool of service accounts including service accounts for the service provider environment, the service accounts having a plurality of distinct configurations; select a service account from the pool of service accounts having an account configuration; instantiate the virtualized service in the service provider environment based on the account configuration of the service account; identify services enumerated in a testing regime for usage of the virtualized service; initiate a test over a network connection using an application programming interface (API); test the services on the virtualized service in a plurality of regions; display a test status and a test history for the test in the plurality of regions; collect metrics for testing of the services in the plurality of regions; determine absolute and relative performance characteristics of the services across the plurality of regions based on the metrics; compare at least one of the absolute or relative performance characteristics for one of the plurality of regions to the absolute or relative performance characteristic for a different region; and provide a notification when at least one of the absolute or relative performance characteristics is outside a predetermined threshold for the one of the plurality of regions, the predetermined threshold being relative to the absolute or relative performance characteristics for the different region.