19. The system of claim 16 , wherein accessing the tagged items that are used by the language understanding model comprises accessing different tagged utterances that are used in training the language understanding model.
16. A system for using human perception in a language understanding model, comprising: a processor and memory; an operating environment executing using the processor; a display; and a perception manager that is configured to perform actions comprising: accessing a language understanding corpus comprising tagged items that are used by the language understanding model, wherein at least one of the tagged items is selected from the group consisting of tagged slots, tagged domains, and tagged intents; displaying a human perception rating panel on the display that includes a display of: a current query; a current result set; and a human perception rating user interface element that is configured to receive a human perception rating for the current result set; creating a first query using the tagged items; obtaining, from a knowledge source in communication with the computing device, a first result for the first query; obtaining, by the computing device, a first human perception rating for the first result using the human perception rating panel; after obtaining the first human perception rating, automatically modifying at least one of the tagged items in the first query, wherein the modifying includes changing the value of the at least one of the tagged items; in response to the modification, creating a second query using the at least one of the modified tagged items; obtaining, from the knowledge source, a second result for the second query; obtaining a human perception rating for the second result using the human perception rating panel; aggregating human perception ratings obtained from a plurality of judges; determining one or more values corresponding to a relative importance of each of the tagged items using the human perception ratings; and providing the one or more values as input to a language understanding model to train the language understanding model to minimize perceptual error as determined from the human perception ratings.