12. The system of claim 10 , wherein the operations further comprise: determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has a higher priority than the other viewable element, wherein the action is initiated based on determining that the particular viewable element has a higher priority than the other viewable element.
10. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: obtaining (i) a view hierarchy that is of a user interface and that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of a particular viewable element in the user interface, and (ii) a transcription of a speech input that was spoken while the user interface was displayed; determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and in response to determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken, initiating an action that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken.