1. A method comprising: receiving speech from a user as part of a speech dialog between the user and a speech recognition service; identifying, based on the speech, a speech pattern of the user; identifying, based on the speech pattern of the user, a plurality of speech recognition models stored in a cloud computing storage environment, each speech recognition model of the plurality of speech recognition models being from a respective speech recognition domain; combining, via a processor, the plurality of speech recognition models, to yield a multi-domain combined speech recognition model; identifying in the speech a specific speech recognition domain, wherein the specific speech recognition domain does not match a specific speech recognition model in the plurality of speech recognition models; receiving sample data associated with the specific speech recognition domain, wherein the sample data is independent of the speech dialog between the user and the speech recognition service; when the sample data is more than a minimum threshold, generating a new domain-specific speech recognition model for the specific speech recognition domain; and when the sample data is less than the minimum threshold, modifying the multi-domain combined speech recognition model specifically to the specific speech recognition domain by weighting components of the multi-domain combined speech recognition model associated with the specific speech recognition domain to have more influence in recognition of the speech from the user.