1. A speech recognition apparatus comprising: a generating unit configured to generate sequences of speech features from characteristics of entered speech for respective frames having an arbitrary temporal width; a model storage unit having a plurality of phrases expressed on basis of grammar and one or more continuous phrase segments obtained by dividing the respective phrase, the model storage unit configured to store state transition models which express time series changes of the speech features for respective phrase segments as state-to-state transition relating to the speech features; a first grammar storage unit configured to store grammar segments relating to one or more continuous phrase segments which belong to each of the phrases; a second grammar storage unit configured to store at least part of the grammar segments transferred from the first grammar storage unit and to be able to read out information stored therein in a reading time shorter than that required for the first grammar storage unit; a first decoder configured to obtain forward probabilities of respective states of the state transition models for the sequence of speech features generated by the generating unit with respect to each of the frames, by referring to the grammar segments stored in the second grammar storage unit and the state transition models stored in the model storage unit; a grammar transfer unit configured to transfer a trailing grammar segment relating to a trailing phrase segment which trails one of said continuous phrase segments, from the first grammar storage unit to the second grammar storage unit when the forward probability of final state among said states of the state transition models is obtained by the first decoder; a second decoder configured to obtain the forward probabilities of the respective states of the state transition models for a sequence of trailing speech features as the sequence of speech features for the trailing segment as generated by the generating unit with respect to each of the frames, continuously after the speech feature sequences, by referring to the grammar segments stored in the second grammar storage unit and the state transition models stored in the model storage unit; a third decoder configured to obtain the forward probabilities of the respective states of the state transition models for the trailing speech feature sequences for the respective frames, by referring to the trailing grammar segment transferred to the second grammar storage unit and the state transition models stored in the model storage unit; a recognition control unit configured to (1) carry out recognition for the respective phrases, (2) activate the first decoder until the transfer of the trailing grammar segment is started, (3) activate the second decoder in parallel to the transfer from the start to the completion of the transfer, (4) activate the third decoder upon completion of the transfer, and (5) repeat the operations from (2) to (4) until all the operations for the phrase segments belonging to the respective phrases to obtain final forward probabilities for the respective phrases; and a recognizing unit configured to output the phrase which give the highest forward probability from among the respective final forward probabilities of the plurality of phrases as a result of recognition of the speech feature sequence.