'Requires:
line -- line to parse
Returns:
nothing
Logic:
This function collects all the token information and adds it to
the text string until the end of the field is found.
First check of the end of the information field. If found, write
the text string to the file.
If not found, get the relevant information from the text string.
This information cannot be directly added to the text string,
because it exists in abbreviated form.  (num-of-wor)
I want to check this information in a dictionary to convert it
to a longer, readable form. If the key does not exist in the
dictionary, print out an error message. Otherise add the value
to the text string.
(num-of-wor => number-of-words)'

def __collect_tokens_func(self, line):

    if (self.__token_info == 'mi<mk<docinf-end'):
        self.__state = 'in_info_table'
        self.__write_obj.write(('%s\n' % self.__text_string))
        self.__text_string = ''
    else:
        att = line[6:16]
        value = line[20:(-1)]
        att_changed = self.__token_dict.get(att)
        if (att_changed is None):
            if (self.__run_level > 3):
                msg = ('No    dictionary    match    for    %s\n' % att)
                raise self.__bug_handler, msg
        else:
            self.__text_string += ('<%s>%s' % (att_changed, value))
