'Add a token to the format decision state.
Allow the heuristic to try out adding the token with and without a newline.
Later on, the algorithm will determine which one has the lowest penalty.
Arguments:
newline: (bool) Add the token on a new line if True.
dry_run: (bool) Don\'t commit whitespace changes to the FormatToken if
True.
must_split: (bool) A newline was required before this token.
Returns:
The penalty of splitting after the current token.'

def AddTokenToState(self, newline, dry_run, must_split=False):

    penalty = 0
    if newline:
        penalty = self._AddTokenOnNewline(dry_run, must_split)
    else:
        self._AddTokenOnCurrentLine(dry_run)
    return (self.MoveStateToNextToken() + penalty)
