text_extensions_for_pandas/spanner/project.py
def lemmatize(spans: Union[pd.Series, SpanArray, Iterable[Span]],
              token_features: pd.DataFrame,
              lemma_col_name: str = "lemma",
              token_span_col_name: str = "span") -> List[str]:
    """
    Convert spans to their normal form using lemma information in a token
    features table.

    :param spans: Spans to be normalized. Each may represent zero or more
     tokens.
    :param token_features: DataFrame of token metadata. Index must be aligned
     with the token indices in `spans`.
    :param lemma_col_name: Optional custom name for the DataFrame column
     containing the lemmatized form of each token.
    :param token_span_col_name: Optional custom name for the DataFrame column
     containing the span of each token.

    :return: A list containing normalized versions of the tokens
     in `spans`, with each token separated by single space character.
    """
    char_spans = SpanArray.make_array(spans)
    token_spans = TokenSpanArray.align_to_tokens(token_features[token_span_col_name],
                                                 char_spans)
    ret = []  # Type: List[str]
    # TODO: Vectorize this loop
    for i in range(len(token_spans)):
        lemmas = token_features[lemma_col_name][
                 token_spans.begin_token[i]:token_spans.end_token[i]
                 ]
        ret.append(" ".join(lemmas))
    return ret

    Convert spans to their normal form using lemma information in a token
    features table.

    :param spans: Spans to be normalized. Each may represent zero or more
     tokens.
    :param token_features: DataFrame of token metadata. Index must be aligned
     with the token indices in `spans`.
    :param lemma_col_name: Optional custom name for the DataFrame column
     containing the lemmatized form of each token.
    :param token_span_col_name: Optional custom name for the DataFrame column
     containing the span of each token.

    :return: A list containing normalized versions of the tokens
     in `spans`, with each token separated by single space character.
    
Convert spans to their normal form using lemma information in a token
features table.
Convert spans to their normal form using lemma information in a token features table .
def lemmatize(spans: Union[pd.Series, SpanArray, Iterable[Span]],
              token_features: pd.DataFrame,
              lemma_col_name: str = "lemma",
              token_span_col_name: str = "span") -> List[str]:
    char_spans = SpanArray.make_array(spans)
    token_spans = TokenSpanArray.align_to_tokens(token_features[token_span_col_name],
                                                 char_spans)
    ret = []  
    for i in range(len(token_spans)):
        lemmas = token_features[lemma_col_name][
                 token_spans.begin_token[i]:token_spans.end_token[i]
                 ]
        ret.append(" ".join(lemmas))
    return ret
def lemmatize ( spans : Union [ pd . Series , SpanArray , Iterable [ Span ] ] , token_features : pd . DataFrame , lemma_col_name : str = "lemma" , token_span_col_name : str = "span" ) -> List [ str ] : char_spans = SpanArray . make_array ( spans ) token_spans = TokenSpanArray . align_to_tokens ( token_features [ token_span_col_name ] , char_spans ) ret = [ ] for i in range ( len ( token_spans ) ) : lemmas = token_features [ lemma_col_name ] [ token_spans . begin_token [ i ] : token_spans . end_token [ i ] ] ret . append ( " " . join ( lemmas ) ) return ret
Convert spans to their normal form using lemma information in a token
features table.
Convert spans to their normal form using lemma information in a token features table .
"""
    Convert spans to their normal form using lemma information in a token
    features table.

    :param spans: Spans to be normalized. Each may represent zero or more
     tokens.
    :param token_features: DataFrame of token metadata. Index must be aligned
     with the token indices in `spans`.
    :param lemma_col_name: Optional custom name for the DataFrame column
     containing the lemmatized form of each token.
    :param token_span_col_name: Optional custom name for the DataFrame column
     containing the span of each token.

    :return: A list containing normalized versions of the tokens
     in `spans`, with each token separated by single space character.
    """ # Type: List[str] # TODO: Vectorize this loop
A list containing normalized versions of the tokens
in `spans`, with each token separated by single space character.
A list containing normalized versions of the tokens in ` spans ` with each token separated by single space character .
spans
Union[pd.Series, SpanArray, Iterable[Span]]
Spans to be normalized. Each may represent zero or more
tokens.
Spans to be normalized . Each may represent zero or more tokens .
token_features
pd.DataFrame
DataFrame of token metadata. Index must be aligned
with the token indices in `spans`.
DataFrame of token metadata . Index must be aligned with the token indices in ` spans ` .
lemma_col_name
str
Optional custom name for the DataFrame column
containing the lemmatized form of each token.
Optional custom name for the DataFrame column containing the lemmatized form of each token .
token_span_col_name
str
Optional custom name for the DataFrame column
containing the span of each token.
Optional custom name for the DataFrame column containing the span of each token .