how to avoid tokenizing w/ sklearn feature extraction I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be split into 'Part', 'C1', '11' and 'Reading# and 'Status'.

I've got the vague feeling that the token_pattern is the parameter I need to adjust so I tried to specify the beginning and the end of a string like so:

from sklearn.feature_extraction.text import CountVectorizer
cvo = CountVectorizer(token_pattern='^$',lowercase=False)
OriginCV = cvo.fit_transform(log['Message_Origin']).toarray()


However, the last line throws an error:
ValueError: empty vocabulary; perhaps the documents only contain stop words

I've also tried to explictly include dot and space in the token_pattern like so:

cvo = CountVectorizer(lowercase=False, token_pattern=r"(?u)\b\w\w+\b|\.|\s")


Throws no errors but does not do the trick (no change except for an additional token '.')

Not changing the default token_pattern does split the string at the spaces and colons though. I found this solution, which however modifies the string by removing e.g. the colons.
Any other idea how to solve this?

Thanks,
Mark
You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split

tokenizer=str.split


From the official documentation,


  tokenizer callable, default=None
  
  Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if analyzer == 'word'.