torchfly/text/tokenizer.py
def tokenize(tokens: List[str])->List[np.ndarray]:
        "Input must be list of string"
        final_res = []
        for t in tokens:
            final_res.append(np.array([c for c in t.encode("utf-8")]))
        return final_res
Input must be list of string
Input must be list of string
Input must be list of string
def tokenize(tokens: List[str])->List[np.ndarray]:
        final_res = []
        for t in tokens:
            final_res.append(np.array([c for c in t.encode("utf-8")]))
        return final_res
def tokenize ( tokens : List [ str ] ) -> List [ np . ndarray ] : final_res = [ ] for t in tokens : final_res . append ( np . array ( [ c for c in t . encode ( "utf-8" ) ] ) ) return final_res
Input must be list of string
Input must be list of string
"Input must be list of string"
tokens
List[str]