How to extract character ngram from sentences? - python
Question: The following `word2ngrams` function extracts character 3grams from a word:

    
    
    >>> x = 'foobar'
    >>> n = 3
    >>> [x[i:i+n] for i in range(len(x)-n+1)]
    ['foo', 'oob', 'oba', 'bar']
    

This post shows the character ngrams extraction for a single word, [Quick
implementation of character n-grams using
python](http://stackoverflow.com/questions/18658106/quick-implementation-of-
character-n-grams-using-python).

But what if i have sentences and i want to extract the character ngrams, **is
there a faster method other than iteratively call the`word2ngram()`**?

**What will be the regex version of achieving the same`word2ngram` and
`sent2ngram` output? would it be faster?**

I've tried:

    
    
    import string, random, time
    from itertools import chain
    
    def word2ngrams(text, n=3):
      """ Convert word into character ngrams. """
      return [text[i:i+n] for i in range(len(text)-n+1)]
    
    def sent2ngrams(text, n=3):
        return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))
    
    def sent2ngrams_simple(text, n=3):
        text = text.lower()
        return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]
    
    # Generate 10000 random strings of length 100.
    sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]
    
    start = time.time()
    x = [sent2ngrams(i) for i in sents]
    print time.time() - start        
    
    start = time.time()
    y = [sent2ngrams_simple(i) for i in sents]
    print time.time() - start        
    
    print x==y
    

[out]:

    
    
    0.0205280780792
    0.0271739959717
    True
    

**EDITED**

The regex method looks elegant but it performs slower than iteratively calling
`word2ngram()`:

    
    
    import string, random, time, re
    from itertools import chain
    
    def word2ngrams(text, n=3):
      """ Convert word into character ngrams. """
      return [text[i:i+n] for i in range(len(text)-n+1)]
    
    def sent2ngrams(text, n=3):
        return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))
    
    def sent2ngrams_simple(text, n=3):
        text = text.lower()
        return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]
    
    def sent2ngrams_regex(text, n=3):
        rgx = '(?=('+'\S'*n+'))'
        return re.findall(rgx,text)
    
    # Generate 10000 random strings of length 100.
    sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]
    
    start = time.time()
    x = [sent2ngrams(i) for i in sents]
    print time.time() - start        
    
    start = time.time()
    y = [sent2ngrams_simple(i) for i in sents]
    print time.time() - start        
    
    start = time.time()
    z = [sent2ngrams_regex(i) for i in sents]
    print time.time() - start  
    
    print x==y==z
    

[out]:

    
    
    0.0211708545685
    0.0284190177917
    0.0303599834442
    True
    


Answer: Why not just `(?=(...))`

**edit** Same thing, but not whitespace `(?=(\S\S\S))`  
**edit2** You can use just what you want as well. Ex. uses alphanum only
`(?=([^\W_]{3}))`

Uses a lookahead to capture 3 characters. Then the engine bumps the position
up 1 time each  
match. Then captures next 3.

Result of `foobar` is  
foo  
oob  
oba  
bar

    
    
     # Compressed regex
     #  (?=(...))
    
     # Expanded regex
     (?=                   # Start Lookahead assertion
          (                     # Capture group 1 start
               .                     # dot - metachar, matches any character except newline
               .                     # dot - metachar
               .                     # dot - metachar
          )                     # Capture group 1 end
     )                     # End Lookahead assertion
    

