title: Simple tokenizer for C++ in Python
body: <p>Struggling to find a Python library of script to tokenize (find specific tokens like function definition names, variable names, keywords etc.). </p>
  
  <p>I have managed to find keywords, whitespaces etc. using something like <a href="https://docs.python.org/3/library/re.html#writing-a-tokenizer" rel="nofollow">this</a> but I found it quite a challenge for <strong>function/class definition names etc</strong>. I was hoping of using a pre-existent script; I explored <a href="http://pygments.org/docs/lexers/#lexers-for-c-c-languages" rel="nofollow">Pygments</a> with no success. Its <a href="https://bitbucket.org/birkenfeld/pygments-main/src/ef81d4814bc6b6894cc054b3b66bb2dcbf79add9/pygments/lexers/c_cpp.py?at=default&amp;fileviewer=file-view-default" rel="nofollow">lexer</a> seems amazing for what I want but have no idea how to utilize it in Python and to also get <strong>positions</strong> for each found token.</p>
  
  <p>For example I am looking at doing something like that:</p>
  
  <pre><code>int fac(int n)
  {
      return (n&gt;1) ? n∗fac(n−1) : 1;
  }
  </code></pre>
  
  <p>from the source code above I would like to get:</p>
  
  <blockquote>
    <p>function_name: 'fac' at position (x, y)
    variable_name: 'n' at position (x, y+8)</p>
  </blockquote>
  
  <p>EDITED:
  Any suggestions will be appreciated since I am in the dark here regarding tokenizations and parsing in C++?</p>