title: How to tokenize a code snippet using a Textmate grammar in Node
body: <p>I'm trying to syntax-highlight code snippets on my library website. I've tried Highlight.js and Prism, but neither of them tokenize the code correctly (it's Ruby), so in the end the code is not syntax-highlighted properly. This is because they both implement their own tokenization regexes, which is an approach that's bound to have flaws.</p>
  
  <p>I've then discovered that <a href="https://github.com/github/linguist/blob/b4a068abf3b9ee66d9413bab5d4dc16ef7c9a71a/vendor/README.md" rel="nofollow noreferrer">GitHub</a>, <a href="https://github.com/atom/language-ruby/blob/2323b3ebdda18e38510c8ffb8f351b9bf2cc11d9/grammars/ruby.cson" rel="nofollow noreferrer">Atom</a> and <a href="https://github.com/microsoft/vscode/blob/7460d86cc642e50953ba3471877b5657c24e79da/extensions/ruby/syntaxes/ruby.tmLanguage.json" rel="nofollow noreferrer">VSCode</a> all use <a href="https://github.com/textmate/ruby.tmbundle/blob/089e0ce05032050e9784a30169e5b3e574c3d8f4/Syntaxes/Ruby.plist" rel="nofollow noreferrer">TextMate grammars</a> for their tokenization. This to me sounds like the right approach, to have language grammars maintained in a single place, so that other tools can then reuse them instead of each defining their own.</p>
  
  <p>My question is: <strong>how to tokenize a code string using a TextMate grammar in Node?</strong> My goal is to have something like:</p>
  
  <pre><code>const codeSnippet = `
  class Foo
    def bar
      puts "baz"
    end
  end
  `
  
  const tokenized = tokenizeCode(codeSnippet, 'ruby')
  
  tokenized // some kind of array of tokens, e.g:
  // [
  //   ['keyword', 'class'],
  //   ['whitespace', ' '],
  //   ['class', 'Foo'],
  //   ...
  // ]
  </code></pre>
  
  <p>I've tried <a href="https://github.com/microsoft/vscode-textmate" rel="nofollow noreferrer">vscode-textmate</a>, which is what VSCode seems to use for its own syntax highlighting. However, I couldn't figure out how to use it to achive the functionality above.</p>
  
  <p>Ultimately I want to end up with HTML that I can syntax-highlight:</p>
  
  <pre><code>&lt;pre&gt;
    &lt;code&gt;
      &lt;span class="token kewyord"&gt;class&lt;/span&gt; &lt;span class="token class"&gt;Foo&lt;/span&gt;
      &lt;!-- ... --&gt;
    &lt;/code&gt;
  &lt;/pre&gt;
  </code></pre>
  
  <p>Again, I've tried highlight.js and Prism, but they both incorrectly tokenize even the simplest Ruby code.</p>
  
  <h2>Edit</h2>
  
  <p>Here are some examples where Prism and Highlight.js incorrectly tokenize Ruby code:</p>
  
  <ul>
  <li><p>Highlight.js – doesn't tokenize <code>Post</code> as a "constant"</p>
  
  <pre><code>const hljs = require("highlight.js/lib/highlight.js");
  hljs.registerLanguage('ruby', require('highlight.js/lib/languages/ruby'));
  
  const rubyCode = `Post.create(params[:post])`
  const html = hljs.highlight('ruby', rubyCode).value
  
  console.log(html)
  // Post.create(params[&lt;span class="hljs-symbol"&gt;:post&lt;/span&gt;])
  </code></pre></li>
  <li><p>Prism – doesn't tokenize <code>foo:</code> as a "symbol"</p>
  
  <pre><code>const Prism = require('prismjs');
  const loadLanguages = require('prismjs/components/');
  loadLanguages(['ruby']);
  
  const rubyCode = `{ foo: "bar" }`
  const html = Prism.highlight(rubyCode, Prism.languages.ruby, 'ruby')
  
  console.log(html)
  // &lt;span class="token punctuation"&gt;{&lt;/span&gt; foo&lt;span class="token punctuation"&gt;:&lt;/span&gt; &lt;span class="token string"&gt;"bar"&lt;/span&gt; &lt;span class="token punctuation"&gt;}&lt;/span&gt;
  </code></pre></li>
  </ul>