title: most efficient way to parse this scripting language
body: <p>I'm implementing an interpreter for a long-outdated text editor's scripting language, and I'm having some trouble getting a lexer to work properly.</p>
  
  <p>Here's an example of the problematic part of the language:</p>
  
  <pre><code>T
  L /LOCATE ME/
  C /LOCATE ME/CHANGED ME/ * *
  C ;CHANGED ME;CHANGED ME AGAIN; 1 *
  </code></pre>
  
  <p>The <code>/</code> characters seem to quote strings and also act as a delimiter for the <code>C</code> (<code>CHANGE</code>) command in a <code>sed</code>-type syntax, although it allows any character as a delimiter.</p>
  
  <p>I've probably implemented about half the most common commands, just using <code>parse_tokens(line.split())</code> until now. That was quick and dirty, but it worked surprisingly well.</p>
  
  <p>To avoid writing my own lexer, I tried <code>shlex</code>.</p>
  
  <p>It works pretty well, except for the <code>CHANGE</code> cases:</p>
  
  <pre><code>import shlex
  
  def shlex_test(cmd_str):
      lex = shlex.shlex(cmd_str)
      lex.quotes = '/'
      return list(lex)
  
  print(shlex_test('L /spaced string/'))
  # OK! gives: ['L', '/spaced string/']
  
  print(shlex_test('C /spaced string/another string/ * *'))
  # gives   : ['C', '/spaced string/', 'another', 'string/', '*', '*']
  # desired : any format that doesn't split on a space between /'s
  
  print(shlex_test('C ;a b;b a;'))
  # gives   : ['C', ';', 'b', 'a', ';', 'a', 'b', ';']
  # desired : same format as CHANGE command above
  </code></pre>
  
  <p>Anyone know an easy way to accomplish this (with <code>shlex</code> or otherwise)?</p>
  
  <p>EDIT:</p>
  
  <p>If it helps, here's the <code>CHANGE</code> command syntax given in the help file:</p>
  
  <pre><code>'''
  C [/stg1/stg2/ [n|n m]]
  
      The CHANGE command replaces the m-th occurrence of "stg1" with "stg2"
  for the next n lines.  The default value for m and n is 1.'''
  </code></pre>
  
  <p>The similarly difficult to tokenize <code>X</code> and <code>Y</code> commands:</p>
  
  <pre><code>'''
  X [/command/[command/[...]]n]
  Y [/command/[command/[...]]n]
  
      The X and Y commands allow the execution of several commands contained
  in one command.  To define an X or Y "command string", enter X (or Y)
  followed by a space, then individual commands, each separated by a
  delimiter (e.g. a period ".").  An unlimited number of commands may be
  placed in the X or Y command string.  Once the command string has been
  defined, entering X (or Y) followed optionally by a count n will execute
  the defined command string n times.  If n is not specified, it will
  default to 1.'''
  </code></pre>