Perl Compatible Regular Expression (PCRE) in Python
Question: I have to parse some strings based on PCRE in Python and I've no idea how to
do that.

Strings I want to parse looks like :

    
    
    match mysql m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s p/MySQL/ i/$1/
    

In this example, I have to get this different items :

    
    
    "m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s" ; "p/MySQL/" ; "i/$1/"
    

The only thing I've found relating to PCRE manipulation in Python is this
module : <http://pydoc.org/2.2.3/pcre.html> (but it's written it's a .so ..)

Do you know if some python module exist to parse this kind of string ?


Answer: # Be Especially Careful with non‐ASCII in Python

There are some really subtle issues with how Python deals with, or fails to
deal with, non-ASCII in patterns and strings. Worse, these disparities vary
substantially according not just to which version of Python you are using, but
also whether you have a “wide build”.

In general, when you’re doing Unicode stuff, **Python3 with a wide build works
best** and Python2 with a narrow build works worst, but all combinations are
still a pretty far cry far from how Perl regexes work _vis‐à‐vis_ Unicode. If
you’re looking for ᴘᴄʀᴇ patterns in Python, you may have to look a bit further
afield than its old `re` module.

**EDIT** : The vexing “wide-build” issues have **_finally_ been fixed** once
and for all — provided you use a sufficiently advanced release of Python.
Here’s an excerpt from [the v3.3 release
notes](http://docs.python.org/dev/whatsnew/3.3.html):

> ## _Functionality_
>
> Changes introduced by [PEP 393](http://www.python.org/dev/peps/pep-0393) are
> the following:
>
>   * Python now always supports the full range of Unicode codepoints,
> including non-BMP ones (i.e. from U+0000 to U+10FFFF). The distinction
> between narrow and wide builds no longer exists and Python now behaves like
> a wide build, even under Windows.
>   * With the death of narrow builds, the problems specific to narrow builds
> have also been fixed, for example:
>     * `len()` now always returns 1 for non-BMP characters, so
> `len('\U0010FFFF') == 1`;
>     * surrogate pairs are not recombined in string literals, so
> `'\uDBFF\uDFFF' != '\U0010FFFF'`;
>     * indexing or slicing non-BMP characters returns the expected value, so
> `'\U0010FFFF'[0]` now returns `'\U0010FFFF'` and not `'\uDBFF'`;
>     * all other functions in the standard library now correctly handle non-
> BMP codepoints.
>   * The value of `sys.maxunicode` is now always 1114111 (0x10FFFF in
> hexadecimal). The `PyUnicode_GetMax()` function still returns either 0xFFFF
> or 0x10FFFF for backward compatibility, and it should not be used with the
> new Unicode API (see [issue 13054](http://bugs.python.org/issue13054)).
>   * `The ./configure` flag `--with-wide-unicode` has been removed.
>

## The Future of Python Regexes

In contrast to what’s currently available in the standard Python
distribution’s `re` library, [Matthew Barnett’s `regex` module for both Python
2 and Python 3 alike](http://pypi.python.org/pypi/regex) is much, much better
in pretty much all possible ways, and will quite probably replace `re`
eventually. Its particular relevance to your question is that his `regex`
library is far more ᴘᴄʀᴇ (_i.e._ **it’s much more Perl‐compatible**) in every
way than `re` now is, which will make porting Perl regexes to Python easier
for you. Because it is a ground‐up rewrite (as in from‐scratch not as in
hamburger :), it was written with non-ASCII in mind, which `re` was not.

The `regex` library therefore much more closely follows the (current)
recommendations of [UTS#18: Unicode Regular
Expressions](http://unicode.org/reports/tr18/) in how it approaches things. It
**meets or exceeds the UTS#18 Level 1 requirements** in most if not all
regards, something you normally have to use the ICU regex library or Perl
itself for — or if you are especially courageous, the new Java 7 update to its
regexes, as that also conforms to the [Level One
requirements](http://unicode.org/reports/tr18/#Basic_Unicode_Support) from
UTS#18.

Beyond meeting those Level One requirements, which are all absolutely
essential for basic Unicode support but which are **not met by Python’s
current`re` library,** the awesome `regex` library also meets the Level Two
requirements for [RL2.5](http://unicode.org/reports/tr18/#Name_Properties)
Named Characters (`\N{...})`),
[RL2.2](http://unicode.org/reports/tr18/#Default_Grapheme_Clusters) Extended
Grapheme Clusters (`\X`), and the new RL2.7 on Full Properties from [revision
14 of UTS#18](http://www.unicode.org/reports/tr18/tr18-14.html).

Matthew’s `regex` module also does Unicode casefolding so that case
insensitive matches work reliabably on Unicode, **which`re` does not.**

**EDIT** : The following is no longer true, because `regex` now supports full
Unicode casefolding, like Perl and Ruby.

> > ~~One super‐tiny difference is that for now, Perl’s case‐insensitive
> patterns use full string‐oriented casefolds while his`regex` module still
> uses simple single‐char‐oriented casefolds, but this is something he’s
> looking into. It’s actually a very hard problem, one which apart from Perl,
> only Ruby even attempts.~~

Under full casefolding, this means that (for example) `"ß"` now correct
matches `"SS"`, `"ss"`, `"ſſ"`, `"ſs"` (etc.) when case-insensitive matching
is selected. (This is admittedly more important in the Greek script than the
Latin one.)

See also the slides or doc source code from [my 3rd OSCON2011
talk](http://training.perl.com/OSCON2011/index.html) entitled _“**Unicode
Support Shootout: The Good, the Bad, and the (mostly) Ugly** ”_ for general
issues in Unicode support across Javascripts, PHP, Go, Ruby, Python, Java, and
Perl. If can’t use either Perl regexes or possibly the ICU regex library
(which doesn’t have named captures, alas!), then Matthew’s `regex` for Python
is probably your best shot.

* * *

Nᴏᴛᴀ Bᴇɴᴇ s.ᴠ.ᴘ. (= _s’il vous plaît, et même s’il ne vous plaît pas_ :) The
following unsolicited noncommercial nonadvertisement was _not_ actually put
here by the author of the Python `regex` library. :)

## Cool `regex` Features

The Python `regex` library has a **cornucopeia of superneat features** , some
of which are found in no other regex system anywhere. These make it very much
worth checking out no matter whether you happen to be using it for its
ᴘᴄʀᴇ‐ness or its stellar Unicode support.

A few of this module’s outstanding features of interest are:

  * **Variable‐width lookbehind** , a feature which is quite rare in regex engines and very frustrating not to have when you really want it. This may well be the most frequently requested feature in regexes.
  * Backwards searching so you don’t have to reverse your string yourself first.
  * Scoped `ismx`‐type options, so that `(?i:foo)` only casefolds for foo, not overall, or `(?-i:foo)` to turn it off just on foo. This is how Perl works (or can).
  * Fuzzy matching based on edit‐distance (which Udi Manber’s `agrep` and `glimpse` also have)
  * Implicit shortest‐to‐longest sorted named lists via `\L<list>` interpolation
  * Metacharacters that specifically match only the start or only the end of a word rather than either side (`\m`, `\M`)
  * Support for all Unicode line separators (Java can do this, as can Perl albeit somewhat begrudgingly with `\R` per [RL1.6](http://unicode.org/reports/tr18/#Line_Boundaries).
  * Full set operations — union, intersection, difference, and symmetric difference — on bracketed character classes per [RL1.3](http://unicode.org/reports/tr18/#Subtraction_and_Intersection), which is much easier than getting at it in Perl. 
  * Allows for repeated capture groups like `(\w+\s+)+` where you can get all separate matches of the first group not just its last match. (I believe C♯ might also do this.)
  * A more straightfoward way to get at overlapping matches than sneaky capture groups in lookaheads.
  * Start and end positions for all groups for later slicing/substring operations, much like Perl’s `@+` and `@-` arrays.
  * The branch‐reset operator via `(?|...|...|...|)` to reset group numbering in each branch the way it works in Perl.
  * Can be configured to have your coffee waiting for you in the morning.
  * Support for the more sophisticated word boundaries from [RL2.3](http://unicode.org/reports/tr18/#Default_Word_Boundaries).
  * Assumes Unicode strings by default, and fully supports [RL1.2a](http://unicode.org/reports/tr18/#Compatibility_Properties) so that `\w`, `\b`, `\s`, and such work on Unicode.
  * Supports `\X` for graphemes.
  * Supports the `\G` continuation point assertion.
  * Works correctly for 64‐bit builds (`re` only has 32‐bit indices).
  * Supports multithreading.

Ok, that’s enough hype. :)

# Yet Another Fine Alternate Regex Engine

One final alternative that is worth looking at if you are a regex geek is the
[Python library bindings](http://pypi.python.org/pypi/re2/) to Russ Cox’s
awesome [RE2 library](http://swtch.com/~rsc/regexp/). It also supports Unicode
natively, including simple char‐based casefolding, and unlike `re` it notably
provides for both the Unicode General Category and the Unicode Script
character properties, which are the two key properties you most often need for
the simpler kinds of Unicode processing.

Although RE2 misses out on a few Unicode features like `\N{...}` named
character support found in ICU, Perl, and Python, it has extremely serious
computational advantages that make it **the regex engine of choice** whenever
you’re concern with starvation‐based denial‐of‐service attacks through regexes
in web queries and such. It manages this by forbidding backreferences, which
cause a regex to stop being regular and risk super‐exponential explosions in
time and space.

Library bindings for RE2 are available not just for C/C++ and Python, but also
for Perl and most especially for Go, where it is slated to very shortly
replace the standard regex library there.

