Python: testing for utf-8 character in string
Question: I need to test whether a string that has already been encoded with
str.encode('utf-8') is right-to-left. I tried

    
    
    if u'\u200f' in str.decode('utf-8'):
      print 'found it'
    

It neither complains nor works.

Q: What is the correct syntax to test for the occurrence of a single non-ASCII
character in a string? Python 2.6 and I can't use 3.

Q: I remember reading that predominantly right-to-left characters default to
RTL even without an explicit RML. Does anyone know a way to test such a string
without knowing which language to expect (i.e. the string can be in Arabic,
Hebrew or any other RTL language)?

Thanks for all help.


Answer: Every unicode character has a "bidirectional" class. You can find the
bidirectional class using
[unicodedata.bidirectional](http://docs.python.org/2/library/unicodedata.html#unicodedata.bidirectional).
The function returns a string, e.g. 'L', 'R', 'AL', etc. with the [following
meaning](http://www.unicode.org/reports/tr44/tr44-4.html):

    
    
    | L   | Left_To_Right           | any strong left-to-right character                                |
    | LRE | Left_To_Right_Embedding | U+202A: the LR embedding control                                  |
    | LRO | Left_To_Right_Override  | U+202D: the LR override control                                   |
    | R   | Right_To_Left           | any strong right-to-left (non-Arabic-type) character              |
    | AL  | Arabic_Letter           | any strong right-to-left (Arabic-type) character                  |
    | RLE | Right_To_Left_Embedding | U+202B: the RL embedding control                                  |
    | RLO | Right_To_Left_Override  | U+202E: the RL override control                                   |
    | PDF | Pop_Directional_Format  | U+202C: terminates an embedding or override control               |
    | EN  | European_Number         | any ASCII digit or Eastern Arabic-Indic digit                     |
    | ES  | European_Separator      | plus and minus signs                                              |
    | ET  | European_Terminator     | a terminator in a numeric format context, includes currency signs |
    | AN  | Arabic_Number           | any Arabic-Indic digit                                            |
    | CS  | Common_Separator        | commas, colons, and slashes                                       |
    | NSM | Nonspacing_Mark         | any nonspacing mark                                               |
    | BN  | Boundary_Neutral        | most format characters, control codes, or noncharacters           |
    | B   | Paragraph_Separator     | various newline characters                                        |
    | S   | Segment_Separator       | various segment-related control codes                             |
    | WS  | White_Space             | spaces                                                            |
    | ON  | Other_Neutral           | most other symbols and punctuation marks                          |
    

For instance:

    
    
    In [3]: import unicodedata as UD
    In [5]: UD.bidirectional(u'\u0688')
    Out[5]: 'AL'
    
    In [6]: UD.bidirectional(u'\u200f')
    Out[6]: 'R'
    
    In [7]: UD.bidirectional(u'H')
    Out[7]: 'L'
    

* * *

So you might be able to _guess_ if a string is right-to-left by determining if
it is composed mainly of characters whose bidirectional class is `R` or `AL`.

For example,

    
    
    # coding: utf-8
    import unicodedata as UD
    
    texts = ['ڈوگرى'.decode('utf-8'),
             u'Hello']
    for text in texts:
        x = len([None for ch in text if UD.bidirectional(ch) in ('R', 'AL')])/float(len(text))
        print('{t} => {c}'.format(t=text.encode('utf-8'), c='RTL' if x>0.5 else 'LTR'))
    

yields

    
    
    ڈوگرى => RTL
    Hello => LTR
    

* * *

Regarding the first question:

> Q: What is the correct syntax to test for the occurrence of a single non-
> ASCII character in a string? Python 2.6 and I can't use 3.

Your method for testing if a character is in a `unicode` is correct. If
`u'\u200f' in str.decode('utf-8')` neither complains nor works, then
`u'\u200f'` is not in the `unicode`.

