Fastest and/or most pythonic way to convert string into a number
Question: first of all, I've seen quite a few questions related to this (convert string
to float, etc etc), but I need something more generic, which I could not find
(so I hope this will also help out other people with a similar problem). I
have made a solution, but am wondering whether it is the best solution in
terms of 1) performance and 2) pythonic elegance.

The problem in short:

  * I get data from a variety of sources, these are made into a list with dicts (as a row/column table setup). 
  * The variety means that I cannot rely on a fixed input type (basically they might be string, boolean, int, float) but the user can designate which columns (keys in the dict) are values. 
  * Which I then need to convert to actual value types (we're talking about 100s of millions of rows of data here, so performance is rather key). 
  * If the input is not a real number (like: 'aaa'), then it should return None.
  * There might be currency symbols and thousand separators (which need to be removed), and decimal separators (which need to be replaced by the standard dot, if it's not a dot)

So what have I made:

    
    
    import ast
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
                if s:
                    result = 1
                else:
                    result = 0
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
        else: # convert a string
            # prepare the string for conversion
            if currency_sign != '':
                s = s.replace(currency_sign, '')
            if thousand_sep != '':
                s = s.replace(thousand_sep, '')
            if decimal_sep != '.':
                s = s.replace(decimal_sep, '.')        
            s = s.strip()
            # convert the string
            if s == '':
                result = None
            else:
                try:
                    # convert the string by a safe evaluation
                    result = ast.literal_eval(s)
                    # check if result of the evaluation is a number type
                    if not isinstance(result, NumberTypes):
                        result = None
                except ValueError:
                    # if the conversion gave an error, the string is not a number
                    result = None
        return result
    

You can test it by:

    
    
    mk_value(True)
    mk_value(1234)
    mk_value(1234.56)
    mk_value('1234')
    mk_value('1234.56')
    mk_value('1,234.56') # without an explicit decimal separator this is not a number
    mk_value('1.234.567,89 EUR', currency_sign='EUR', thousand_sep='.', decimal_sep=',') # all exceptions
    

So this works (as far as I can see); but is this the best/most pythonic way?
Are there faster ways? Should I look into Cython for this? Any ideas on
improving this would be really helpful!

BR

Carst

Edit: I've updated my code based on the suggestions by Andrew and WoLpH. It
now looks like this:

    
    
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
                if s:
                    result = 1
                else:
                    result = 0
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
        else: # convert a string
            # prepare the string for conversion
            if currency_sign:
                s = s.replace(currency_sign, '')
            if thousand_sep:
                s = s.replace(thousand_sep, '')
            if decimal_sep != '.':
                s = s.replace(decimal_sep, '.')        
            s = s.strip()
            # convert the string
            if not s: # if the string is empty, it's not a number
                result = None
            else:
                try: # try int
                    result = int(s)
                except ValueError:
                    try: # if there's an error, try float
                        result = float(s)
                    except ValueError:                    
                        # if the conversion gave an error, the string is not a number
                        result = None
        return result
    

the previous code's performance was this:

    
    
    >>> timeit.timeit("mk_value(1234)", 'from __main__ import mk_value', number=100000)
    0.050575971603393555
    >>> timeit.timeit("mk_value(1234.56)", 'from __main__ import mk_value', number=100000)
    0.07073187828063965
    >>> timeit.timeit("mk_value('1234')", 'from __main__ import mk_value', number=100000)
    0.8333430290222168
    >>> timeit.timeit("mk_value('1234.56')", 'from __main__ import mk_value', number=100000)
    0.8230760097503662
    >>> timeit.timeit("mk_value('1,234.56', thousand_sep=',')", 'from __main__ import mk_value', number=100000)
    0.9358179569244385
    

the new code's performance:

    
    
    >>> timeit.timeit("mk_value(1234)", 'from __main__ import mk_value', number=100000)
    0.04723405838012695
    >>> timeit.timeit("mk_value(1234.56)", 'from __main__ import mk_value', number=100000)
    0.06952905654907227
    >>> timeit.timeit("mk_value('1234')", 'from __main__ import mk_value', number=100000)
    0.1798090934753418
    >>> timeit.timeit("mk_value('1234.56')", 'from __main__ import mk_value', number=100000)
    0.45616698265075684
    >>> timeit.timeit("mk_value('1,234.56', thousand_sep=',')", 'from __main__ import mk_value', number=100000)
    0.5290899276733398
    

So that's a lot faster: almost twice as fast for the most complex one and much
much faster for the int (I guess as it's the first in the try/except logic)!
Really great, thanks for your input.

I'm going to leave it open for now to see if someone has a brilliant idea on
how to improve more :) At the very least I hope this will help other people in
the future (it must be a very common issue)


Answer: It could be slightly more Pythonic imho, but I'm not sure about the best
solution yet.

# Code

## `benchmark.py`

    
    
    # vim: set fileencoding=utf-8 :
    import timeit
    import pyximport
    pyximport.install()
    
    def timer(func, mod):
        import_ = 'from %s import mk_value' % mod
        time = timeit.timeit(func, import_, number=100000)
        ms = 1000 * time
        us = 1000 * ms
    
        if func[40:]:
            func_short = func[:37] + '...'
        else:
            func_short = func
    
        print '%(mod)s.%(func_short)-40s %(ms)6dms %(us)12dμs' % locals()
    
    for mod in 'abcd':
        timer("mk_value(1234)", mod)
        timer("mk_value(1234.56)", mod)
        timer("mk_value('1234')", mod)
        timer("mk_value('1234.56')", mod)
        timer("mk_value('1,234.56', thousand_sep=',')", mod)
        timer("mk_value('1.234.567,89 EUR', currency_sign='EUR', "
            "thousand_sep='.', decimal_sep=',')", mod)
    

## `a.py`

    
    
    import ast
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
                if s:
                    result = 1
                else:
                    result = 0
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
        else: # convert a string
            # prepare the string for conversion
            if currency_sign != '':
                s = s.replace(currency_sign, '')
            if thousand_sep != '':
                s = s.replace(thousand_sep, '')
            if decimal_sep != '.':
                s = s.replace(decimal_sep, '.')        
            s = s.strip()
            # convert the string
            if s == '':
                result = None
            else:
                try:
                    # convert the string by a safe evaluation
                    result = ast.literal_eval(s)
                    # check if result of the evaluation is a number type
                    if not isinstance(result, NumberTypes):
                        result = None
                except ValueError:
                    # if the conversion gave an error, the string is not a number
                    result = None
        return result
    

## `b.py`

    
    
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
                if s:
                    result = 1
                else:
                    result = 0
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
        else: # convert a string
            # prepare the string for conversion
            if currency_sign:
                s = s.replace(currency_sign, '')
            if thousand_sep:
                s = s.replace(thousand_sep, '')
            if decimal_sep != '.':
                s = s.replace(decimal_sep, '.')        
            s = s.strip()
            # convert the string
            if not s: # if the string is empty, it's not a number
                result = None
            else:
                try: # try int
                    result = int(s)
                except ValueError:
                    try: # if there's an error, try float
                        result = float(s)
                    except ValueError:                    
                        # if the conversion gave an error, the string is not a number
                        result = None
        return result
    

## `c.pyx`

    
    
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
            if s:
                result = 1
            else:
                result = 0
    
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
    
        else: # convert a string
            # prepare the string for conversion
            if currency_sign:
                s = s.replace(currency_sign, '')
            if thousand_sep:
                s = s.replace(thousand_sep, '')
            if decimal_sep != '.':
                s = s.replace(decimal_sep, '.')        
            s = s.strip()
            # convert the string
            if not s: # if the string is empty, it's not a number
                result = None
            else:
                try: # try int
                    result = int(s)
                except ValueError:
                    try: # if there's an error, try float
                        result = float(s)
                    except ValueError:                    
                        # if the conversion gave an error, the string is not a number
                        result = None
        return result
    

## `d.pyx`

    
    
    import types
    NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType)
    
    def  mk_value(s, currency_sign='', thousand_sep='', decimal_sep='.'):
        if isinstance(s, bool): # make boolean into a 0/1 value
            if s:
                result = 1
            else:
                result = 0
    
        elif isinstance(s, NumberTypes): # keep numbers as/is
            result = s
    
        elif s:
            if currency_sign:
                s = s.replace(currency_sign, '')
    
            result = _mk_value(s, currency_sign, thousand_sep, decimal_sep)
    
        else:
            result = None
    
        return result
    
    cdef object _mk_value(char *s, char *currency_sign, char *thousand_sep, char *decimal_sep):
        cdef int i=0, j=0
        result = None
    
        while s[i]:
            if s[i] == decimal_sep[0]:
                s[j] = '.'
                j += 1
            elif s[i] == thousand_sep[0]:
                pass
            elif s[i] == ' ':
                pass
            else:
                s[j] = s[i]
                j += 1
    
            i += 1
    
        # convert the string
        if not s: # if the string is empty, it's not a number
            result = None
        else:
            try: # try int
                result = int(s)
            except ValueError:
                try: # if there's an error, try float
                    result = float(s)
                except ValueError:                    
                    # if the conversion gave an error, the string is not a number
                    pass
        return result
    

# Results

    
    
    a.mk_value(1234)                               27ms        27526μs
    a.mk_value(1234.56)                            42ms        42097μs
    a.mk_value('1234')                            502ms       502109μs
    a.mk_value('1234.56')                         520ms       520395μs
    a.mk_value('1,234.56', thousand_sep=',')      570ms       570749μs
    a.mk_value('1.234.567,89 EUR', currency...    627ms       627456μs
    b.mk_value(1234)                               27ms        27082μs
    b.mk_value(1234.56)                            40ms        40014μs
    b.mk_value('1234')                             94ms        94444μs
    b.mk_value('1234.56')                         276ms       276519μs
    b.mk_value('1,234.56', thousand_sep=',')      315ms       315310μs
    b.mk_value('1.234.567,89 EUR', currency...    374ms       374861μs
    c.mk_value(1234)                               11ms        11482μs
    c.mk_value(1234.56)                            22ms        22765μs
    c.mk_value('1234')                             69ms        69251μs
    c.mk_value('1234.56')                         176ms       176908μs
    c.mk_value('1,234.56', thousand_sep=',')      226ms       226709μs
    c.mk_value('1.234.567,89 EUR', currency...    285ms       285431μs
    d.mk_value(1234)                               11ms        11483μs
    d.mk_value(1234.56)                            22ms        22355μs
    d.mk_value('1234')                             69ms        69151μs
    d.mk_value('1234.56')                         169ms       169364μs
    d.mk_value('1,234.56', thousand_sep=',')      187ms       187460μs
    d.mk_value('1.234.567,89 EUR', currency...    233ms       233935μs
    

