Cython wrapper not fast enough, how to speed it up?
Question: I have a C++ library and I want to wrap some of its functionality in python.

The function splits the given character array into 5 parts, not actual
splitting but the structure we pass a pointer to, contains the information
about the parts after the function returns. The 5 structures each contain 2
integers, one denoting the beginning of the part, and the other, the part's
length.

The python wrapper should accept a python string and return a dictionary or
tuple of the 5 parts(as python strings also).

My current approach of calling the function and then splitting the python
string based on the sub-part information using python slicing syntax has not
yielded any significant speed gains. I realize that there are many similar
questions, but none of those cases have been helpful to me.

The Cython definition code is -

    
    
    cdef extern from "parse.h" namespace util
        ctypedef struct part:
            int begin;
            int len;
    
        ctypedef struct Parsed:
            part part1;
            part part2;
            part part3;
            part part4;
            part part5;
    
        void ParseFunc(const char* url, int url_len, Parsed* parsed)
    

The Cython code is -

    
    
    cimport parseDef
    
    def parse(url, url_len):
        cdef parseDef.Parsed parsed
        parseDef.parseFunc(url, url_len, &parsed)
    
        part1 = url[parsed.part1.begin:parsed.part1.begin+parsed.part1.len]
        #similar code for other parts
    
        return (part1, part2, part3, part4, part5)
    

Typical string size for this wrapper will be 10-50 generally.


Answer: You can get a small benefit from doing the indexing on `const char*` instead
of the string

    
    
    cimport parseDef
    
    def parse(url, url_len):
        cdef const char* url_as_char_ptr = url # automatic conversion
        cdef parseDef.Parsed parsed
        parseDef.parseFunc(url, url_len, &parsed)
    
        part1 = url_as_char_ptr[parsed.part1.begin:parsed.part1.begin+parsed.part1.len]
        #similar code for other parts
    
        return (part1, part2, part3, part4, part5)
    

I don't think you can beat this by much is that, mostly because the c-code
generated is actually pretty efficient. The indexing line is translated to
something like

    
    
    __pyx_t_2 = __Pyx_PyBytes_FromStringAndSize(__pyx_v_url_as_char_ptr + idx1, idx2 - idx1)
    

(noting that I've replaced `parsed.part1.begin` with `idx1` just for the sake
of readability and because I'm testing this with slightly different code since
I don't have `parseFunc`. You can check your exact code with `cython -a
yourfile.pyx` and looking at the html output).

This is basically just calling the Python c-api string constructor function.
That will necessarily make a copy of the string it is passed, but you can't
avoid that (the Python string constructor always makes a copy). That doesn't
leave a lot of overhead to remove.

