UDF (User Defined Function) python gives different answer in pig
Question: I want to write a UDF python for pig, to read lines from the file called like

    
    
    #'prefix.csv' 
    spol.
    LLC 
    Oy
    OOD
    

and match the names and if finds any matches, then replaces it with white
space. here is my python code

    
    
    def list_files2(name, f):
        fin = open(f, 'r')
        for line in fin:
            final = name
            extra = 'nothing'
            if (name != name.replace(line.strip(), ' ')):
                extra = line.strip()
                final = name.replace(line.strip(), ' ').strip()
                return final, extra,'insdie if'
        return final, extra, 'inside for'
    

Running this code in python,

    
    
    >print list_files2('LLC nakisa', 'prefix.csv' )
    >print list_files2('AG company', 'prefix.csv' )
    

returns

    
    
     ('nakisa', 'LLC', 'insdie if')
     ('AG company', 'nothing', 'inside for')
    

which is exactly what I need. But when I register this code as a UDF in apache
pig for this sample list:

    
    
    nakisa company LLC
    three Oy 
    AG Lans 
    Test OOD
    

pig returns wrong answer on the third line:

    
    
    ((nakisa company,LLC,insdie if))
    ((three,Oy,insdie if))
    ((A G   L a n s,,insdie if))
    ((Test,OOD,insdie if))
    

The question is why UDF enters the if loop for the third entry which does not
have any match in the prefix.csv file.


Answer: I don't know `pig` but the way you are checking for a match is strange and
might be the cause of your problem.

If you want to check whether a string is a substring of another, `python`
provides the `find` method on strings:

    
    
    if name.find(line.strip()) != -1:
        # find will return the first index of the substring or -1 if it was not found
        # ... do some stuff
    

additionally, your code might leave the file handle open. A way better
approach to handle file operations is by using the `with` statement. This
assures that in any case (except of interpreter crashes) the file handle will
get closed.

    
    
    with open(filename, "r") as file_:
        # Everything within this block can use the opened file.
    

Last but not least, `python` provides a module called `csv` with a `reader`
and a `writer`, that handle the parsing of the csv file format.

Thus, you could try the following code and check if it returns the correct
thing:

    
    
    import csv
    
    def list_files2(name, filename):
        with open(filename, 'rb') as file_:
            final = name
            extra = "nothing"
            for prefix in csv.reader(file_):
                if name.find(prefix) != -1:
                    extra = prefix
                    final = name.replace(prefix, " ")
                    return final, extra, "inside if"
            return final, extra, "inside for"
    

Because your file is named `prefix.csv` I assume you want to do prefix
substitution. In this case, you could use `startswith` instead of `find` for
the check and replace the line `final = name.replace(prefix, " ")` with `final
= " " + name[name.find(prefix):]`. This assures that only a prefix will be
substituted with the space.

I hope, this helps

