Using the file descriptor returned from C extension to read the files in python
Question: Bare with me, am new to python.

When I try reading a file in python, it blocks other processes from editing
that file. Even though the file is opened in read mode.

I couldn't find and option which would enable me to achieve this. So, what am
trying to do is, send the file name to a C extension and open the file there
with required option and return the file descriptor from there. And, use this
descriptor to get the file object and read the file.

The code I have tried is:

The C code fileread.h

    
    
    #include <python.h>
    
    static PyObject* fileread(PyObject *self, PyObject *args)
    {
        char* filename = NULL;
        int fd = 0;
        if (!PyArg_ParseTuple(args, "s", &filename)) {
            return NULL;
        }
    
        fd = _sopen(filename, 0x0000, 0x40, 0x0100);
        // _sopen(filename,_O_RDONLY, _SH_DENYNO, _S_IREAD);
        return Py_BuildValue("i", fd);
    }
    
    static PyMethodDef fileread_funcs[] = {
        { "fileread", (PyCFunction)fileread,
        METH_VARARGS, "read file in blocks" },
        { NULL, NULL, 0, NULL }
    };
    
    void initfileread(void)
    {
        Py_InitModule3("fileread", fileread_funcs,
            "Extension for file read!");
    }
    

And, the fileread.py is:

    
    
    import os
    import fileread
    
    def ReadDataBlockByBlock(dirPath, fileName):
        path = os.path.join(dirPath, fileName)
    
        if os.access(path, os.R_OK):
            fd = PyObjectAsFileDescriptor(fileread.fileread(path))
            fp = os.fdopen(fd,'r') #Is Error: Expects integer
    
        for block in read_in_chunks(fp):
            print block
            print '*' * 80
    
        os.close(fd)
    
     def read_in_chunks(file_object, chunk_size=1096):
        """Function (generator) to read a file piece by piece.
        Default chunk size: 1k."""
    
        while True:
            data = os.read(file_object, chunk_size)
            if not data:
                break
            yield data
    

When I try to do fdopen() here, it throws an error. What am I doing wrong?

This may not be the right approach, am new to python. Any better way of
achieving this is also very much appreciated

Thanks, Harsh


Answer: By default, Python does not lock files, but see the `fcntl` module if you need
to do so.

But other processes which do lock files may not be able to acquire the lock if
a Python process has a file open. (This is heavily OS-dependent behavior.)

To prove that it is not Python that is keeping the other process from
accessing the file, open two different terminal programs or cmd windows, fire
up Python in both of them, and open the file for reading in each one. This
should work, and will show that it's the other process complaining that it
cannot open (and lock) a file that your Python process has open, and not
Python itself acquiring a lock on the file.

In general, the best way to deal with this is to open the file, do your file
operations, and close it again right away. Unfortunately, though, if your
editor does not let other processes open files, that is a problem that you are
going to have to deal with. You should check your editor configuration
settings, to see if it has one for exclusive access that you can turn off, and
if not, you should consider a different editor.

