Functionality to read large files (> 3 GB) in chunks Here is a function I wrote that can read chunks of large files (> 3 GB). It's designed to be used contentiously so that one can use it in a while loop until it returns EOF. 

It's an early prototype and is only written to work under 32-bit Linux.

I'm okay with feedback on readability, maintainability, or anything else.

/**
 * Reads a chunk of given size into memory (heap) from file. Chunk size is
 * measured in amount of pages.
 * 
 * The chunk will be allocated before reading starts. So no matter what
 * the return code you always need to decref the chunk.
 *             
 * Parameters: 
 * - pages    : The amount of pages to read to memory. The size of a page
 *              a page differs from different systems, but it is usually 4 KiB.
 *              See getpagesize or _SC_PAGESIZE for sysconf in unistd.h for
 *              details.
 * - start_pos: The position in the file to start reading from
 * - *end_pos : Pointer to the location where the end position of the
 *              reading shall be placed. This will be the byte after the last
 *              read byte, so that read_chunk can be called with this value
 *              as start_pos directly.
 * - **chunk  : Pointer to the pointer of an byte array where the result
 *              will be placed in.
 *             
 * Return values:
 * - Returns 0 on success      
 * - Returns -1 if chunk size was above 2^31 bits (~ 2 GB)             
 * - Returns -2 if an fseek error occured, see errno for details       
 * - Returns -3 if an fread error occured, see errno for details       
 * - Returns -4 on EOF         
 */

int8_t read_fachunk(           
        FILE *f,               
        uint16_t pages,        
        uint64_t start_pos,    
        uint64_t *end_pos,     
        struct byte_array **chunk)     
{              
        assert(f != NULL);     
        assert(end_pos != NULL);       
        assert(chunk != NULL); 
        assert(pages != 0);    

        uint64_t chunk_b = pages * sysconf(_SC_PAGESIZE);              

        if(chunk_b &gt; MAX_CHUNK_SIZE) { 
            return -1;     
        }      

        uint64_t curr_pos = 0; 
        int64_t err = lseek64(fileno(f), start_pos, SEEK_SET);

        *chunk = byte_array_create(chunk_b);           

        if (err == -1) return -2;      

        curr_pos = fread((*chunk)-&gt;bytes, sizeof(uint8_t), chunk_b, f);
        (*chunk)-&gt;curr_i = curr_pos;   

        if (ferror(f) != 0) {  
            err = -3;      
            goto complete; 
        }      

        if (feof(f) != 0) {    
            err = -4;      
            goto complete; 
        }      

        err = 0;               

        complete:              
    *end_pos = start_pos + curr_pos;       
    return err;    
}
Here are a few comments on your function.

Firstly taking the prototype, why return an int8_t. A simple int is better
as uint8_t will probably be slower.  Also returning several types of error
is unusual in UNIX functions like this.  I'd expect errno to tell what
failed and suggest you return the number of bytes read, 0 on EOF and -1 on
error (like read(2)).

And again on the prototype, why are you passing a FILE instead of a
descriptor?  Passing a FILE and then messing around with the raw descriptor
makes me uneasy.



You allocate a strange structure in the function.  This is odd for a low level
utility.  It is unnecessary and arguably wrong, as it means the caller cannot
avoid an allocation even if the buffer from the last call can be reused or a
static buffer is available.  Just pass a buffer and its size (as a size_t)
into the call.

And it would be more normal to pass in the size to read (as a size_t)
instead of the number of pages. If you pass a buffer and its size as I
suggested above, you don't need an extra size to say how much to read.



You are mixing use of buffered fread (operating on a FILE) and lseek (on
an fd).  This is a bad idea as you are messing with something the FILE has
reason to call its own.  If you are going to seek, then use read to read the
data.

And in your use of fread, the call returns the number of items read (number
of bytes in this case) so assigning that to a variable called curr_pos is
not sensible. The name nread would make more sense.  And keeping that value
in the byte_array structure with a different, equally meaningless name,
curr_i, is unnecessary.

Also, reading up to end-of-file might return less than requested.  That is
not an error from the perspective of this function (it might be from an
application perspective if the file size is expected to be multiples of page
size, but the handling of such an error doesn't belong here).  So calling
feof and treating end-of-file as an error is wrong.  There is only a
possible error it nread != chunk_b.  But you should be using read(2)...



Some other points:


your goto statements are unnecessary.  Occasionally gotos are useful in
simplifying error recovery, but in this case they simplify nothing.
your end_pos is unnecessary.  The file descriptor knows its current
offset, which is available with fseek(fd, 0, SEEK_CUR)
chunk_b is a bad name for a size, don't you think?  
test the return from lseek immediately. If you have a stderr, use perror
to print the reason for failure (perhaps best in the calling functions).
use of sizeof(uint8_t) is silly - just use 1


EDIT

FILEs and fopen, fwrite, fread, fprintf, fgets etc are provided by stdio for use as buffered IO in which the standard library maintains a buffer through which your requests are served.  When you use fread to read (say) 10 bytes, the library might read a whole page or disc sector or line of input from the terminal (on the assumption that you'll want the rest later) and give you just the amount you asked for, saving the rest in the buffer.  The "file descriptor" (fd), which is returned by fileno, is the lower level interface to the actual file returned by open (type man 2 open on UNIX/Linux/MacOS) - open is used by fopen to open the actual file/device. 

If you are doing unbuffered or 'raw' IO, you can use open/write/read/lseek etc to save the overhead of the stdio and to give you more control. Since you are explicitly seeking to the location you want to read, you are certainly doing raw (unbuffered) IO and the lower-level interface makes more sense. 

On the interaction between buffered and unbuffered IO on the same underlying file descriptor, I've never read anything that suggests this is forbidden.  But intuitively it seems that using buffered IO and then moving the file's position by seeking behind stdio's back is asking for trouble. Also, if you are on a 32-bit system, stdio probably does not support files bigger than 2GB (or perhaps 4). 

I just would never mix calls to the FILE and the fd.  Hopefully someone else with deeper knowledge will shed some more light on this.