Python: calculating checksum of a file for a weird protocol
Question: I'm having troubles calculating something like a checksum of a file for a
weird protocol that I'm trying to port to python.

The checksum is a 4 byte unsigned integer that is the result of adding all the
4-bytes unsigned integers of a file. For example, suppose the following file
(note that a real one is about 16MB):

    
    
    ff fe fd fc fb fa f9 f8  f7 f6 f5 f4 f3 f2 f1 f0
    ef ee ed ec eb ea e9 e8  e7 e6
    

With my implementation (see below), the computation is as this:

    
    
    0xfffefdfc + 0xfbfaf9f8 + 0xf7f6f5f4 + 0xf3f2f1f0 + 0xefeeedec + 0xebeae9e8 + 0xe7e60000 = 0x6aba3b7ac
    

But, it should be `0xaba3b7ac`

I've tried to do something like this:

    
    
    import mmap
    import struct
    
    # Prepare file
    file = open("file.bin", 'rb')
    map = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
    
    # Calculate checksum
    checksum = 0
    while (map.tell() < map.size()):
        checksum += struct.unpack('>I', map.read(4))[0]
    
    print "checksum: ", checksum
    
    # Close file.
    map.close()
    file.close()
    

But I see two problems.

  1. First of all, the output number is too big. I need a 4 byte number. With the previous code, the output for a typical test file is something like `0x165c0458b224ae` but it should be something like `0xcaac5458` (a 4-byte unsigned integer).
  2. My method is slow. The original code (which is written in C) does the calculation much faster.

I'm really stuck with this so any help will be really appreciated.

Thanks in advance and sorry for my bad english.

* * *

**Update:**

The first problem was solved by Serge Ballesta. The solution is to add the
following line before printing the checksum:

    
    
    checksum &= 0xffffffff
    

But the computation is still slow. I would like to have a fast solution but I
don't know how to do it.


Answer: One way is to use the `array` module and load all the four bytes into memory,
sum those, then force back to a 4 byte (as Serge has already mentioned).

    
    
    import os
    from array import array
    
    #with open('data.bin', 'wb') as fout:
    #   fout.write(os.urandom(16800000))
    
    with open('data.bin', 'rb') as fin:
        arr = array('L')
        arr.fromfile(fin, 16800000 / 4)
        arr.byteswap()
        checksum = sum(arr) & 0xFFFFFFFF
    

Takes less than a second on my laptop... not sure how much faster you want it
to go...

