Function to get the most frequently appearing pixel What i have so far is a nice and rapid algorithm for finding the pixel dominant from bmp image.
I am not sure how to make it to work.. i though it works, but the program freezes.

RGB
bitfox_get_primecolor_direct
(char *FILE_NAME)
{
    RGB primecolor;
    BYTE rgb[3];

    dword *counts = calloc(pow(256, 3), sizeof(*counts));
    dword max_count = 0;

    FILE* fp = fopen(convert(FILE_NAME), "rb");

    fseek(fp, 54, SEEK_SET);
    while (fread (rgb, sizeof(BYTE), 3, fp) != EOF)
    {
        dword idx = (((dword)rgb[0]) &lt;&lt; 16) | (((dword)rgb[1]) &lt;&lt; 8) | (dword)rgb[2];
        if (++counts[idx] &gt; max_count) max_count = idx;
    }

    primecolor.R = (max_count &gt;&gt; 16) &amp; 0xFF;
    primecolor.G = (max_count &gt;&gt; 8) &amp; 0xFF;
    primecolor.B = max_count &amp; 0xFF;

    free(counts);
    fclose(fp);
    return primecolor;
}


BYTE is unsigned char

RGB is struct containg BYTE R, G and B.
There are numerous problems with the code. First of all, fread never returns EOF (which is -1). The loop should end when it reads less than 3 bytes.

Second, max_count is index, not a color value. I presume you meant rgb[max_count]. 

Test a file pointer returned by fopen for not being NULL.

Test a return value of fseek.

Avoid magic numbers. I suppose 54 is a size of certain header. Create a constant with a meaningful name. Better yet, define a header structure and take its sizeof.