Video frame to Minecraft map colors
Iterate on x, not y
This computation loop:

for (int y = 0; y &lt; params-&gt;ctx-&gt;height; y++) {
    p = params-&gt;x * 3 + y * params-&gt;ctx-&gt;rgbFrame-&gt;linesize[0];
    r = params-&gt;ctx-&gt;rgbFrame-&gt;data[0][p];
    g = params-&gt;ctx-&gt;rgbFrame-&gt;data[0][p + 1];
    b = params-&gt;ctx-&gt;rgbFrame-&gt;data[0][p + 2];
    params-&gt;ctx-&gt;data[(params-&gt;x / 128) + (params-&gt;ctx-&gt;width / 128) *
          (y / 128)][((params-&gt;x % 128) + 128 * (y % 128))] = matchColor(r, g, b);
}


is inefficient because you are looping over y whereas your arrays are stored in x order.  For example, from the first iteration to the next, you will be reading from:
(y = 0) params-&gt;ctx-&gt;rgbFrame-&gt;data[0][0]
(y = 1) params-&gt;ctx-&gt;rgbFrame-&gt;data[0][linesize]

If you had iterated over x, you would be reading from:
(x = 0) params-&gt;ctx-&gt;rgbFrame-&gt;data[0][0]
(x = 1) params-&gt;ctx-&gt;rgbFrame-&gt;data[0][3]

Similarly, you are writing to:
(y = 0) params-&gt;ctx-&gt;data[0][0]
(y = 1) params-&gt;ctx-&gt;data[0][128]

If you iterated over x, you would be writing to:
(x = 0) params-&gt;ctx-&gt;data[0][0]
(x = 1) params-&gt;ctx-&gt;data[0][1]

There are two reasons why accessing nearby addresses is faster:

You will be utilizing the CPU cache efficiently by reusing the same cache lines instead of causing lots of cache misses by accessing memory all over the place.
Since your program is multithreaded, your threads will stop contending on the same cache lines.  In other words, right now each of your threads is writing to the same cache lines as each other thread, because they are writing to neighboring addresses in the data array.  By switching to x order, your threads will each write to their own separate rows in the data array, which means they will be writing to different cache lines from each other (for the most part).  This will reduce cache contention and speed things up.