Estimate query speed for +1.5 billion rows mysql table
(UNHEXing is not a significant part of the problem.)

The real problem is the randomness of hashes.  The leads to jumping around lots of places on disk.  Let's dissect the query.


The IN list is a list of values scattered throughout the INDEX(hash).
Each value is looked up by drilling down a BTree (found in .MYI file) that is cached in MyISAM's key_buffer.  What is the value of key_buffer_size?  What is the result of `SHOW TABLE STATUS LIKE 'fingerprints' \G ?
If the index_size is greater than key_buffer_size, then many of the lookups will hit the disk.
At the leaf node of each BTree lookup will be a 5-byte (I think) "record number".
Now to lookup the row -- This will be a random disk access (a seek, no BTree) into the fingerprints.MYD at offset = 17 * record_number.  (The records appear to be FIXED length of 17 bytes.)
Again, we are looking at a probable disk hit -- now assuming that the remaining free space on disk is less than the Data_length (see the TABLE STATUS).


What to do?

Case 1:  Data_length + Index_length &lt; RAM size:  Have key_buffer_size a little bigger than Index_length.  Gradually both caches will fill with index and data, and the I/O will go away.

Case 2:  That sum is slightly bigger than RAM:  Pick one of the caches to make big enough.

Case 3:  The sum is a lot bigger than RAM:  You are stuck with lots of I/O until you get more RAM.

I suspect the Data_length and Index_length are about equal.  I would split available RAM in half -- half for key_buffer_size, the rest for data caching.

Here are 2 more ideas:


Rather than fetching the ints in a second step, have KEY(hash, int1, int2)  This means that only the BTree lookup is needed; the data will sitting in the leaf node.  With this approach, you could set key_buffer_size to 'most' of available RAM.  That SELECT won't touch the data, only the index.
Switch to InnoDB.  It's blocks are 16KB, not 1KB.  This may make things faster.  But the disk footprint will be 2-3 times as much.  Again, use the 3-column index, but shrink key_buffer_size to 20M and raise innodb_buffer_pool_size to 70% of RAM.


Other Notes:


"Sending data" does not tell you anything.  Profiling, in general, is useless.
SSDs run a lot faster than HDDs.
You appear to be I/O bound.
Whether you are I/O-bound or not, the total query time is roughly proportional to the number of hashes being looked up.  (This can be deduced from my dissection.)
MEMORY is not likely to be significantly faster or slower than MyISAM.  And if your data needs to persist, then there is a hassle because MEMORY is volatile.
I predict that compression is useless since you have only 6 bytes to compress.  (And the hash, itself, is probably not compressible.)
If your provider is limiting the IOPs, that is a problem.  If your index is fully cached (and not so big that it is unnecessarily eating RAM), then the IOPs are fetches of data blocks.  The 3-byte key would be about 70% larger; will a large enough key_buffer fit in RAM?  If so, that approach might be optimal.