How can I calculate the SHA-256 "midstate"? Recently I've been trying to implement some Bitcoin-related code, and I've stumbled upon a weird concept, a SHA-256 "midstate". Some explanation is given here.

The general concept is that Bitcoin relies on performing SHA hashing of a 128 byte data chunk many times, but only the second half of that chunk changes. This is why the concept of "midstate" is used. As I understand it, in order to perform SHA on 128 bytes of data, one needs to divide the hashing into operations on first and second 64 bytes of the data. As results of first hashing are constant, one would save them as this "midstate", and before hashing the changed second half of bytes, one would restore the "midstate" instead of calculating it again.

Can anyone explain how would I calculate this "midstate" having a library that calculates a whole SHA-256 hash, or alternatively which libraries support calculating of something like that?
SHA-256 uses an internal compression function $f$ which takes two inputs, of size 512 and 256 bits respectively, and outputs 256 bits. Hashing works like this:


Input message $M$ is first padded by appending between 129 and 640 bits (inclusive), resulting into a padded message $M&#39;$ whose length (in bits) is a multiple of 512.
$M&#39;$ is split into $n$ 512-bit blocks $M_1$, $M_2$,... $M_n$ (each block has length 64 bytes).
Set $X_0$ (a 256-bit value) to a conventional initial value (which is specified in the SHA-256 standard, section 5.3.3).
Process each block in due sequence, by computing $X_i = f(M_i, X_{i-1})$ for all $i$ from $1$ to $n$.
The hash value is $X_n$.


The $X_i$ values can be seen as the successive contents of a state variable $X$ (indeed, when you have computed $X_i$, you can discard $X_{i-1}$ since that value will not be used thereafter).

Your "midstate" is $X_1$: that's the contents of the "state" after having processed the first block. Since all your 128-byte messages begin with the exact same 64-byte header, all the hash computations formally begin by processing the same block $M_1$, with the same $X_0$ (the conventional initial value), resulting in the same $X_1 = f(M_1, X_0)$. You would like to "restart" each computation from that value $X_1$ directly, to avoid recomputing it again and again.

Doing that over an existing library implementing SHA-256 may or may not be easy, depending on what facilities that library offers. For a basic SHA-256 library, there are two possible issues:


The library may refuse to output $X_1$, because it would insist on computing hash values over a padded message only; you want to have the output of $f$ over a given $M_1$ block without padding.
The library may refuse to start a new computation starting with the $X_1$ you provide instead of the conventional initial value $X_0$.


Some libraries offer a bit more. For instance, consider sphlib. With that library, there are two ways to achieve what you seek:


The implementation works over a context structure (of type sph_sha256_context) which represents the current state. You can thus begin a computation by processing the header, and then clone the context to make many hash computations which start from that exact point. It would look like this:




sph_sha256_context sc;

sph_sha256_init(&amp;sc);
sph_sha256(&amp;sc, header, 64);
for (/* all 128-byte messages */) {
        sph_sha256_context sc2;
        unsigned char second_half[64];
        unsigned char out[32];

        /* set second_half[] to the second half of the 128-byte message */
        sc2 = sc;
        sph_sha256(&amp;sc2, second_half, 64);
        sph_sha256_close(&amp;sc2, out);
        /* SHA-256 output is in out[] */
}



sphlib offers an sph_sha256_comp() function which implements exactly the compression function $f$. You can use it to compute SHA-256 "manually", block by block, starting from any state value $X$ you wish. You would have to take care of encoding issues (SHA-256 is big-endian throughout, use sph_dec32be() and sph_enc32be() to do that properly and portably) and padding. If all your messages have length exactly 128 bytes, then padding is always a full 64-byte block, consisting in a byte of value 0x80, followed by 61 bytes of value 0x00, and then two bytes of value 0x04 0x00. Expressed in 16 32-bit words, first word has numerical value 0x80000000, followed by 14 words of value zero, then a final word of value 0x00000400.


For heavy hash crunching (I understand that's the point), your best bet is still probably to extract the inner loop from an opensource implementation, and integrate it directly in your code, so as to avoid any overhead from the library API. You also probably want to avoid heavy encoding/decoding, and work over 32-bit words directly (while SHA-256 takes as input a bit sequence, it soon converts it to a sequence of 32-bit words and works over that). I invite you to do your own SHA-256 implementation, working from the standard: it is not hard, that standard is reasonably clear, and it will grant you enough knowledge on how SHA-256 works to optimize your code (even if you end up reusing parts of some other implementation).