Using weak hash functions to construct a stream cipher I am in the awkward position of developing a secure encryption scheme that provides data-at-rest security for a very limited embedded system. The system is only guaranteed to have 4 MiB of memory (some systems have an extra 32 MiB), and they do not even have an MMU. They are running an old version of μClinux. While the resources are not that big an issue (even a cheap 8051 could run a secure cipher), the real issue is that I can only use the utilities that are present on the system. Unfortunately, there are no scripting languages other than a rather limited ash shell, so implementing a real stream cipher is out of the question. However, the md5sum utility is present, as is od which is necessary to combine the keystream with the plaintext of a binary stream.

Because implementing a stream cipher in ash shell would be too slow, I thought of using MD5, where the keystream $H$ at position $i$ is the hash of key $K$, nonce $N$, and counter $i$ concatenated. Encryption of plaintext $P$ and decryption of ciphertext $C$ is done as in any stream cipher.

\begin{align*}
  C_i &amp;= P_i \oplus \operatorname{MD5}(K \mathbin\| N \mathbin\| i) \\
  P_i &amp;= C_i \oplus \operatorname{MD5}(K \mathbin\| N \mathbin\| i)
\end{align*}

I understand that it is possible to design a stream cipher from a hash function, but I would like to know specifically if MD5 is an acceptable hash function in this particular scenario. As far as I am aware, collision attacks are irrelevant for use in a stream cipher, but I want to make sure I missed nothing. Is this scheme appropriate for my unfortunate hardware / software limitations?

I am also aware that designing a cryptographic scheme as an amateur is very bad. Unfortunately, I have little choice as I can't load my own executables onto the device, and thankfully I do not need authentication or any security properties other than data-at-rest security (i.e. it is a situation where unauthenticated AES-CTR would suffice).
As long as your encoding of $K$, $N$, and $i$ is injective (i.e., for each encoded string you can uniquely derive $K$, $N$, and $i$) and fixed-length, this is almost certainly a decent stream cipher.

It won't break any speed records, obviously!  But collision resistance is not relevant here, and I have no reason to suspect that the distribution of $x \mapsto \operatorname{MD5}(k \mathbin\Vert x)$ for fixed-length $x$ is distinguishable from uniform under a uniform choice of $k$.  Any distinguisher would probably suggest a better than generic partial preimage attack on MD5, and in spite of some papers claiming the contrary in nonsensical cost models, so far there's no evidence of preimage attacks on MD5.

That said: Consider carefully whether you also need to do authentication!  Unauthenticated encryption is seldom useful in real-world applications.  Using the tools you have, you can probably compute HMAC-MD5 too without too much trouble.  Encrypt-then-MAC with MD5-CTR and HMAC-MD5 may be goofy, but it probably makes a reasonably secure authenticated encryption scheme.

Do make sure, though, to limit yourself to ${\lll}2^{64}$ messages under any one key because the 128-bit state of HMAC-MD5 admits a birthday forger, and make sure to use at least a 256-bit key in case there is any danger of multi-target attacks.