Building a stream-cipher out of a hash function? I've already read this:
Is it feasible to build a stream cipher from a cryptographic hash function?

However, my proposed construction differs…

Suppose the hash generates N bits. These bits are split into two parts:


$S$ bits are kept secret,  
$X$ bits are used for the encryption, xor'ing the bytes the usual way
$S + X = N$
$H_i^S$ is the $S$ bits of the current hash $H_i$, in iteration $i$
$H_i^X$ is the $X$ bits of the same hash value
$P_i^X$ is $X$ bits of buffered plaintext in iteration $i$
$C_i^X$ is $X$ bits of ciphertext in iteration $i$


The algorithm:


generate a random key $K$ that will be shared between the peers  
$i=0$, generate $H_0 = hash(K)$  
$C_i^X = P_i^X \oplus H_i^X$
$H_{i+1} = hash(H_i^S|P_i^X)$  
$i=i+1$, goto 3.


Analysis: 

Even if the attacker has the plaintext and thus can retrieve $H_0^X$, say in the first round, since $H_0^S$ is unknown, he will not be able to calculate $H_1$. If $S$ is sufficiently large, guessing/brute forcing will be unfeasible. On the receiver side, if the stream was tampered, the bytes decoded in that round will have the same bit errors, but then in the next round, the hash's avalanche effect will kick in, making decoding bytes in the next rounds impossible.

Note: 

The bytes used for xor'ing could come directly from the hash's state (keccak comes to mind), and the original bytes could go there, too, performing the specific bit mixing operations of the hash after each cipher round.

Eg: 

For SHA256, 128 bits could be used for $S$, and 128 bits for $X$. Or, 64 bits for $S$ and 192 bits for $X$. The later would result in less processing per byte, with somewhat less security.

What is your take on this?
As I understand the construction (and it isn't well written), it looks weak; it is likely to leak information given real plaintexts.

The reason that I say it isn't well written is that it's not stated what you use to exclusive-or the plaintext bytes to generate the ciphertext-bytes; is it the hash you generated in step 2, or is it the "most recent hash" (the one in step 2 for the first iteration, or the one is step 4 for the next -- and step 5 really should be "goto step 3").

Assuming that you meant the latter, well, this appears to be 'Plaintext Feedback Mode':


In step 3, "the original bytes are buffered in $X'$"; that is, $X'$ is set to a segment of plaintext bytes.
In step 4, you construct the $H_1 = hash(S | X' )$, where $S$ is a secret key (constant for a specific stream), and $X'$ are plaintext bytes you stored in step 3.
In step 3, you compute $Ciphertext = Plaintext \oplus Trim(H_1)$. where $Trim$ is the first $X$ bits of $H_1$


The problem here is that real plaintext often have repeats; it is not unusual to see occurances of the exact same 16 characters repeated -- this is precisely why ECB mode is considered insecure.  However, when this happens for your construction, then $X'$ will be repeated, and so will $H_1$; the 16 bytes immediately following that will have the exact same exclusive-or (and this can be detected and exploited).

The obvious fix would be to change this into Ciphertext Feedback Mode; all you would need to do is set $X'$ to the ciphertext bytes, not the original plaintext bytes.

This still isn't a great encryption method (it's slow, and needs an integrity check to be secure); however if all you have is a hash function, it could be made to work.