Building a stream-cipher out of a hash function?
As I understand the construction (and it isn't well written), it looks weak; it is likely to leak information given real plaintexts.

The reason that I say it isn't well written is that it's not stated what you use to exclusive-or the plaintext bytes to generate the ciphertext-bytes; is it the hash you generated in step 2, or is it the "most recent hash" (the one in step 2 for the first iteration, or the one is step 4 for the next -- and step 5 really should be "goto step 3").

Assuming that you meant the latter, well, this appears to be 'Plaintext Feedback Mode':


In step 3, "the original bytes are buffered in $X'$"; that is, $X'$ is set to a segment of plaintext bytes.
In step 4, you construct the $H_1 = hash(S | X' )$, where $S$ is a secret key (constant for a specific stream), and $X'$ are plaintext bytes you stored in step 3.
In step 3, you compute $Ciphertext = Plaintext \oplus Trim(H_1)$. where $Trim$ is the first $X$ bits of $H_1$


The problem here is that real plaintext often have repeats; it is not unusual to see occurances of the exact same 16 characters repeated -- this is precisely why ECB mode is considered insecure.  However, when this happens for your construction, then $X'$ will be repeated, and so will $H_1$; the 16 bytes immediately following that will have the exact same exclusive-or (and this can be detected and exploited).

The obvious fix would be to change this into Ciphertext Feedback Mode; all you would need to do is set $X'$ to the ciphertext bytes, not the original plaintext bytes.

This still isn't a great encryption method (it's slow, and needs an integrity check to be secure); however if all you have is a hash function, it could be made to work.