RC4 encryption/ decryption with hashing I am working on RC4 encryption/ decryption algorithm based on the following:

A wants to send M to B, for that A did:

H = Hash (K||M)
C = Ek (M||H)


A sent ciphertext C to B. (I have completed until this part)

Upon receiving of ciphertext B calculate:

M||H = Dk (C)


Now, how do we obtain M as I need to verify if H' = Hash (K||M) holds.
Since you already know $K$, you just need to know the length of $H$ based on the hash algorithm you are using, which should be constant, then simply split the output of $D_k$ into the 2 parts of appropriate length.

You can then perform the keyed hash on the first part and match it against the last part, which will match if the decryption was successful.

It should be noted that your method is not aligned with best practices, such as NOT hashing using key prefixing (unless you are using Keccak or other sponge based hash), and NOT hashing the plaintext, but rather the ciphertext, as attacks are possible, especially with RC4. Also it is not recommended to use the same key for both encryption and authentication as per your example.

A better option is:

$C'$ = $E_k (M)$

$C$ = $C'$ $||$ $HMAC_{k'} (C')$

Where $C$ is the final transmitted message, and the hash is checked prior to decryption of the message, and $k'$ is a separate or derived key just for authentication.