Signature generation using hash function I want to generate a valid signature for a message using the below algorithms:

Hash function: $H(x) = (x+3) \bmod 10$
(I am unable to understand the given parameter for hash function and how am I supposed to use it in hashing a message.)

Encryption: RSA with the following parameters


Private key: 11
Public key: 5
Modulus: 14
(I had tried but failed in using these parameters in RSA encryption. I want to know the possible way or hint to be able to use these exponents in encrypting the hash of a message.)
The private key is actually the private exponent. Signature generation is modular exponentiation using the private exponent and - of course - the modulus. So you first hash a value for x using the hash function, giving you another value, say $h$. Now you perform the modular exponentiation on it and return the result. And that's all there is to it.
Note that you should not perform modular exponentiation by first perform exponentiation and then take the modulus: you can use the modulus in the intermediate steps (e.g. multiplication).
Do not think of signature generation as encryption with the private key. It only is similar in for textbook RSA, but otherwise that kind of comparison only hampers understanding of signature generation.