How to prove security of a randomized encryption algorithm? I have designed an random encryption in public key setting. I am convinced about its security guarantee but I need a formal way to prove its robustness. How can I prove the security of this random-public-key-encryption algorithm? Any proof method such as induction-method, proof-by-contradiction-method or anything else?

I am trying to publish this algorithm in a journal paper. This is why I need a formal proof.
What you want to do is the right way. However it's not an easy one.

What you really want to do is to show the following implication:
$$
\newcommand{\hard}{\operatorname{hard}}
\newcommand{\assumption}{\text{assumption}}
\newcommand{\secure}{\operatorname{secure}}
\hard(\assumption_1)\land\ldots\land\hard(\assumption_n)
=\bigwedge_{i=1}^n\hard(\text{assumption}_i)\\
\implies\secure(\text{cryptosystem})$$

for a (hopefully) small number of assumptions which are already well-studied.

The usual intuition for such proofs is a proof-by-reduction, which is a logical contraposition:
$$\neg\secure(\text{cryptosystem})\implies\bigvee_{i=1}^n\neg\hard(\assumption_i)$$
Meaning that you would need to show that breaking the security of your cryptosystem implies that any of your assumptions is not hard.

Of course you first need to find out what "$\hard(\assumption_i)$" and "$\secure(\text{cryptosystem})$" actually mean. The latter is a (preferably) strong security definition, most likely from this list. The former very much depends on the actual assumption at hand and you need to look at the actual, precise definition of said assumption. For example this blog post covers RSA and this one covers the typical DH assumptions.

Now the final and actually hardest part: How to actually make the proof. There's no simple answer to this question. For example you could give game-hopping a look as well as simulation based proofs. You could also look whether proof / verification tools can help you along the way, such as described by Biv here and here. There isn't much more to say on this, except for: Look at the references given, work your way from there through the references they give, continue going through the references until you have an understanding of these proofs and are able to carry out the one you need. It will probably be a hard and / or long journey, but it will pay off.