How to determine the composition of a gas mixture after a combustion?
This is clearly a homework question. Yet, since OP has given good effort to solve the question, I'd like to give OP some insight to solve these kinds of questions, taking current question as an example.

Suppose the mixture contains $x\;\pu {g}$ of $\ce{CH4}$ and $y\;\pu {g}$ of $\ce{C2H4}$. The combustion reactions of $\ce{CH4}$ and $\ce{C2H4}$ can be given as:

$$\ce{CH4 + 2 O2 -&gt; CO2 + 2H2O} \tag{1}$$
$$\ce{C2H4 + 3 O2 -&gt; 2 CO2 + 2 H2O} \tag{2}$$

Thus, the amount of $\ce{CO2}$ produced from the reaction $(1)$ is:

$$x\,\pu{g}\,\ce{CH4} \times\frac{\pu{1 mol}\,\ce{CH4}}{\pu{16 g}\,\ce{CH4}} \times \frac{\pu{1 mol}\,\ce{CO2}}{\pu{1 mol}\,\ce{CH4}}\times \frac{\pu{44 g} \,\ce{CO2}}{\pu{1 mol}\,\ce{CO2}}=\frac{11}{4}x\,\pu{g} \,\ce{CO2}$$

And, the amount of $\ce{H2O}$ produced from the reaction $(1)$ is:

$$x\,\pu{g}\,\ce{CH4} \times\frac{\pu{1 mol}\,\ce{CH4}}{\pu{16 g}\,\ce{CH4}} \times \frac{\pu{2 mol}\,\ce{H2O}}{\pu{1 mol}\,\ce{CH4}}\times \frac{\pu{18 g} \,\ce{H2O}}{\pu{1 mol}\,\ce{H2O}}=\frac{9}{4}x\,\pu{g} \,\ce{H2O}$$

Similarly, the amount of $\ce{CO2}$ produced from the reaction $(2)$ is:

$$y\,\pu{g}\,\ce{C2H4} \times\frac{\pu{1 mol}\,\ce{C2H4}}{\pu{28 g}\,\ce{C2H4}} \times \frac{\pu{2 mol}\,\ce{CO2}}{\pu{1 mol}\,\ce{C2H4}}\times \frac{\pu{44 g} \,\ce{CO2}}{\pu{1 mol}\,\ce{CO2}}=\frac{22}{7}y\,\pu{g} \,\ce{CO2}$$

And, the amount of $\ce{H2O}$ produced from the reaction $(2)$ is:

$$y\,\pu{g}\,\ce{C2H4} \times\frac{\pu{1 mol}\,\ce{C2H4}}{\pu{28 g}\,\ce{C2H4}} \times \frac{\pu{2 mol}\,\ce{H2O}}{\pu{1 mol}\,\ce{C2H4}}\times \frac{\pu{18 g} \,\ce{H2O}}{\pu{1 mol}\,\ce{H2O}}=\frac{9}{7}y\,\pu{g} \,\ce{H2O}$$

Since the mixture produced $\pu{1.185 g}$ of $\ce{CO2}$ and $\pu{0.6 g}$ of $\ce{H2O}$, we can derive two equations with two variables, $x$ and $y$:

$$\frac{11}{4}x+\frac{22}{7}y=1.185$$
and,
$$\frac{9}{4}x+\frac{9}{7}y=0.6$$

Up to this point OP has taken correct path. Now, let's consider these two questions. These equations can be simplify as:

$$77 x + 88y=1.185\times 28$$
or 
$$ 7x + 8y=\frac{1.185\times 28}{11}=3.016 \tag {3}$$
and,
$$ 7x + 4y=\frac{0.6\times 28}{9} = 1.867 \tag {4}$$

Thus, $(3)-(4)$ gives you:

$$4y = 3.016 - 1.867 = 1.149 $$
$$\therefore \; y = \frac{1.149}{4} = \pu{0.2873 g}\,\ce{C2H4}$$

And $(4)\times 2-(3)$ gives you:

$$7x = 1.867 \times 2 - 3.016 = \pu{0.718 g}\,\ce{CH4}$$
$$\therefore \; x = \frac{0.718}{7} = \pu{0.1026 g}\,\ce{CH4}$$

Thus, it is clear that you were still in the correct path on solving the same two equations for $x$ and $y$.

According to my solution, mass percentage of $\ce{CH4}$ is:

$$\frac{0.1026}{0.1026 + 0.2873} \times 100 = 26.31\% \; (w/w)$$

And mass percentage of $\ce{C2H4}$ is:

$$\frac{0.2873}{0.1026 + 0.2873} \times 100 = 73.69\% \; (w/w)$$

Since this is not the one of the given answers, let's check for mol% like what you have done:

$$n_{\ce{CH4}}=\frac{0.1026}{16}= \pu{0.0064 mol}$$

$$n_{\ce{C2H4}}=\frac{0.2873}{28}= \pu{0.0103 mol}$$

Thus, the percentage of composition for $\ce{CH4}$ would be:

$$\frac{0.0064}{0.0064+0.0103}\times 100 = 38.32\%\: \pu{mol}$$

Similarly, the percentage of composition for $\ce{C2H4}$ would be:

$$\frac{0.0103}{0.0064+0.0103}\times 100 = 61.68\%\: \pu{mol}$$

Accordingly, the most closest answer is (5), although both given values are $\approx 2\%$ off from calculated values. It is noteworthy that OP's calculations are correct to the end.

Note: This question is not well written. That's why I have to go through two calculations to see which $\%$ is they are talking about. Also, the sum of compositions given in answer choice (4) is not $100\%$.