Different reactions of clean and rusty iron nails in vinegar (acetic acid)
In your first experiment, the rust ($\ce{Fe2O3. xH2O}$) probably reacted with vinegar to form Iron (III) acetate which makes the solution reddish in color.

In your second experiment, the iron on the clean nail surface would have reacted with vinegar to form Iron (II) acetate. There are two main reasons why the solution might appear to have no change:


Usually, Iron (II) acetate is prepared using concentrated acetic acid and scrap iron (or ferrous oxide/hydroxide). Since vinegar is extremely dilute (around 5-8% acetic acid solution in water), the reaction would be very slow and the conc. of Iron (II) acetate formed will be very low.
Iron (II) acetate happens to be very soluble in water and has a very faint light green color in solution which is not easily detectable in lower concentrations.


Then again, when you added $\ce{H2O2}$ to the second solution, you mentioned that a reddish color appeared. This shows that the dissolved $\ce{Fe^2+}$ ions are oxidized to $\ce{Fe^3+}$. The $\ce{Fe^3+}$ formed will react with the acetate ions to form red Iron (III) acetate.