Leaking fluid from hydraulic brake caliper
Are you 100% sure that it is leaking from the nipple?
There are other ways to leak oil:

From a badly crushed olive in the hose connection
Cracked piston
Damaged piston seal

And other issue could be &quot;pad glazing&quot;. This has similar symtoms as getting oil on the pads.
If you are 100% sure that the bleed nipple is the issue. You can just replace and bleed the brakes.
You can't just get a new dust cap though. You would have to get a new bleed nipple. The dust cap has nothing to do with making sure the system doesn't leak. It's purpose is to keep dust out so when you bleed the system, the dust doesn't contaminate by mixing dust in with the oil. Adding a dust cap, won't stop a leak. The pressure from braking will still push the oil out.