Why is my main water supply shut off valve leaking even while closed?
I think what is leaking is the drain. Does it look like the object sticking out to the side in the below picture?  If so, this is used to drain water from the house side of the supply when the valve is off.

It sounds like you over tightened it in the first place which started the leaking.  If you've broken only the cap, you might be able to find a replacement.  If the stem and threads are damaged, long-term you will need to replace the entire valve.  As a short-term fix, you just need to block it some how.  Marine epoxy is handy for emergency plumbing repairs.