Can I ground junction box directly to panel?
If that wire nut comes loose it sounds like you could end up with not only a HOT BOX but perhaps HOT CONDUIT also.  

I trust you can imagine the possibilities of misfortune that can result from that.  

It is well worth whatever it takes to prevent that from happening.  What you want is to have a breaker trip instead.

Splicing ground wire is safe when done correctly.  But if your length simply can't reach or if the splice can't be made to fit inside the conduit, then the only option would be a new length of ground wire.

Call it cheap insurance.