Broken underground ground wire
You lost your neutral

And this is an emergency.  You need to fix this before using the power there. 

The safety ground isn't making a difference because that is neutral's job.  Your neutral is completely gone.  


  How do I locate the break?


Now, unless you've been trenching, 99% of the time the problem is at a termination, which is probably just 2 or 4 places at most.  I'm guessing you did not see it because you've been checking ground, not neutral.  And depending on how much "redneck engineering" you've been doing, it might have been installed incorrectly in the first place. 

If you have any ability to measure AC amps, another thing you can test for is current flow on the safety ground, e.g. out to the grounding rods.  There should never be any.  If you see current flow on the ground, that means neutral is not doing its job.  

You never, ever, ever use earth as a current return. 


  Why does the ground to soil rod not taking up the task?


Same reason they go to all the trouble to mine copper, instead of just using dirt. Dirt is a very poor conductor.  The only time dirt ever works as a conductor is with extremely low current (thus extremely high voltages), such as Australian rural distribution or a hack done in the third world with microwave oven transformers.

This work is incorrect by modern standards. A ground wire should have been trenched along with the conductors if installed in the past 20+ years.  A ground wire can be retrofitted and does not need to follow the same route. 


  My house is 325 ft away from the barn, my redneck engineering says to go from my shop earth ground rod to my house earth ground with bare copper. and save ripping up a paved driveway.


I'd run it by the inspector, but in principle, that is compatible with the gist of the retrofit ground rules.

Neutral is not safety ground

And by that I mean ground in the mains electrical sense, not GND as "common" in the electronics sense.  If you're an EE and trained to sink all current to GND/Vss, that thing is what we call neutral. In mains power, you sink current to neutral (or the other hot). You never sink current to ground. Ever.  It is never a current return; current never flows on ground except during fault conditions. 

Electricity doesn't want to go back to ground; it wants to go back to source. The difference is neutral goes back to the transformer (source for human-made power), and ground goes to the earth (return for naturally made power: lightning and ESD).

A lot of people conflate the two, on the logic that they're tied together at the service panel. That should only be one place in your whole service.  The job of the neutral-ground equipotential bond is to a) prevent hots and neutral from floating at thousands of volts (Say due to capacitive coupling inside a supply transformer)... and b) allow fault current moving on ground an easy path back to neutral, so the fault current can flow enough current to overcurrent-trip the breaker.   (otherwise fault current flows indefinitely, and simply energizes your grounds). 

One option: rewire it as a main service panel

Now, in the old days, they allowed groundless 3-wire feeders to subpanels in outbuildings.  I gather your installation is that old.  That's dangerous generally; however with such very long distances there is some sense to treating the line to this outbuilding like it's a Service.  Service lines (e.g. your pole drop from the Power Co.) do not carry grounds.  Talk to your local inspector about giving you waiver for that... if your very old installation isn't grandfathered.  (if it was built to code at the time, you don't need to modernize it). 

If you have your grandfathering or waiver, you treat it just like a service and the panel like a main panel:  2 local ground rods at the building (or an Ufer ground or 1 rod that passes the expensive test), which you need anyway, because it's an outbuilding.  Then install the one neutral-ground equipotential bond there in the "main" panel.  

Another option: Go 240V with a transformer

If you are not grandfathered and your inspector won't give a waiver, you will need to wire a ground wire from the main panel. Given the cost of trenching, I would say you are better off going with a transformer, which has 2 effects: 


It only needs 2 wires, freeing up your third wire to be used as safety ground. 
It turns this barn panel into a separately derived service which means you do indeed bond neutral and ground.  


I mention this because most people spec their wiring runs for 3% drop, and #4 aluminum at 500' only gives you 19A @ 3% drop.  So I'm guessing you only intend lights and odd plugs/sockets (dual 20A).  Well, that's only 4800 VA, and 5k VA transformers are $100 on Craigslist. 

In a transformer setup,  the 3 wires are  tasked to 2 hots and 1 safety ground.  No neutral. The pole service feeds the 2 hots at 240V via a 20A breaker.  At the barn, the 2 hots go into the primary of the 5 KVA transformer.  The service panel is then served by the secondary's L1, L2 and neutral.  Now you have 20A @ 240V, or dual 20A @ 120V. 

If you want more, you could go with a >=7.2 KVA transformer and run 30A that same way... at 4.8% voltage drop, which is acceptable. But the transformer will be more money.  

You could do the same with a 10 KVA transformer, running 40A, but voltage drop would be 6.5% at full current, which is really, really pushing it. 

If you want much more, you can get 25 KVA transformers at both ends back to back (primary to primary both jumpered for 480V).  Then backfeed the transformer at the pole, so it's a 240V-240V pair of transformers with 480V for transmission. The wire would run 480V@50A for only 4.1% voltage drop, perfectly reasonable!  That will give you 100A@240V at the garage.  So if you want a Tesla 80A charger, you're all set.  

If you want even yet more, you can do ditto-ditto- dual 240V/600V transformers at hard wire limit of 65A@600V (39 KVA) at 4.2% drop. That will deliver 163A @ 240V to the shop, though realistically, 175A because you'd need to round up to the next available breaker. 

All of the above is a no-trenching option.