Fuse box coming off of fuse at meter socket
You'll need to stuff a ground wire down that conduit first

The first order of business in this job is to get a ground wire down that conduit to the barn.  You'll need several feet of 8AWG bare stranded copper for this, with one end going to the combination N/G bar in the pole panel, and the other end left hanging in the barn panel for now. (You'll also want to turn the main breaker at the pole off while doing this pull.)  Don't worry about conduit fill, by the way: a 1.25" Schedule 40 PVC conduit supplies about twice as much fill as you'll need for everything here.

Then you can replace the barn panel with something modern

Once you have a ground wire in the conduit and hooked up to the main panel, you can then replace the subpanel in the barn with a modern breaker box.  I recommend a 100A, 24-space, main breaker panel as a minimum for this application; the main breaker here simply serves as a local shutoff means for power to the barn, so it can be larger than the feeder breaker without causing any issues, and slots are cheap to buy now when you're changing the panel out anyway but rather expensive to add later.  Also, make sure to fit the panel with separate ground bars if it doesn't come with them fitted from the factory, and to pull the (usually green) bonding screw from the panel's neutral assembly so that it's configured properly as a subpanel.

And the barn will need ground rods, too

While you're working on the barn, you'll need to drive a couple of 8' ground rods 8' apart and use more of that 8AWG bare copper to connect both ground rods to the barn subpanel.  This takes care of the other side of grounding the barn; the ground wire to the main panel returns wayward utility electricity back to the utility, while the ground rods return wayward natural electricity (such as lightning-induced transients or even just static charges) back to terra firma.

Torque matters!

One more thing is that you'll want to use an inch-pound torque wrench and/or torque screwdriver when making up the lug connections on breakers and loadcenters so that you tighten the screws to the labeled specification torques; this is required by 2017 NEC 110.14(D), and is a good idea anyway lest a bad connection in your electrical system slow your roll.