Cannot pickup an object twice (Use child of a second time)
After looking over the file i saw that you used one "Child Of" constraint, animated it with 0 influence until one frame before the time it needs to be activated, then activated it with 1 influence to pick it up and later used the reverse, keyed 1 influence and one frame later the 0 influence to drop the "Child Of", so it could be picked up later again.

All that was working fine, a few too many keyframes were used before the actual action, as you only need to use two keys for activating and deactivating. If not needed, i would suggest to remove unneeded keys though.
Those two times would have been enough, but that did not cause the problem.

What is important though, is that you can not use the same "Child Of" Constraint for a second time without it keeping the relative position after it was used once. (all my tests either caused the later or previous positions relative to the parent to be messed up. Sure you can use the Set Inverse or Clear Inverse buttons, but they pretty much do mess up any previous usage or an already keyed later usage.

Solution:
Use one "Child Of" constraint per pick up -> manipulating -> drop phase.

Here a little visual helper:

I extended the animation a bit, to showcase a second pickup.

Also here the file to examine, as it may help to understand looking at the details:


While i believe it is better to know how it works, i do not want to withhold the fact that there is an addon called Dynamic Parent, that kinda makes it easier to attach items to different parents. Beware though that if you use it, it can become very fast messy with many "Child Of" constraints. So understanding what happens first helps a lot.

Happy Blending