Research assistant: I am responsible for scheduling subjects, how do I deal with attrition in my PI's longitudinal study?
Document. As scientists, your PI and supervisor should hopefully be receptive to data. Use a time-tracking app (maybe one of these push-button mobile apps?) to track and make statistics on how long it takes you to track down addresses per person, how many people you contacted, the call answer rate, the success rate in bringing them in, etc. Then, if anyone is concerned about how you use your time, you have hard facts to show them exactly why the productivity is what it is.

If you have another meeting with your PI after that, emphasize what else you could be doing with that time that is not getting done. Something like "You've asked me to prioritize subject scheduling. I'm currently using X% of time on tracking them down. This means I only have Y% remaining for these other tasks. (If you also have some metrics on how efficient you are at those, you could emphasize your productivity: ...but despite the limited time, I managed to get Z items done.) Should we keep going this way, or should we make any adjustments in how we use my work time?"

Or, if you really want to stop doing so much of this, not just worry about your higher-ups understanding why productivity is what it is, you can also instead say "It's really burning me out spending X hours on the phone or tracking down addresses each day. Can we get someone else to share that work so that I can spend more time on project B instead?" You'll know your context, but maybe there's a way to bring a student in for some part-time work calling people.