As a tutor, how should I handle anxious/avoidant adult math student?
Let me make a suggestion that may or may not target the real reason. I don't have a way to evaluate its relevance. 

I've almost never had to give up on a student, but it has happened. The reason wasn't them or me, but just time. I've had students in math who were really terrible at it and didn't really know why. Nothing seemed to work for them. 

But in exploring things, I learned that somewhere in their background some teacher (or several) just failed to do their job and let them slide by. In order to help them learn (I think it was) statistics, I would have had to take them back through elementary school math from the beginning. They had never learned to add or multiply properly.

Even worse, no one ever gave them any incentive to learn math, nor any insight into why it might be important. They never had any experience with figuring out how to translate a problem into symbolic form, much less equations. 

I had to give up, just because I didn't have the time to teach them ten or more years of things they'd missed. 

It is possible, I suppose, that someone has figured out how to short-circuit that process and bring a student up to speed relatively quickly, but at the time, I had no real idea how to do it. 

My action suggestion is to give your student a diagnostic test of some sort, even quite informal, to find out if they are just missing so many basics that the new material has no foundation on which to build. 

I'll note that in my personal case, I was neglected by one teacher and didn't learn my multiplication tables. I managed to overcome the problem and finally forced myself to learn them - just after I completed my doctorate in mathematics. Now, I can faithfully recite seven times eight = .........