Am I disadvantaged by declining authorship?
Declining authorship is quite common among my collaborators. There are cases when you disagree with the paper and you do not want to assume responsibility for what is written there. Declining authorship is the right thing to do. I assume that in your case, you agree with your coauthors. In this case, you did the right thing to decline authorship when you feel that your contribution is too small, even if this could be a disadvantage to you.

I declined authorship twice so far (and I don't have that many publications). In both cases, I felt that my contribution was too small to warrant authorship. In the second case, I didn't agree fully with the paper, but I was involved in the experiments leading to the paper. In both cases, however, my coauthor(s) insisted of including me. In this case, I tried to contribute more in ways I could, such as by reading the draft thoroughly and giving constructive feedback that would improve the quality of the paper.

There are different ways your potential employer assess your publications. Some base it on the total number of papers and citations you authored; in this case you could be at a disadvantage if you have fewer papers than you would have by not declining. Some do not care about papers you are not a first author of (in my field the first author is assumed to have performed most of the work); in this case it does not matter if you have more papers or not.

However, in a job interview, your potential employer might ask you about your specific role in a paper, especially if the paper turns out to be highly cited (I was asked this question before). If you are not able to give a satisfactory answer, this could be a disadvantage instead of an advantage to you.