Drawbacks of participating in a conference boycott
Yes, participating in such a boycott can come at a personal cost.  That is true for any boycott; the existence of this cost is what makes the boycott worthwhile in the first place.  If there was no cost at signing such a pledge, it would be much less impressive if many people sign.

However, again as with any boycott, as O. R. Mapper says, the marginal cost of participating drops if more academics participate.  Indeed, if a very large number of academics participate in a boycott, organisers may relocate or cancel the event, which brings your cost of boycotting, compared to people who chose not to boycott, to zero.