Vote-switching optimization
We cant review the code if you don't provide it, telling us that it is a bit long will not help. You should provide the full code especially since it is likely that refactoring will be useful between the different cases.

Regardless, here are some observations from what you did provide:


In essence, whatever the user does, you will always call setValue(), once you realize this you can simplify the first if statement

if ($("#up_vote").hasClass("voted") || $("#down_vote").hasClass("voted")) {
  if ($(this).hasClass("voted")) {
    action = "remove_vote";
  } else {
    action = "switch_vote";
  }
} else {
  action = "add_vote";
}
setValue();

And, since you only call setValue() once, you can even take it out of it's function and just put it after the first if statement.
You might even consider replacing the if statement inside setValue with a ternary statement: value = e.target.id === "up_vote" ? 1 : -1;
When posting data, you should use "POST", not "GET"
I feel votes.php should give back the vote counts. Just adding 1 or removing 1 from the last vote count is cheating, and I think users can tell.
Finally, and this is a matter of taste, I would not send +1 or -1 back, I would send the value of e.target.id back, and let the back-end do the conversion.