State machine for upvoting/downvoting
Both portions of code could be simplified if you use numbers, instead of words, to represent upvotes and downvotes. So I would change your calls to sendVote to look like this:

sendVote(this.parentNode, 1);
sendVote(this.parentNode, -1);


When modifying elements with javascript, I suggest using element.style.x (or element.style.cssText for multiple attributes) instead of classes, where possible. This is more efficient (browser doesn't have to re-test css selectors for the rest of the page), simpler (don't have to polyfill classList), and it means putting javascript-only css in js files, which means they aren't loaded in browsers that don't support javascript and therefore don't want them anyway. But the main reason is that is allows you to shorten your code to this:

var voteCSS = {
    '1': 'background:blue;',
    '0': 'background:grey;',
    '-1': 'background:red;'
};

function sendVote(element, newVote) {
    var countEl = element.querySelector('.m p'),
        prevVote = +element.dataset.prevVote || 0;

    if (newVote == prevVote)
        newVote = 0;
    countEl.innerHTML = +countEl.innerHTML - prevVote + newVote;
    element.dataset.prevVote = newVote;
    element.style.cssText = voteCSS[newVote];
    // ajax here to /votes/submit?id=x&amp;direction=`newVote`
}


Alternatively, if you want to keep all your CSS in one place, you can make sure that "upvoted" or "downvoted" is the only class the element has, then just use .className. You could still apply default styling by selecting it from above, something like this: .votecontainer div {}.

ps: notice that I didn't use jQuery even once. Try to solve your problems without jQuery before you let yourself use it - a lot of the time you'll find that your solutions are simpler and more efficient without it.