How to get overall rank based on data regardless of scope (all items, user specific, etc.)?
This query uses a variable for ranking:

SELECT t.id,
    t.user_id,
    t.slug,
    t.votes,
    @rank := @rank + 1 AS rank
    FROM (
    SELECT 
        s.id,
        s.user_id,
        s.slug,
        (
            SELECT 
                COUNT(sv.id)
            FROM
                server_votes AS sv
            WHERE
                DATE(sv.created_at) &gt; '2016-01-24 19:11:15'
            AND 
                sv.server_id = s.id
        ) AS votes
    FROM
        servers AS s
    LEFT JOIN
        server_votes AS sv 
    ON 
        sv.server_id = s.id
    WHERE 
        s.game_id = 1
    GROUP BY s.id
) t, (SELECT @rank := 0) r
ORDER BY 
    t.votes DESC,
    t.id ASC


See SQL Fiddle.

Output:

id  user_id slug    votes   rank
4   1       Server4 3       1
2   1       Server2 2       2
1   1       Server1 0       3
3   2       Server3 0       4
5   2       Server5 0       5


Once the above query is put in a subquery with a WHERE clause, it outputs data for a specific user:

SELECT 
    r.id,
    r.user_id,
    r.slug,
    r.votes,
    r.rank
FROM (
   ...
) r
WHERE r.user_id = 2;


It will output id 3 and 5. 

See SQL Fiddle