Are having TRIGGERs for every upvote/downvote detrimental in terms of performance?
This is called an aggregate table. This is a valid strategy.

You could also defer the calculation of user reputation if you are not bothered with real-time updates. Some sites do DB maintenance during the night, rebuild indexes etc. Since we are talking about user reputation, it's no big deal if the current stats are only reflected the day after. I don't think people will even notice. 

User_Reputation could be a field in your table Users. It is simply an integer value that is incremented or decremented when an upvote or downvote is recorded. If you are writing more than one records, then wrap it up in a transaction:


Add a record in Voting_Records
Update table users: increment/decrement the value of User_Reputation


It doesn't have to be a trigger, it could be a standalone SQL statement or a block of code in a stored procedure, as long as there is a transaction to ensure consistency and as mentioned above, you have to consider concurrent updates.

You can even make the DB read-only for a short while when maintenance is in progress, I think that's what Stack Exchange does (sometimes).