What would be the most effective self-learning algorithm for a 7 player social deduction game?
It is very likely that you want an algorithm like Counterfactual Minimax Regret. This algorithm has several variants, but they differ mostly in their efficiency.

CFR is the algorithm that was used to solve 2-Player Poker, although the solution comes from one of its more advanced versions. The algorithm is highly applicable to other games of incomplete information and games with roles, like Secret Hitler. Essentially, it learns how to act so that the actions it takes leak as little information to the other players as possible about its role (or hand of cards), while also learning about the roles of other players from their actions, and maximizing its chances of winning the game.