How to use DQN to handle an imperfect but complete information game?
Would it be helpful to use a LSTM and reduce the input state?


I'd bet, no. LSTM is more complicated and harder to learn, while the input is 4 * 9 * 36 bits is still rather limited.

However, you may want to aggregate the information somehow, e.g., add additional bits informing about what cards were already played (no matter when). This information is redundant, but by providing it, you may save the network quite some learning.

At the same time, you may want to use symmetries (all colors but trumps are equivalent and the therefore the weights should be the same).


  How to handle invalid moves?


That's simple: There are no invalid moves. The network provides 36 outputs of how much it wants to play a given card. You simply take the one valid card having the greatest output value. You don't try to make the network learn what moves are valid as this is neither needed nor helpful.


  Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)


I can't tell. But it should matter at the moment. First make you network clearly beat the random players, then you can look for more. Or start with Self-Play, as you want probably have both for comparison.


  Or is this the whole approach absolute nonsense?


I don't think so, but ... (see below)


  I design the input state as a vector (one hot encoded) with 36 "bits" for every player


This doesn't sound good. Every player has 9 of 36 cards and so should be the encoding. A player doesn't know the cards of other players.


  The reward is +1 for winning, -1 for losing,


In most card games I know, it matters by how much you win (unlike e.g., in Go). Even when it doesn't matter, using this information at the early learning stages is IMHO useful.


  -0.1 for a invalid action and


Drop the invalid action. Just transform anything the network produces to a valid action, add no penalty (as written above).


  ... 0 for an action which doesn't lead to a terminal state


All action but the last one lead to a non-terminal state. You can use some Temporal Difference Learning or use the fact that the game has a small fixed number of moves and reward/punish all actions takes in a the whole game.