openai gym - what is an agent I can use with a multi-discrete action space?
OpenAI Baselines - or for me even better, Stable Baselines - has many model options which can handle MultiDicrete Action and/or Observation spaces.  Building a custom gym environment is also quite straightforward.