OpenAI Gym: Multiple actions in one step
What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.
I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.
action = {}
for agent_id, agent_obs in obs.items():
    # get the policy of the current agent
    policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
    # use the policy and the agent's observations to compute its next action
    action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)

My environment's step(self, action) function expects action to be such a dict and knows how to handle it.
It applies all actions in the dict before calculating each agent's reward and progressing time in the environment.
So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.
For the multi-agent RL approach, I used PPO and the Ray RLlib framework.