Deep Reinforcement Learning for Sequential Combinatorial Auctions
Mechanism Design;Auctions;Game Theory;Differential Economics;Reinforcement Learning;Deep Learning;Market Design;AI for Economics
Thank you for your thoughtful review!

We appreciate your suggestions and would love to know if there are specific baselines you have in mind. This way, we can potentially incorporate additional experiments and update our results accordingly. We’ll address the remaining points on weaknesses shortly. 

**Choice of Baselines:**  
We focused on PPO and SAC, as they are the state-of-the-art methods for continuous action spaces. As noted in Section 1, DQN and other Q-learning approaches are suitable only for discrete action spaces, which limits their applicability here. TRPO, while valuable, is considered a predecessor to PPO, and it tends to be less sample-efficient and slower in practice. Empirically, PPO often demonstrates superior performance, which is why we chose to prioritize it over TRPO in our results. We also tested DDPG in some settings, but it proved unstable in our experiments.
Baselines
Yes
No ethics review needed.
Isn't FTI an existing algorithm? I don't see a citation on that.
1. This work numerically solves sequential auctions with combinatorial action space. The solution is through an organic synergy between policy gradient and the auction process.
2. It scales to an action space with as many as 50 items.
3. Experiments show that it outperforms previous baselines.
This work investigates sequential combinatorial auctions, an instance of auction where the bidders arrive in a sequence and place bids on bundles (combinations) of items. Sequential combinatorial auctions is relevant due to its simplicity, its strategyproofness, and its generality. This work aims on numerically finding the optimal solution. Previously, this was achieved by formulating the problem into an MDP, and then running PPO on it. But the previous work was under the additive assumption so it did not really tackle the combinatorial nature of the problem. This obviously skipped the exponential action space brought by the bundles.

This work leverages the particular structure of auction, and proposes a gradient that uses the knowledge of the world model. This allows a first-order gradient feedback of the update. The work is inspired by fitted policy iteration (which I'm not sure why the manuscript did not cite any previous work) and the policy improvement step of which could benefit from the gradient derived.

With some additional techniques and tricks, the work is capable of hosting a decent size (at most 50 items) problem. Experiments are conducted on both additive and combinatorial settings against baselines including mechanism design methods and the previous RL approach. The proposed method seems to work well on the experiments.
1. The topic (sequential auction + combinatorial + numerical solution) is a bit limited to the specific sub-community. I'm not seeing the method/techniques to be of a general interest.
2. The experiments are conducted only on toy examples. Given the numerical nature of the work, I was expecting some real data, or even real system, experiments.
Yes
No ethics review needed.
Please see weakness.
- The paper is generally well-written and easy to follow, the motivation on solving limitations from RL on sample inefficiency and issues with convergence in large, continuous action spaces is valid
- Use of Analytical Gradients seems to be effective in enhancing sample efficiency and convergence.
- The approach demonstrates scalability to scenarios with empirical results.
This paper proposed a method on designing revenue-optimal mechanisms for sequential combinatorial auctions to allocate items to multiple agents in stages where the agents place bids on bundles of items based on their preferences using DRL with designs like analytical gradients.
-  Despite improvements, the method may still face computational challenges in extremely large-scale auctions, similar to issues noted in Pieroth et al. (2023). 
- The reliance on known valuation distributions may limit applicability in settings where such information is unavailable which limits the use cases of this method, more explanation on this would be good
- The baseline selection needs to be justified, why not also compare with more state-of-the-art algorithms?
- This approach uses a fixed order of agent visits, which can limit its optimality in situations where the order of bidding impacts the auction’s overall revenue.
- Whether and how much this work is sensitive to hyperparameters is not explained.
Yes
N/A
No ethics review needed.
1. What is the complexity of the new proposed method?
2. How to reduce the reliance on agents' valuation distributions?
1. The paper provides theoretical foundations for its policy optimization approach and demonstrates its effectiveness through extensive experiments. 
2. The paper introduces a new way to handle DRL in SCAs, particularly in combinatorial and high-dimensional settings.
The paper explores designing revenue-maximizing mechanisms in sequential combinatorial auctions (SCAs) using deep reinforcement learning (DRL). SCAs present auction items sequentially, allowing bidders to make strategic choices at each stage. Although traditional sequential auctions have theoretical guarantees, they lack mechanisms to maximize revenue in complex environments. This paper addresses these limitations by proposing a deep reinforcement learning framework that leverages analytical gradients to optimize auctions involving multiple agents and items efficiently.
* The method involves fitted policy iterations and analytical gradients. The complexity can be increased and needs to be measured.
* The framework assumes knowledge of agents' valuation distributions, which may not always be accessible or accurate in practice.
Yes
No ethics review needed.
No
This paper introduces a new approach that uses fitted policy iteration and analytical gradients for learning revenue-maximizing sequential combinatorial auctions. Using twofold manner: initially refining the value function to align with the policy function and subsequently adjusting the policy function to maximize rewards. This method also can used in continuous action spaces via gradient descent.
This paper introduces a new approach that uses fitted policy iteration and analytical gradients for learning revenue-maximizing sequential combinatorial auctions. Using twofold manner: initially refining the value function to align with the policy function and subsequently adjusting the policy function to maximize rewards. This method also can used in continuous action spaces via gradient descent.
No
@inproceedings{
anonymous2024deep,
title={Deep Reinforcement Learning for Sequential Combinatorial Auctions},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SVd9Ffcdp8},
note={under review}
}
Revenue-optimal auction design is a challenging problem with significant theoretical and practical implications. Sequential auction mechanisms, known for their simplicity and strong strategyproofness guarantees, are often limited by theoretical results that are largely existential, except for certain restrictive settings. Although traditional reinforcement learning methods such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are applicable in this domain, they struggle with computational demands and convergence issues when dealing with large and continuous action spaces. In light of this and recognizing that we can model transitions differentiable for our settings, we propose using a new reinforcement learning framework tailored for sequential combinatorial auctions that leverages first-order gradients.  Our extensive evaluations show that our approach achieves significant improvement in revenue over both analytical baselines and standard reinforcement learning algorithms. Furthermore, we scale our approach to scenarios involving up to 50 agents and 50 items, demonstrating its applicability in complex, real-world auction settings. As such, this work advances the computational tools available for auction design and contributes to bridging the gap between theoretical results and practical implementations in sequential auction design.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mechanism Design Auctions Game Theory Differential Economics Reinforcement Learning Deep Learning Market Design AI for Economics
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/76a04cf22dfcf617273a1fe12f16eadeb21122ce.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/12a315a8bd6a7212cc3f0e674f398cc57688c21f.zip
Deep Reinforcement Learning for Sequential Combinatorial Auctions
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission