Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning
LLM;HRL
Yes
No ethics review needed.
I have outlined all my concerns in the weaknesses section.
1. This research addresses a key problem in LLM agents, using LLMs as intrinsic reward generators to enhance RL algorithm's sample efficiency.

2. The proposed algorithm is straightforward, with clear writing and experiments supporting the method's claims.

3. Experimental results on Minigrid, NetHack, and Crafter show that the method outperforms LLMxHRL and LLM4Teach in performance and token efficiency.
The paper proposes a two-stage algorithm aimed at enhancing exploration and reducing token usage. In the first stage, the RL agent imitates the policy generated by the LLM to improve its exploration capabilities. This phase leverages the high-level guidance from the LLM, helping the agent navigate the environment more efficiently. In the second stage, a vanilla PPO approach is applied to fine-tune the policy. The paper compares this two-stage algorithm with existing methods like LLMxHRL and LLM4Teach across several environments, including Minigrid, NetHack, and Crafter. Results indicate that this approach not only achieves higher performance but also significantly lowers token consumption, demonstrating both efficacy and efficiency in LLM-guided RL tasks.
1. The reduction in token use may be due to early stopping: Table 3 shows a 10% pre-training percentage, and Figure 8’s last row indicates that success rates for 3 of 4 tasks saturate after 10% of training, suggesting the two-stage algorithm may reduce to a one stage algorithm. Additionally, the paper’s claim of a 90-95% reduction in token use likely stems from the fact that LLMs only generate guidance in the first phase. If the algorithm doesn't improve in the second phase, the improvement in token efficiency is less compelling. I recommend plotting the training curve and marking the transition to the second phase on the curve to highlight the effectiveness of the two-stage method.

2. ActionNet may introduce an unfair comparison: The paper uses ActionNet to translate options into low-level actions via a pre-defined mapping. It’s unclear if baseline algorithms also use ActionNet; if they don’t, this could create an unfair advantage, which should be addressed in the experimental section.
Yes
No ethics review needed.
Please see the weakness part.
It's an interesting and novel idea to convert the common knowledge of LLMs into options to guide the policy learning process in tasks with sparse rewards, which is certainty an important research question for RL. The authors also provide a wide range of evaluation results.
This paper proposes a new training scheme to utilize LLMs to guide RL in tasks with sparse rewards. The evaluation results show consistent improvement over recent related works both in performance and token efficiency.
(a) The technical contribution is a little bit of limited, that is, introducing a KL-regularized pre-training phase to typical RL training processes.

(b) Extra hyperparameters are introduced, such as \alpha, \lambda_t, and the number of training iterations of Phase 1.

(c) From the ablation study results, it seems that PPO only already performed well enough. It's important to show that the new algorithm can perform significantly better in scenarios where PPO only would fail.
Yes
NA
No ethics review needed.
1. How does LLM calculate the probability of high-level action in the equation shown in Line 248?

2. What is the second KL divergence used for in equation 1? And why the first KL divergence is inversed compared to the second one?  

3. What does it mean in Lines 318-319? "For all baselines, we did not train them until they converged".
Empirical studies show that IHAC outperforms baseline methods on MiniGrid, NetHack and Crafter, in terms of sample efficiency and success rate.
This paper introduces a framework IHAC, which combines hierarchical reinforcement learning with large language models to solve complex and sparse-reward environments, where high-level actions, e.g., macro actions or skills are provided. In phrase I, IHAC first leverages LLMs to sample heuristic actions. It applies an annealing strategy to decrease the reliance on LLM progressively. while training, RL agents learned the policy and value in an imitation style. In phase II, it directly uses the standard RL algorithm to train with the learned policy and value function. Empirical studies show that IHAC outperforms baseline methods on MiniGrid, NetHack and Crafter, in terms of sample efficiency and success rate.
This paper suffers from several critical weaknesses:

1. This work has nothing to do with hierarchical RL, however, this concept seems to be the key point and contribution of the paper. Hierarchical RL usually learns both high-level planning and low-level control. However, in this work, high-level actions are already pre-defined and provided and the agent does not learn the low-level control. The setting degenerates into the most common single-layer RL, just like the common robotics setting where high-level skills are provided. Lines 172-180 also do not show the mapping from high-level action to low-level control.

2. The proposed algorithm is trivial and theoretically incorrect. In phase I, the learned value can only be applied to offline policy, since the agents also use LLM to sample actions. However, in Line 201, the authors claimed to run a standard RL algorithm like PPO. Moreover, PPO learns V(s) instead of Q(s, a), which is also incompatible with the proposed method. 

3. The paper is poorly written. Despite the incorrectly used concept of hierarchical RL, which is extremely confusing, the authors have a very limited study of works that leverage LLM to facilitate RL training. HRL and Imitation learning are not necessary to be mentioned in the related work and the LLM Agent section is quite unrelated to the topic of this paper. All the equations lack detailed explanation and analysis.

4. Due to the limited related work study, the baseline selection is also quite limited. Related work, which is not limited to GLAM[1], TWOSOME[2], SayCan[3], DigiRL[4] and ArCHer[5], are not discussed and compared. 

[1] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[2] Tan, Weihao, et al. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." The Twelfth International Conference on Learning Representations.

[3] Brohan, Anthony, et al. "Do as i can, not as i say: Grounding language in robotic affordances." Conference on robot learning. PMLR, 2023.

[4] Bai, Hao, et al. "Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning." arXiv preprint arXiv:2406.11896 (2024).

[5] Zhou, Yifei, et al. "Archer: Training language model agents via hierarchical multi-turn rl." arXiv preprint arXiv:2402.19446 (2024).


Other issues:
1. Algorithm 1 should have a caption.   

2. Line 239 see XXX is not replaced.    

3. Line 274 typo: importantbt  
4. Equation in Line 248 does not have a label.
Yes
None
No ethics review needed.
Despite the weakness, I also have the questions:
1. Can authors provide a more detailed discussion on the importance and effectiveness of using LLM policy in exploration and exploitation respectively? How such design sampling / regularization contributes to the better sample efficiency?
- The paper is well-motivated, since the LLM policy can be potentially informative, it can be used to faciliate both exploration and exploitation procedure via policy mixing and KL regularization
- Comprehensive experiment is conducted, provide a clear picture of practical performance
The IHAC framework models decision-making as a hierarchical RL problem, utilizing a two-phase approach:
- In the first phase, it uses an external LLM for imitation learning, guiding the selection of high-level options to accelerate early learning when the agent's experience is limited  (in both exploration and exploitation side)
- In the second phase, a standard RL algorithm like PPO further refines the policy

Tested on benchmarks like MiniGrid, IHAC outperforms existing methods in efficiency and performance, especially in optimizing LLM token usage.
- My main concern about this paper is it's novelty:
1. To ground LLM as a policy planner in real environment, modelling the decison problem as hierarchical RL is well-discussed in literature (e.g., https://arxiv.org/pdf/2310.10645)
2. Though this work also uses LLM to help learning a RL policy beyond simple imitation, but the key regularized based method, as pointed in the paper, is discussed in https://arxiv.org/pdf/2402.16181

Hence, it seems that the main contribution is to leverage LLM in the exploration stage. I would suggest authors to better discuss and highlight the contributions
@inproceedings{
anonymous2024leveraging,
title={Leveraging Imitation Learning and {LLM}s for Efficient Hierarchical Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6y00rooi7i},
note={under review}
}
In this paper, we introduce an innovative framework that combines Hierarchical Reinforcement Learning (HRL) with Large Language Models (LLMs) to tackle the challenges of complex, sparse-reward environments. A key contribution of our approach is the emphasis on imitation learning during the early training stages, where the LLM plays a crucial role in guiding the agent by providing high-level decision-making strategies. This early-stage imitation learning significantly accelerates the agent's understanding of task structure, reducing the time needed to adapt to new environments. By leveraging the LLM’s ability to generate abstract representations of the environment, the agent can efficiently explore potential strategies, even in tasks with high-dimensional state spaces and delayed rewards. Our method introduces a dynamic annealing strategy in action sampling, balancing the agent's reliance on the LLM’s guidance with its own learned policy as training progresses. Additionally, we implement a novel value function which incorporates the LLM’s predictions to guide decision-making while optimizing token efficiency. This approach reduces computational costs and enhances the agent’s learning process. Experimental results across three environments—MiniGrid, NetHack, and Crafter—demonstrate that our method significantly outperforms baseline HRL algorithms in terms of training speed and success rates. The imitation learning phase proves critical in enabling the agent to adapt quickly and perform efficiently, highlighting the potential of integrating LLMs into HRL for complex tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM HRL
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a371f90db2e065f0b1797da92c7ef963a8b349fa.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/409137b02f48af11880af13e8e58f7be0e9de6ce.zip
Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission