Assessing the Interpretability of Programmatic Policies using Large Language Models
Programmatic Policies;Interpretability;Program Synthesis
Yes
No ethics review needed.
1. What specific challenges in programmatic policy interpretability justify the need for a dedicated metric like LINT?

2. Are there theoretical guarantees that the LINT score reliably reflects policy interpretability?

3. How does LINT compare to other interpretability metrics, and have any baseline comparisons been considered?
- The proposed solution for quantifying the interpretability of programmatic policies is innovative but depends heavily on LLMs, which are inherently stochastic. This reliance introduces potential variability in results, but the authors address this concern by implementing measures such as multi-evaluation runs and verification steps to mitigate inaccuracies and enhance reliability.

- The authors validate the effectiveness of LINT scores across two distinct domains, MicroRTS and Karel the Robot, demonstrating a strong correlation between LINT scores and human interpretability assessments.
The author proposes a novel metric, LLM-based INTerpretability (LINT) score, which uses large-language models (LLMs) to quantify the interpretability of programmatic policies. The method leverages three LLMs to act as explainer (to generate natural language description), reconstructor (to generate policy based on language description from explainer) and verifier (to check if the explainer has not provided any unwanted information about policy). The authors validated the usability of LINT across two domains, MicroRTS and Karel the robot, demonstrating its correlation with human interpretability assessments.
**Major:**
- **Importance of proposed method:** Currently, it is not much clear why there is a need for interpretability score for programmatic policies. From the examples shown in paper (table 1 and 2), these policies are in itself interpretable and might not warrant the need for the proposed score. Have there been any study to support the cause, authors should highlight that in introduction.
- **No theoretical guarantee:** Although the approach proposed by authors is interesting, they don't provide any theoretical guarantee as the scores they achieve is in actuality explains the policy.
- **Lack of baseline evaluation:** Although the authors conducted human study to support the method, they lack comparisons with other baseline methods. Just by looking at results from table 3 and 4 it is hard to conclude if the proposed method is an improvement or not.

**Minor:**
- Adding a brief introduction about programmatic policies in the problem definition will help in better understanding of the paper.
- The authors have described about the limitations of their approach at multiple places (lines 163 - 172 and section 4). Maybe these can be moved towards the end as a Limitations section.
- **Bad representation of data:** The letter superscript presented in table 4 and 5 for column I-score and V-score is really confusing. There should be a better way to represent that data.
- **Typo (line 300):** IFor &#8594; For
Yes
No ethics review needed.
* When I run LINT and receive scores for Action and Return, how should I interpret them? The Action score is somewhat straightforward since it ranges from 0 to 1 although there are challenges as described in the Weaknesses section. However, in grid world scenarios, multiple aliased actions for policies $\pi$ and $\pi^{'}$ can unfairly penalize the Action score. Additionally, if policy $\pi_1$ has an Action score of 0.8, $\pi_2$ has 0.7, and $\pi_3$ has 0.6, can we confidently say that $\pi_2$ is better than $\pi_3$ by the same margin that $\pi_1$ is better than $\pi_2$? Being able to make such claims would be valuable, especially when balancing interpretability against performance.
* It would also be beneficial to include a brief description of the Karel and MicroRTS environments for users who may not be familiar with them.
* The authors have pinpointed a significant challenge in the realm of programmatic policies for sequential decision-making: evaluating the interpretability of these policies. They propose a straightforward yet effective algorithm that addresses this issue, with their methodology and experimental design presented clearly. The prompts used for the LLMs and the specifics of the user studies are well detailed.
* The design choices for LINT—including the three-model setup featuring an explainer, verifier, and reconstructor, as well as the constraints for generation—are straightforward and logical. While the selected metrics, specifically the action and return heuristics, can be debated (e.g., why not consider state visitation?), they serve as defensible and practical starting points.
* The authors demonstrate a strong correlation between the LINT score and the level of obfuscation in programs, indicating that more obfuscated programs are less interpretable and thus receive lower LINT scores. This finding is promising for utilizing LLMs in code interpretability, applicable not only to programmatic policies but to any computational program.
* The user study suggests that LINT has potential as an effective tool for assessing the interpretability of programmatic policies.
Programmatic policies for sequential decision-making problems often promise interpretability, but measuring and evaluating this promise can be challenging. User studies that assess interpretability tend to be subjective and costly. This work introduces LINT, a straightforward solution based on large language models (LLMs) to evaluate the interpretability of programmatic policies. LINT employs an LLM explainer to translate the program into natural language text, adhering to specific constraints, and then uses an LLM reconstructor to regenerate the original program from that text. If the original and reconstructed policies are similar, the original is considered interpretable. The authors demonstrate LINT’s effectiveness as an interpretability evaluator through obfuscation experiments and user studies in two domains: MicroRTS and Karel the Robot.
I have three main concerns with this paper that prevent me from giving it a higher score, which I outline below:
* The primary factor that would make LINT a compelling algorithm and metric is its correlation with user interpretability. While the authors demonstrate a positive correlation through their experiments, the metrics used to assess user interpretability are unconvincing and potentially noisy. Users viewing a programmatic policy alongside the relatively straightforward domain-specific languages (DSLs) in the benchmarks might feel they understand the program and assign a high I-score, even if their understanding is superficial. Although the V-score attempts to address this, the authors acknowledge that it is a noisy metric; designing an evaluation setup with four clear options is challenging, and users can often make educated guesses. This complicates drawing conclusions from the user study results. Additionally, the scores in the table for I-scores and V-scores suggest that obfuscated programs may have scores very close to those of more interpretable ones (as seen in Table 5).
* A second challenge in evaluating LINT is the absence of baseline comparisons. Even if LINT is the first algorithm focused on evaluating the interpretability of programmatic policies, it would be beneficial to compare it against simple baselines using code heuristics—such as the number of lines, loops, and conditional clauses—to gauge interpretability. Such comparisons would strengthen the validity of LINT, particularly in light of the noisy I-scores and V-scores. Without these baselines, it raises the question of whether simple heuristic-based methods could also effectively distinguish between obfuscated and straightforward programs.
* Lastly, the LINT score itself is difficult to interpret. For instance, in Section 6.3, a policy receives a LINT score of (0.626, 0.57). This score leaves me uncertain about the interpretability of the policy; while 0.626 doesn’t seem impressive out of 1, the action score is at least bounded between 0 and 1. The return score, however, is unnormalized and unbounded, making it very domain-specific and hard to interpret. As a user, I am left unsure of how good or bad the policy truly is. In this case, the policy appears quite interpretable to me, but I would not be able to ascertain that just from the LINT score.
Yes
No ethics review needed.
- Since programmatic policies provide explicit code with defined logic, wouldn’t they be inherently interpretable by examining their internal workings? Could you clarify why an additional interpretability metric, like LINT, is necessary when the policy’s behavior can ostensibly be understood directly from its code structure?
- Given that LLMs themselves are black boxes, what is the rationale behind using LLMs as proxies for interpretability? Wouldn't alternative methods, such as rule-based systems offer more transparency in the interpretability assessment?
- For variability in LINT scores across multiple runs? Did you consider using temperature or other settings to stabilize outputs?
- The paper presents the LINT score methodology with clarity, detailing each component the explainer, reconstructor, and verifier and its purpose within the interpretability assessment pipeline.
- Although initial results suggest areas for improvement, the study opens up valuable avenues for refining interpretability metrics, setting a constructive groundwork for further development in areas of robotics and RL.
- The use of varying obfuscation levels and structured user studies across domains demonstrates a good experimental design.
This authors propose a metric called LINT score. This metric leverages LLMs to evaluate the interpretability of programmatic policies used in sequential decision-making tasks by reconstructing a policy based on its natural language description.

The proposed method includes using one LLM describe a policy's behavior in natural language, followed by a second LLM attempting to reconstruct the policy based on this description.  The similarity between the original and reconstructed policies serves as the interpretability measure. The authors hypothesize that higher similarity indicates greater interpretability. The metric was tested in two domains: MicroRTS and Karel. The experiments involve increasing the obfuscation level of policies to simulate varying degrees of interpretability, with user studies to validate the correlation between LINT scores and human perceptions of interpretability
- Since the original policy is directly provided to the explainer, there’s a risk that the natural language description generated is highly specific and detailed, making it relatively straightforward for the reconstructor to recreate the policy. This dependency raises questions about whether the LINT score truly reflects interpretability or merely the ability of another LLM to follow a sufficiently descriptive prompt.
- The paper defines interpretability narrowly as the ability to reconstruct a policy from a natural language description, which may not capture the full human-centric aspects of interpretability.
- The LINT score is tested only on specific domains (MicroRTS and Karel the Robot), both of which are relatively contained environments with domain-specific languages. It’s unclear how well the LINT score would generalize to more complex or diverse environments.
- The method relies on the assumption that a policy that is interpretable should be reconstructable in a way that behaves similarly to the original policy. However, minor discrepancies in LLM-generated reconstructions, even if they don’t reflect actual interpretability, might impact the LINT score due to the stochastic nature of LLMs. Because LLMs are stochastic and may produce different outputs for the same prompt, achieving consistent LINT scores might require multiple runs, which complicates reproducibility.
- The paper relies solely on the LINT score for interpretability assessment without integrating other interpretability measures, such as feature importance or visualization techniques, that might provide a more comprehensive view of a policy's interpretability.
- Although the LINT score aims to be cost-effective compared to user studies, the computational expense of using multiple LLM instances (Explainer, Verifier, Reconstructor) may still be high, especially for larger-scale deployments.
- While I do feel this area of study is important, the methodology in this paper felt more like a pipeline of multiple LLMs strung together, which limits its methodological novelty.
Yes
No ethics review needed.
- As you mentioned in Section 4, the time spent pondering may affect the results. So what is the time limit for users to answer the questions?
- Given that users rate program interpretability on a scale from 1 to 5, can you instruct the LLM to answer the same questions and calculate the correlation with users' responses?
- I noticed that you included the prompts for Karel in the appendix, but I’m still curious whether the LLM has access to task information when generating both the explanation and the reconstructed program. It seems intuitive that understanding a program would be easier when the purpose of the code blocks is provided.
Recent advancements in programmatic reinforcement learning (PRL) primarily focus on improving performance, despite interpretability being a foundational goal of PRL. This makes the motivation for measuring how interpretable a programmatic policy is well-founded, as a program’s interpretability can naturally vary between its language and underlying code. The approach of asking an LLM to explain and then reconstruct the program is both innovative and intuitively sound.
This paper introduces the LINT score, a metric for measuring the interpretability of a programmatic reinforcement learning policy. The idea is based on the intuition that the more interpretable a program is, the easier it is to describe it in natural language. The authors first use an LLM to explain a program in natural language, then ask another LLM to reconstruct the program from the explanation, measuring the similarity between the two versions. Their framework also includes a verifier to ensure that the explanation does not provide line-by-line details for program reconstruction.

They evaluated their method on MicroRTS and Karel the Robot, demonstrating that the more obfuscated the code, the less similar the two versions of the program will be. They also conducted a user study to test whether the proposed metric aligns with human preference.
**The problems of evaluating the results on Karel.** In the Karel the Robot environment, there should be ten tasks, including six from Karel and four from Karel-Hard. What is the reason for selecting the five tasks in the experimental results and not the others? I see you chose Seeder as one of the test beds from Karel-Hard, so why didn’t you include the other tasks as well? Another issue is that there is only one policy for each task, which seems insufficient for a comprehensive evaluation of the results. Additionally, the action metric appears unclear. In Karel the Robot, the action an agent performs depends on both the grid it occupies and the code blocks it is executing. In other words, the agent may perform different actions even when on the same grid. How do you define "state" in this context?

**The experimental evidence is hard to confirm.** In Table 3, the paper lists only the average results for all policies, without including the per-task results. Regarding the level of obfuscation, it appears that you did not specify how you inserted the useless code blocks. Was this done through a rule-based method or with human involvement? Additionally, what are the standards for levels 1 and 2? In Line 168, you mentioned that you only invoked the verifier once, so how can you claim that "all approved explanations satisfied the constraints"?
Yes
No ethics review needed.
Please refer to weaknesses.
- The idea of using LLMs to evaluate policy interpretability is creative and shows promise in leveraging existing language models for new applications.

- Interdisciplinary Relevance: The integration of LLMs into policy interpretability assessment is an exciting intersection of interpretability research and natural language processing, which could be beneficial to both communities.

- The paper is well-structured and the experiments are explained in detail, which makes it easy to follow the motivation and method behind LINT.
The paper proposes the LLM-based Interpretability score as a method for evaluating the interpretability of programmatic policies. The approach leverages LLMs to predict human judgments of interpretability. Experiments are conducted in two domains, MicroRTS and Karel the Robot, with the goal of demonstrating LINT's effectiveness in assessing how easily humans can understand policies.
- The LINT score's novelty would be more compelling if it were benchmarked against existing interpretability metrics. The absence of such comparisons leaves questions about its relative performance. I am confused about why the authors propose using LLMs for evaluation. What are the advantages compared to previous methods? I believe that LLMs are essentially black boxes, which could potentially increase the lack of explainability.

- The user studies are conducted with small sample sizes (n = 60 for MicroRTS, n = 33 for Karel). This limits the statistical power and generalizability of the findings.

- The experiments are confined to just two simple domains, which may not translate well to more complex or diverse real-world scenarios. Broader testing would help in validating the generalizability of LINT. 

- The paper does not adequately address how biases inherent in LLMs might affect the validity of LINT as an interpretability measure. A deeper analysis of LLM bias would help in evaluating the reliability of this method.

- LINT struggles to differentiate policies with small differences in interpretability. Enhancing this sensitivity is crucial for practical applications where subtle interpretability changes matter.
We propose the LINT score, a novel metric using large-language models to assess the interpretability of programmatic policies, and show its correlation with human understanding of interpretability and program behavior.
@inproceedings{
anonymous2024assessing,
title={Assessing the Interpretability of Programmatic Policies using Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tvWD9YueN4},
note={under review}
}
Programmatic representations of policies for solving sequential decision-making problems often carry the promise of interpretability. However, previous work on programmatic policies has only presented anecdotal evidence of policy interpretability. The lack of systematic evaluations of policy interpretability can be attributed to user studies being time-consuming and costly. In this paper, we introduce the LLM-based INTerpretability (LINT) score, a simple and cost-effective metric that uses large-language models (LLMs) to assess the interpretability of programmatic policies. To compute the LINT score of a policy, an LLM generates a natural language description of the policy's behavior. This description is then passed to a second LLM, which attempts to reconstruct the policy from the natural language description. The LINT score measures the behavioral similarity between the original and reconstructed policies. We hypothesized that the LINT score of programmatic policies correlates with their actual interpretability, and evaluated this hypothesis in the domains of MicroRTS and Karel the Robot. Our evaluation relied on a technique from the static obfuscation literature and a user study, where people with various levels of programming proficiency evaluated the interpretability of the programmatic policies. The results of our experiments support our hypothesis. Specifically, the LINT score decreases as the level of obfuscation of the policies increases. The user study showed that LINT can correctly distinguish the ``degree of interpretability'' of programmatic policies generated by the existing algorithms. Our results suggest that LINT can be a helpful tool for advancing the research on interpretability of programmatic policies.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Programmatic Policies Interpretability Program Synthesis
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e62e559dc439871e4c65467dff82a149103eaf87.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Assessing the Interpretability of Programmatic Policies using Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission