A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns
recommender systems;causality;evaluation;auditing;machine learning
Yes
No ethics review needed.
1. How does the computational complexity scale with the number of users, items and time horizon? What are recommended approaches for large-scale recommender systems?

2. What are the practical implications of assuming static embeddings during gradient computation? How would the results change with full retraining?

3. Could the framework be extended to handle more complex recommendation scenarios like slate recommendations or contextual bandits?
1. The technical claims and methodology are very well-supported. The causal framework is rigorously developed with clear mathematical formulations. The empirical evaluation is comprehensive, with well-designed ablation studies showing impact of various stochasticity levels, time horizon lengths and model architecture choices.

2. Novel formalization of reachability and stability metrics presented capture both immediate and long-term effects, handle multi-step recommendation dynamics and account for both user and adversary perspectives.

3. The paper is generally well-written and logically structured. The causal framework is presented clearly with helpful examples.
The paper presents a unified causal framework for auditing recommender systems with focus on user agency. The authors make three main contributions:
1. A general causal framework that formalizes interventional and counterfactual metrics for auditing recommender systems.
2. Two novel classes of metrics - reachability and stability, to measure user agency while accounting for recommendation dynamics.
3. Efficient computational methods for these metrics under different levels of access to the recommender system.

The framework is evaluated empirically using both matrix factorization and recurrent neural network based recommenders, showcasing interesting trade-offs between stability and reachability.
1. The assumption of static user/item embeddings during gradient computation could be better justified. Additional experiments showing impact of this simplification would be valuable.

2. The empirical evaluation focuses on movie recommendations - testing on other domains (e.g. social media, e-commerce, etc.) would strengthen the framework's generalizability claims.

3. The choice of distance metrics for stability measures (L2 distance) could be better justified. Adding discussion of metric sensitivity to adversarial perturbations and analysis of the relationship between local and global notions of reachability would be useful.

4. The paper presents limited discussion of computational complexity and scalability analysis, particularly for large-scale recommender systems. The paper could analyze how the methods scale with number of users, items and time horizon.
Yes
No ethics review needed.
What are the differences and impacts of applying this model to various recommendation models?
The causal approach offers a novel way to address ethical issues, providing a structured method for defining and calculating user-centric metrics.

Offering both gradient-based and black-box methods for metric computation enables broader application
The paper proposes a unified causal framework for auditing recommender systems, specifically to address ethical concerns such as user agency, stability, and reachability. It categorizes auditing metrics from a causal perspective and introduces two key metrics, past- and future-reachability, and stability, which measure a user’s ability to influence recommendations. The empirical studies evaluate the metrics on different recommender models, highlighting the trade-offs between user influence on recommendations and system stability.
The framework’s reliance on specific causal assumptions and models,this may reduce its generalizability across diverse recommender systems.

The paper lacks a discussion about the differences between recommendation systems.
Yes
No ethics review needed.
Please see them in the weaknesses.
S1- This paper provides comprehensive details on the background of the problem.

S2- The authors give detailed experiment settings which improves the reproducibility.
In this work, authors pay attention to recommender system auditing from a causal perspective, and point out the lack of metrics for auditing user agency for the recommendation process. Therefore, two metrics are proposed, including future- and past-reachability and stability, which can measure the impact of users on their own and other users. To calculate these metrics, the authors also design a gradient-based and a black box approach.
W1-The motivation of this paper is not quite clear. For example, what’s the actual relationship between user agency and ethical concerns? 

W2-The experiments are only conducted on ML-1M, which are insufficient to explain the universality of the conclusions since the recommendation senarios are diverse. Experiments on at leasr one dataset from other recommendation senarios are needed.

W3- In Figure 3 for the distribution of past instability values, for MF, Past-5 shows lower proportion of 0.0 than Past-1, but for RRN, Past-5 presents higher proportion of 0.0 than Past-1. Could you please explain the reason for this contrary result?
Yes
No ethics review needed.
In addition to **Q1** to **Q5** mentioned in the Weaknesses, I have several other questions:

- **Q6**: What is the relationship between the proposed metrics and recommendation performance? Does a stronger recommendation model perform better according to these metrics?

- **Q7**: The metric comparisons in Figure 3 are described but lack corresponding explanations. For instance, why do some items show "a user’s recommended list is either heavily affected by the actions of an adversary or is minimally affected by them"?

- **Q8**: Is the time horizon parameter in the experimental parameters equivalent to $k$ in Definitions 4.1 and 4.2? If not, how is $k$ set in the experiments?

I am happy to engage in further discussion, and if these issues are addressed, I am willing to reconsider the score.
- Auditing recommender systems is a highly meaningful area of study, and the paper contributes valuable insights.
- The article is well-written and clearly articulated, making complex concepts accessible.
- It provides methods for auditing from both white-box and black-box perspectives, catering to different levels of system access.
In this work, the authors adopt a causal perspective on recommender system auditing and present a general method for defining auditing metrics. Within this overarching causal auditing framework, they categorize existing audit metrics. Leveraging their framework, they propose two types of metrics: future-/past-reachability and stability, which respectively measure a user's ability to influence recommendations for themselves and for other users. Additionally, they introduce a gradient-based method and a black-box method for calculating these metrics, allowing auditors to assess them at various levels of system access. Empirically, the authors demonstrate the effectiveness of their proposed metrics and use them to examine the design of recommender systems.
- **W1**: **Ambiguity in Definitions**: The definitions in the article lack detailed explanations, which may lead to ambiguity. For example:
  - **Q1-1**: In Definitions 4.1 and 4.2, the authors consider only the intervention on $O_{i,t}$ without accounting for its effect on $A_{i,t+1}$. Why was this setting chosen?
  - **Q1-2**: Are Definitions 4.1 and 4.2 consistent? Specifically, does past-$k$ at time $t+k$ equal future-$k$ at time $t$? It would be helpful if the authors could address this question both intuitively and formally.

- **W2**: **Limited Analysis Scope**: The analysis in Section 5 is confined to $k=1$, representing only a special case of the broader definitions provided.
  - **Q2**: Please describe how the corresponding white-box and black-box methods would operate when $k > 1$.

- **W3**: **Practical Applicability Concerns**: There is a gap between the theoretical propositions and practical scenarios.
  - **Q3**: Proposition 5.1 requires fixing item embeddings, while Proposition 5.2 requires fixing user embeddings. Since these conditions are difficult to meet in real recommender systems, how does this gap affect practical auditing?

- **W4**: **Lack of Experimental Rationale**: Certain experimental setups lack clear justification.
  - **Q4-1**: Section 6.1 mentions different policies for future and past metrics. Why was this setup chosen? Please explain the rationale behind this decision.

- **W5**: **Incomplete Experimental Validation**:
  - **Q5-1**: The use of a single dataset limits the experimental scope and generalizability of the findings.
  - **Q5-2**: The current experiments focus on analyzing existing models within the proposed framework but do not clarify why this framework or these metrics are more valid than existing auditing methods. Additional experiments, such as straightforward case studies, are needed to further validate the framework.
@inproceedings{
anonymous2024a,
title={A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=0vMLqSdsKW},
note={under review}
}
As recommender systems become widely deployed in different domains, they increasingly influence their users’ beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the improvement of recommendation algorithms but also provides ways to assess and address ethical concerns surrounding them. In this work, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them—notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics: future- and past-reachability and stability, that measure the ability of a user to influence their own and other users’ recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. Empirically, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
recommender systems causality evaluation auditing machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b72adb3a7114b3936d6ba44bf74c115ceb2619c2.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e7074aefe8a9a341f2948af3081f1209000c552e.zip
A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission