metadata

title: VerifiableRewardsForScalableLogicalReasoning
datasets: []
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 5.34.2
app_file: app.py
pinned: false
tags:
  - evaluate
  - reward
  - reasoning
  - metric
description: >-
  VerifiableRewardsForScalableLogicalReasoning is a metric for evaluating
  logical reasoning in AI systems by providing verifiable rewards. It computes
  rewards through symbolic execution of candidate solutions against validation
  programs, enabling automatic, transparent and reproducible evaluation in AI
  systems.

Metric Card for Verifiable Logic Rewards

This metric is part of the SLR framework (LG-Anonym/SLR-Bench) and provides rewards for logical reasoning tasks. THe reward model is grounded in the ILP (Inductive Logic Programming) paradigm, testing whether a given hypothesis (logic rule) solves a logical reasoning task. TO check for entailment, the logic rule is executed against a set of background knowledge and examples, ensuring automatic evaluation that is verifiable, transparent, and reproducible.

How it Works

Input: The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
Execution: The candidate rule is executed against the validation program using a Prolog interpreter.
Correctness Criteria: The rule is considered correct if it entails all positive examples and rejects all negative examples.
Metrics: The symbolic judge computes a range of evaluation metrics (detailed below).

Note: A local Prolog interpreter is required to execute validation programs.

Inputs

predictions (list of str): Each prediction should be a Prolog rule like "eastbound(T) :- Body."
references (list of dict): Each reference should contain:
- validation_program (str): Prolog program with background knowledge and examples
- evaluation_config (dict, optional): Configuration with:
  - positive_predicate (str): Predicate identifying positive examples (default: "eastbound")
  - negative_predicate (str): Predicate identifying negative examples (default: "westbound")

Metrics & Output Values

accuracy (float): Proportion of predictions that correctly classify all examples (0.0 to 1.0)
partial_score (float): Average proportion of correctly classified examples (0.0 to 1.0)
syntax_score (float): Proportion of rules with valid Prolog syntax (0.0 to 1.0)
detailed_results (list of dict): Per-example results with:
- is_correct (bool): Whether the rule correctly classifies all examples
- partial_score (float): Proportion of correctly classified examples
- syntax_valid (bool): Whether the rule has valid syntax
- error (str, optional): Any error messages from Prolog evaluation
- exec_time (float, optional): Execution time for evaluation

How to Use with the datasets library

from evaluate import load
from datasets import load_dataset

# Load the symbolic judge for logical reasoning
symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")

# load dataset LG-Anonym/SLR-Bench
ds = load_dataset('LG-Anonym/SLR-Bench', 'v1-All')
ds_test = ds['test'][:5]

# Prepare the predictions and references
rules = ds_test['ground-truth rule']
references = [{'validation_program': p,
                'evaluation_config': {
                    "positive_predicate": "eastbound",
                    "negative_predicate": "westbound"
                }
               } for p in ds_test['validation program']]
# Compute the evaluation
r2 = symbolic_judge.compute(predictions=rules, references=references)
r2

Outputs

{'accuracy': 1.0,
 'partial_score': 1.0,
 'syntax_score': 1.0,
 'detailed_results': [{'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.014362812042236328},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012364625930786133},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012273550033569336},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012486696243286133},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012131929397583008}]}

Examples

Example 1: Evaluating a Single Rule

from evaluate import load

symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")

validation_program = """
eastbound(train0).
has_car(train0, car0_1).
car_num(car0_1, 1).
car_color(car0_1, white).
car_len(car0_1, short).
has_wall(car0_1, full).

westbound(train1).
has_car(train1, car1_1).
car_num(car1_1, 1).
car_color(car1_1, yellow).
car_len(car1_1, short).
has_wall(car1_1, full).
"""

predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."

results = symbolic_judge.compute(
    predictions=[predicted_rule],
    references=[{"validation_program": validation_program,
                 "evaluation_config": {
                     "positive_predicate": "eastbound",
                     "negative_predicate": "westbound"
                 }}]
)

print(results)

Output Example 1

{'accuracy': 1.0,
 'partial_score': 1.0,
 'syntax_score': 1.0,
 'detailed_results': [
     {'is_correct': True,
      'partial_score': 1.0,
      'syntax_valid': True,
      'error': None,
      'exec_time1': 0.012056350708007812}]
 }

Example 2: Evaluating Multiple Rules

correct_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
incorrect_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, green)."

results = symbolic_judge.compute(
    predictions=[correct_rule, incorrect_rule],
    references=[
        {"validation_program": validation_program},
        {"validation_program": validation_program}
    ]
)

print(results)

Example 3: Custom Evaluation Configuration

validation_program = """
% Custom problem
parent(john, mary).
parent(john, bob).
parent(alice, bob).
parent(susan, alice).

% Examples
grandparent(susan, bob).
not_grandparent(john, alice).
"""

rule = "grandparent(X, Y) :- parent(X, Z), parent(Z, Y)."

results = symbolic_judge.compute(
    predictions=[rule],
    references=[{
        "validation_program": validation_program,
        "evaluation_config": {
            "positive_predicate": "grandparent",
            "negative_predicate": "not_grandparent"
        }
    }]
)