Upload 7 files

.gitignore

@@ -0,0 +1 @@
+.idea

README.md

@@ -1,12 +1,196 @@
 ---
 title: VerifiableRewardsForScalableLogicalReasoning
-emoji: 💻
-colorFrom: red
-colorTo: yellow
+datasets: []
+colorFrom: blue
+colorTo: red
 sdk: gradio
-sdk_version: 5.38.2
+sdk_version: 5.34.2
 app_file: app.py
 pinned: false
+tags:
+  - evaluate
+  - reward
+  - reasoning
+  - metric
+description: >-
+    VerifiableRewardsForScalableLogicalReasoning is a metric for evaluating logical reasoning in AI systems by providing verifiable rewards. It computes rewards through symbolic execution of candidate solutions against validation programs, enabling automatic, transparent and reproducible evaluation in AI systems.
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Metric Card for Symbolic Judge: Verifiable Rewards for Scalable Logical Reasoning
+
+This metric is part of the SLR framework (LG-Anonym/SLR-Bench) and provides rewards for logical reasoning tasks.
+THe reward model is grounded in the ILP (Inductive Logic Programming) paradigm, testing whether a given hypothesis (logic rule) solves a logical reasoning task.
+TO check for entailment, the logic rule is executed against a set of background knowledge and examples, ensuring automatic evaluation that is verifiable, transparent, and reproducible.
+
+
+### How it Works
+- **Input:** The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+- **Execution:** The candidate rule is executed against the validation program using a Prolog interpreter.
+- **Correctness Criteria:** The rule is considered correct if it entails all positive examples and rejects all negative examples.
+- **Metrics:** The symbolic judge computes a range of evaluation metrics (detailed below).
+
+**Note:** A local Prolog interpreter is required to execute validation programs.
+
+---
+
+### Inputs
+- **predictions** (`list` of `str`): Each prediction should be a Prolog rule like "eastbound(T) :- Body."
+- **references** (`list` of `dict`): Each reference should contain:
+  - **validation_program** (`str`): Prolog program with background knowledge and examples
+  - **evaluation_config** (`dict`, optional): Configuration with:
+    - **positive_predicate** (`str`): Predicate identifying positive examples (default: "eastbound")
+    - **negative_predicate** (`str`): Predicate identifying negative examples (default: "westbound")
+
+### Metrics & Output Values
+- **accuracy** (`float`): Proportion of predictions that correctly classify all examples (0.0 to 1.0)
+- **partial_score** (`float`): Average proportion of correctly classified examples (0.0 to 1.0)
+- **syntax_score** (`float`): Proportion of rules with valid Prolog syntax (0.0 to 1.0)
+- **detailed_results** (`list` of `dict`): Per-example results with:
+  - **is_correct** (`bool`): Whether the rule correctly classifies all examples
+  - **partial_score** (`float`): Proportion of correctly classified examples
+  - **syntax_valid** (`bool`): Whether the rule has valid syntax
+  - **error** (`str`, optional): Any error messages from Prolog evaluation
+  - **exec_time** (`float`, optional): Execution time for evaluation
+
+---
+## How to Use with the datasets library
+```python
+from evaluate import load
+from datasets import load_dataset
+
+# Load the symbolic judge for logical reasoning
+symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")
+
+# load dataset LG-Anonym/SLR-Bench
+ds = load_dataset('LG-Anonym/SLR-Bench', 'v1-All')
+ds_test = ds['test'][:5]
+
+# Prepare the predictions and references
+rules = ds_test['ground-truth rule']
+references = [{'validation_program': p,
+                'evaluation_config': {
+                    "positive_predicate": "eastbound",
+                    "negative_predicate": "westbound"
+                }
+               } for p in ds_test['validation program']]
+# Compute the evaluation
+r2 = symbolic_judge.compute(predictions=rules, references=references)
+r2
+```
+
+### Outputs
+
+```python
+{'accuracy': 1.0,
+ 'partial_score': 1.0,
+ 'syntax_score': 1.0,
+ 'detailed_results': [{'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.014362812042236328},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012364625930786133},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012273550033569336},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012486696243286133},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012131929397583008}]}
+```
+
+
+---
+
+## Examples
+
+### Example 1: Evaluating a Single Rule
+
+```python
+from evaluate import load
+
+symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")
+
+validation_program = """
+eastbound(train0).
+has_car(train0, car0_1).
+car_num(car0_1, 1).
+car_color(car0_1, white).
+car_len(car0_1, short).
+has_wall(car0_1, full).
+
+westbound(train1).
+has_car(train1, car1_1).
+car_num(car1_1, 1).
+car_color(car1_1, yellow).
+car_len(car1_1, short).
+has_wall(car1_1, full).
+"""
+
+predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+
+results = symbolic_judge.compute(
+    predictions=[predicted_rule],
+    references=[{"validation_program": validation_program,
+                 "evaluation_config": {
+                     "positive_predicate": "eastbound",
+                     "negative_predicate": "westbound"
+                 }}]
+)
+
+print(results)
+```
+
+### Output Example 1
+
+```python
+{'accuracy': 1.0,
+ 'partial_score': 1.0,
+ 'syntax_score': 1.0,
+ 'detailed_results': [
+     {'is_correct': True,
+      'partial_score': 1.0,
+      'syntax_valid': True,
+      'error': None,
+      'exec_time1': 0.012056350708007812}]
+ }
+
+```
+
+### Example 2: Evaluating Multiple Rules
+
+```python
+correct_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+incorrect_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, green)."
+
+results = symbolic_judge.compute(
+    predictions=[correct_rule, incorrect_rule],
+    references=[
+        {"validation_program": validation_program},
+        {"validation_program": validation_program}
+    ]
+)
+
+print(results)
+```
+
+### Example 3: Custom Evaluation Configuration
+
+```python
+validation_program = """
+% Custom problem
+parent(john, mary).
+parent(john, bob).
+parent(alice, bob).
+parent(susan, alice).
+
+% Examples
+grandparent(susan, bob).
+not_grandparent(john, alice).
+"""
+
+rule = "grandparent(X, Y) :- parent(X, Z), parent(Z, Y)."
+
+results = symbolic_judge.compute(
+    predictions=[rule],
+    references=[{
+        "validation_program": validation_program,
+        "evaluation_config": {
+            "positive_predicate": "grandparent",
+            "negative_predicate": "not_grandparent"
+        }
+    }]
+)
+```

VerifiableRewardsForScalableLogicalReasoning.py

@@ -0,0 +1,371 @@
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Symbolic Judge: Verifiable Rewards for Logical Reasoning at Scale """
+
+import os
+import subprocess
+import tempfile
+import evaluate
+import logging
+import datasets
+from tqdm import tqdm
+import time
+import multiprocessing as mp
+import re
+
+logger = logging.getLogger(__name__)
+
+_CITATION = """\
+@misc{anonymous2025slr, 
+      author = {Anonymous}, 
+      title = {SLR: Anonymous}, 
+      year = {2025}
+}
+"""
+
+_DESCRIPTION = """\
+Verifiable Rewards for Scalable Logical Reasoning (**SLR**) provides verifiable rewards via logic programm execution.
+It deterministically evaluates candidate hypotheses by executing them against the validation program and verifying all positive examples ($E^+$) are entailed and all negative examples ($E^-$) are not entailed .
+Evaluations performed are fully verifiable and grounded in formal logic, ensuring an automatic, transparent, and reproducible standard for evaluation and reward in both supervised and reinforcement learning settings.
+How it Works:
+  - Input: A candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+  - Execution: The candidate rule is executed against the validation program using a Prolog interpreter.
+  - Correctness Criteria: The rule is considered correct if it entails all positive examples and rejects all negative examples.
+  - Metrics: We provide a range of evaluation metrics (detailed below).
+  - Usage: see **Documentation tab** for details on how to use Verifiable Rewards for Scalable Logical Reasoning in your own projects.
+
+Example usage:
+  from evaluate import load
+
+  symbolic_judge = load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")
+
+  validation_program = \"\"\"
+  eastbound(train0).
+  has_car(train0, car0_1).
+  car_num(car0_1, 1).
+  car_color(car0_1, white).
+  car_len(car0_1, short).
+  has_wall(car0_1, full).
+
+  westbound(train1).
+  has_car(train1, car1_1).
+  car_num(car1_1, 1).
+  car_color(car1_1, yellow).
+  car_len(car1_1, short).
+  has_wall(car1_1, full).
+  \"\"\"
+
+  predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+
+  results = symbolic_judge.compute(
+      predictions=[predicted_rule],
+      references=[{"validation_program": validation_program,
+                   "evaluation_config": {
+                       "positive_predicate": "eastbound",
+                       "negative_predicate": "westbound"
+                   }}]
+  )
+
+Note: A local Prolog interpreter is required to execute validation programs.
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions (`list` of `str`): Each prediction should be a Prolog rule like "pred(T) :- Body."
+    references (`list` of `dict`): Each reference should contain:
+        - 'validation_program' (`str`): Background knowledge in Prolog syntax
+        - 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
+        Define: positive_predicate, and negative_predicate, the positive one should match the head of the rule to evaluate. 
+Returns:
+    accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
+    partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
+    syntax_score (`float`): Proportion of rules with valid syntax. Value is between 0 and 1.
+    detailed_results (`list` of `dict`): Per-example results including correctness, partial score, execution time, and any errors encountered.
+"""
+
+
+def validate_rule_no_hardcoded_cars(prediction):
+    """Reject rules that hardcode specific car identifiers"""
+    import re
+
+    # Look for has_car with a constant (lowercase) in second position
+    hardcoded_pattern = r'has_car\([^,]+,\s*([a-z][a-z0-9_]*)\)'
+    matches = re.findall(hardcoded_pattern, prediction)
+
+    if matches:
+        return False, f"Cars must be variables: {matches[0]}"
+
+    return True, "Rule is valid"
+
+
+def _evaluate_with_prolog(prediction, validation_program, eval_config, timeout=5):
+    """
+    Evaluates a predicted rule against the validation program using Prolog.
+    """
+
+
+    # Extract configuration
+    positive_pred = eval_config.get("positive_predicate", "eastbound")
+    negative_pred = eval_config.get("negative_predicate", "westbound")
+    allow_multiple_rules = eval_config.get("allow_multiple_rules", False)
+
+    # extract predicate from rule_to_evaluate
+    rule_to_evaluate = extract_ilp_from_text_v2(prediction, positive_pred, allow_multiple_rules)
+
+    is_valid, validation_msg = validate_rule_no_hardcoded_cars(rule_to_evaluate)
+    if not is_valid:
+        return {
+            "is_correct": False,
+            "partial_score": 0.0,
+            "syntax_valid": False,
+            "error": f"Rule validation failed: {validation_msg}"
+        }
+
+    if positive_pred not in rule_to_evaluate:
+        p = prediction.replace('\n', ' ')
+        return {
+            "is_correct": False,
+            "partial_score": 0.0,
+            "syntax_valid": False,
+            "error": f"Invalid Syntax: Logic Rule not found for symbol '{positive_pred}': {p}"
+        }
+
+    pos_examples = re.findall(rf'{positive_pred}\(([^)]+)\)', validation_program)
+    neg_examples = re.findall(rf'{negative_pred}\(([^)]+)\)', validation_program)
+
+    # Determine arity by counting commas in first example plus 1
+    arity = 1  # default to unary
+    if pos_examples:
+        arity = pos_examples[0].count(',') + 1
+    elif neg_examples:
+        arity = neg_examples[0].count(',') + 1
+
+    # Create variables based on arity
+    vars = ", ".join([f"X{i}" for i in range(1, arity + 1)])
+
+    symbolic_judge = f"""
+% Dynamic evaluation predicates
+check({vars}) :- pos({vars}), {positive_pred}({vars}).      % positive covered
+check({vars}) :- neg({vars}), \\+ {positive_pred}({vars}).  % negative rejected
+
+% Count successful checks
+check_count(Count) :- 
+    (setof(({vars}), ((pos({vars}); neg({vars})), check({vars})), CorrectExamples) ->
+        length(CorrectExamples, Count)
+    ;
+        Count = 0
+    ).
+
+check_all :- forall((pos({vars});neg({vars})), check({vars})).
+    """
+    # Add the rule to evaluate
+    validation_program = re.sub(rf'\b{positive_pred}\b', 'pos', validation_program)
+    validation_program = re.sub(rf'\b{negative_pred}\b', 'neg', validation_program)
+
+    pos_negs = validation_program.count("pos(") + validation_program.count("neg(")
+    validation_program = '\n'.join(sorted(validation_program.splitlines()))
+    full_program = validation_program + "\n\n" + symbolic_judge + "\n\n" + rule_to_evaluate + "\n\n"
+
+    with tempfile.NamedTemporaryFile(suffix='.pl', mode='w', delete=False) as f:
+        f.write(full_program)
+        temp_file = f.name
+
+    try:
+        eval_start_time = time.time()
+        # Execute the Prolog program
+        cmd = ['swipl', '-s', temp_file, '-g', 'check_count(Count), writeln(Count)', '-t', 'halt']
+        result = subprocess.run(
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            timeout=timeout,
+            text=True
+        )
+        partial_score = 0.0 if result.stdout.strip() == '' else int(result.stdout.strip())
+        # Extract partial score from output
+        partial_score = partial_score / pos_negs if pos_negs > 0 else 0.0
+
+        is_correct = True if partial_score == 1.0 else False
+
+        error =  f'{result.stderr} -> Eval Rule "{rule_to_evaluate}"' if result.stderr else None
+        t1 = time.time()
+
+        return {
+            "is_correct": is_correct,
+            "partial_score": partial_score,
+            "syntax_valid": True,
+            "error": error,
+            "exec_time1": t1 - eval_start_time,
+        }
+
+    except subprocess.TimeoutExpired:
+        r = rule_to_evaluate.replace('\n', ' ')
+        return {"is_correct": False, "partial_score": 0.0, "syntax_valid": False,
+                "error": "Evaluation timed out after {timeout} seconds for rule: '{r}'"}
+    except Exception as e:
+        return {"is_correct": False, "partial_score": 0.0, "syntax_valid": False,
+                "error": f"Error evaluating rule '{rule_to_evaluate}' returns: '{result.stdout.strip() if result else 'No error message'}' with error: {e}"}
+    finally:
+        if os.path.exists(temp_file):
+            os.remove(temp_file)
+
+def extract_ilp_from_text(text):
+    rule_patterns = [
+        # Pattern with body (full rule with implication)
+        r'([a-zA-Z_][a-zA-Z0-9_]*\([^)]*\)\s*:-[^.]*\.)',
+        # Pattern for facts (no body)
+        # r'([a-zA-Z_][a-zA-Z0-9_]*\([^)]*\)\s*\.)'
+    ]
+    p_code = ''
+    for pattern in rule_patterns:
+        matches = re.findall(pattern, text)
+        for match in matches:
+            # Ensure the rule ends with a period
+            statement = match.strip()
+            if not statement.endswith('.'):
+                statement += '.'
+            p_code += statement + '\n'
+    return p_code
+
+
+def extract_ilp_from_text_v2(text, target_predicate=None, allow_multiple_rules=False):
+    text = re.sub(r'%.*?(?=\n|$)', '', text) # remove comments
+    # Pre-process: collapse code blocks to single lines
+    text = re.sub(r'\n\s*', ' ', text)  # crude: flatten all to one line
+    # Rule pattern, across newlines
+    rule_pattern = re.compile(rf'({target_predicate}\([^()]*\)\s*:-.*?\.)')
+    rules = list(rule_pattern.findall(text))
+    if len(rules) > 1 and not allow_multiple_rules:
+        # logger.warning(f"Found multiple rules in text, but allow_multiple_rules is set to False. Using only the last match.")
+        rules = rules[-1:]
+    # Remove rules that are also captured as facts
+    p_code = ''
+    for rule in rules:
+        # Ensure the rule ends with a period
+        statement = rule.strip()
+        if not statement.endswith('.'):
+            statement += '.'
+        p_code += statement + '\n'
+    return p_code.strip()  # Ensure no trailing whitespace
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class VerifiableRewardsForScalableLogicalReasoning(evaluate.Metric):
+    def __init__(self, config_name=None, **kwargs):
+        """
+        Initializes the PrologEval metric.
+
+        Args:
+            config_name (str, optional): Name of the configuration to use.
+            **kwargs: Additional keyword arguments.
+        """
+        super().__init__(config_name=config_name, **kwargs)
+        self.config_name = config_name or "default"
+        self._info = self._info()
+        self._download_and_prepare(dl_manager=None)
+
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features({
+                'predictions': datasets.Value('string'),
+                'references': {
+                    'validation_program': datasets.Value('string'),
+                    'evaluation_config': {
+                        'positive_predicate': datasets.Value('string'),
+                        'negative_predicate': datasets.Value('string')
+                    }
+                },
+            }),
+            codebase_urls=["https://github.com/LG-Anonym/SLR-Bench"],
+            reference_urls=["https://huggingface.co/datasets/LG-Anonym/SLR-Bench"]
+        )
+
+    def _download_and_prepare(self, dl_manager):
+        """Checks if SWI-Prolog is installed or warns the user."""
+        try:
+            subprocess.run(
+                ["swipl", "--version"],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                check=True
+            )
+        except (subprocess.CalledProcessError, FileNotFoundError):
+            logger.warning(
+                "SWI-Prolog not found. Please install it:\n"
+                "Ubuntu/Debian: sudo apt-get install swi-prolog\n"
+                "macOS: brew install swi-prolog\n"
+                "Windows: download from https://www.swi-prolog.org/download/stable"
+            )
+
+    def _compute(self, predictions: list, references: list):
+        """Calculates the accuracy of predictions using Prolog for evaluation with multiprocessing."""
+        if not isinstance(predictions, list):
+            predictions = [predictions]
+
+        if len(predictions) != len(references):
+            raise ValueError(
+                f"Number of predictions ({len(predictions)}) and references {len(references)}) don't match")
+
+        TIMEOUT = 10 if len(predictions) > 500 else 5
+        # Prepare evaluation inputs
+        eval_inputs = []
+        for i, (prediction, reference) in enumerate(zip(predictions, references)):
+            validation_program = reference.get("validation_program", reference.get("validation program"))
+
+            # Extract configuration parameters directly from reference
+            # This is the key fix: look for config values at the top level if evaluation_config doesn't exist
+            eval_config = reference.get("evaluation_config", {
+                    "positive_predicate": "eastbound",
+                    "negative_predicate": "westbound"
+                })
+
+            if not validation_program:
+                raise ValueError(f"Example {i} does not contain validation program field")
+
+            eval_inputs.append((prediction, validation_program, eval_config, TIMEOUT))
+
+        # if more than 1k predictions, we use multiprocessing to speed up the evaluation
+        if len(eval_inputs) > 500:
+            # Process evaluations in parallel
+            num_cpus = max(1, mp.cpu_count() - 1)  # Leave one CPU free
+            with mp.Pool(processes=num_cpus) as pool:
+                results = list(tqdm(
+                    pool.starmap(_evaluate_with_prolog, eval_inputs),
+                    total=len(eval_inputs),
+                    desc=f"Evaluating rules (parallel processing with {num_cpus} CPUs)"
+                ))
+        else:
+            # Evaluate in the main thread (no multiprocessing)
+            results = []
+            for prediction, validation_program, eval_config, t in tqdm(eval_inputs, total=len(predictions), desc="Evaluating rules"):
+                results.append(_evaluate_with_prolog(prediction, validation_program, eval_config, timeout=t))
+
+        # Calculate metrics
+        partial_scores = [result["partial_score"] for result in results]
+        correct_count = sum(1 for result in results if result["is_correct"])
+        syntax_valid_count = sum(1 for result in results if result["syntax_valid"])
+
+        accuracy = correct_count / len(predictions) if predictions else 0
+        partial_score = sum(partial_scores) / len(predictions) if partial_scores else 0
+        syntax_score = syntax_valid_count / len(predictions) if predictions else 0
+
+        return {
+            "accuracy": accuracy,
+            "partial_score": partial_score,
+            "syntax_score": syntax_score,
+            "detailed_results": results
+        }

app.py

@@ -0,0 +1,269 @@
+import evaluate
+import gradio as gr
+import os
+
+
+# Create your own Gradio interface instead of using the built-in widget
+def create_interface(module):
+    def evaluate_fn(prediction, references, pos_pred, neg_pred):
+        # Check if all required fields are filled
+        if not prediction or prediction.strip() == "":
+            return "", "", "", "Please provide a candidate hypothesis to evaluate."
+
+        if not references or references.strip() == "":
+            return "", "", "", "Please provide a validation program."
+
+        if not pos_pred or pos_pred.strip() == "":
+            return "", "", "", "Please specify the positive predicate name."
+
+        if not neg_pred or neg_pred.strip() == "":
+            return "", "", "", "Please specify the negative predicate name."
+
+        # Process a single prediction instead of multiple
+        pred = prediction.strip()
+
+        # Create reference with evaluation_config at the top level
+        ref = {
+            "validation_program": references.strip(),
+            "evaluation_config": {
+                "positive_predicate": pos_pred,
+                "negative_predicate": neg_pred
+            }
+        }
+
+        # Use a list with a single prediction and reference
+        results = module.compute(predictions=[pred], references=[ref])
+
+        # Extract the error message from detailed_results if it exists
+        error_msg = ""
+        if results["detailed_results"] and len(results["detailed_results"]) > 0:
+            error = results["detailed_results"][0].get("error")
+            if error:
+                error_msg = error
+
+        return (
+            f"Accuracy score: {results['accuracy']:.4f}",
+            f"Partial score: {results['partial_score']:.4f}",
+            f"Syntax score: {results['syntax_score']:.4f}",
+            error_msg
+        )
+
+    # Helper function to load example data
+    def load_example(example):
+        return (
+            example["rule"],
+            example["validation"],
+            example["pos_pred"],
+            example["neg_pred"]
+        )
+
+    # Read README.md content
+    readme_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "README.md")
+    with open(readme_path, 'r') as f:
+        readme_content = f.read()
+        readme_content = '# Metric Card ' + readme_content.split('# Metric Card ')[1]
+
+    # Define examples for quick use
+    example_train = {
+        "description": "Basic Train Problem",
+        "validation": """eastbound(train0).
+has_car(train0, car0_1).
+car_num(car0_1, 1).
+car_color(car0_1, white).
+car_len(car0_1, short).
+has_wall(car0_1, full).
+
+westbound(train1).
+has_car(train1, car1_1).
+car_num(car1_1, 1).
+car_color(car1_1, yellow).
+car_len(car1_1, short).
+has_wall(car1_1, full).
+""",
+        "rule": "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white).",
+        "pos_pred": "eastbound",
+        "neg_pred": "westbound"
+    }
+
+    example_family = {
+        "description": "Family Relationships",
+        "validation": """% Custom problem
+parent(john, mary).
+parent(john, bob).
+parent(alice, bob).
+parent(susan, alice).
+
+% Examples
+grandparent(susan, bob).
+not_grandparent(john, alice).""",
+        "rule": "grandparent(X, Y) :- parent(X, Z), parent(Z, Y).",
+        "pos_pred": "grandparent",
+        "neg_pred": "not_grandparent"
+    }
+
+    with gr.Blocks(title="Symbolic Judge") as demo:
+        with gr.Tab("Evaluation"):
+            gr.Markdown("# Symbolic Judge: Verifiable Rewards for Scalable Logical Reasoning")
+            gr.Markdown("""
+Verifiable Rewards for Scalable Logical Reasoning (**SLR**) provides verifiable rewards via logic programm execution.
+It deterministically evaluates candidate hypotheses by executing them against the validation program and verifying all positive examples ($E^+$) are entailed and all negative examples ($E^-$) are not entailed .
+Evaluations performed are fully verifiable and grounded in formal logic, ensuring an automatic, transparent, and reproducible standard for evaluation and reward in both supervised and reinforcement learning settings.
+How it Works:
+  - Input: A candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+  - Execution: The candidate rule is executed against the validation program using a Prolog interpreter.
+  - Correctness Criteria: The rule is considered correct if it entails all positive examples and rejects all negative examples.
+  - Metrics: We provide a range of evaluation metrics (detailed below).
+  - Usage: see **Documentation tab** for details on how to use Verifiable Rewards for Scalable Logical Reasoning in your own projects.
+**Note:** A local Prolog interpreter is required to execute validation programs.
+""")
+
+            with gr.Row():
+                with gr.Column(scale=1):
+                    with gr.Group():
+                        gr.Markdown("### Model Output")
+                        prediction_input = gr.Textbox(
+                            label="Candidate Hypothesis to be evaluated(predicted rule by the model)",
+                            placeholder="eastbound(T) :- has_car(T, C), short(C), open(C).",
+                            lines=5
+                        )
+
+                        with gr.Group():
+                            gr.Markdown("### Validation Program")
+
+                            references_input = gr.Textbox(
+                                label="The validation program contains background knowledge and examples for testing",
+                                placeholder="% Background knowledge\ncar(car_1). car(car_2).\nlong(car_2). short(car_1).\nopen(car_1). closed(car_2).\n\n% Examples\neastbound(train_1).\nwestbound(train_2).\n\n% Train configurations\nhas_car(train_1, car_1).\nhas_car(train_2, car_2).",
+                                lines=12
+                            )
+
+                            with gr.Row():
+                                pos_pred_input = gr.Textbox(
+                                    label="Positive Validation Examples",
+                                    value="eastbound",
+                                    placeholder="eastbound",
+                                    info="The predicate name identifying positive examples in the validation program"
+                                )
+                                neg_pred_input = gr.Textbox(
+                                    label="Negative Validation Examples",
+                                    value="westbound",
+                                    placeholder="westbound",
+                                    info="The predicate name identifying negative examples in the validation program"
+                                )
+
+                        eval_button = gr.Button("Evaluate", variant="primary")
+
+                with gr.Column(scale=1):
+                    with gr.Group():
+                        gr.Markdown("### Evaluation Metrics")
+                        with gr.Group():
+                            accuracy_output = gr.Textbox(
+                                label="Overall Accuracy",
+                                info="Proportion of hypotheses that solve the tasks",
+                                container=True
+                            )
+                            partial_score_output = gr.Textbox(
+                                label="Partial Score",
+                                info="Proportion of examples that are correctly classified in the tasks",
+                                container=True
+                            )
+                            syntax_score_output = gr.Textbox(
+                                label="Syntax Score",
+                                info="Proportion of syntactically valid hypothesis",
+                                container=True
+                            )
+                            error_output = gr.Textbox(
+                                label="Syntax Details",
+                                info="Error messages for syntax errors or execution failures",
+                                container=True,
+                            )
+                            gr.Markdown("Note: This interface evaluates a single hypothesis at a time. Use Python API for batch processing")
+            # Add the examples section
+            # Example list
+            examples = [
+                ["Train Problem", example_train],
+                ["Family Relationships", example_family]
+            ]
+
+            with gr.Accordion("Example Logical Reasoning Tasks", open=True):
+                example_radio = gr.Radio([ex[0] for ex in examples], label="Select an example", value="Train Problem")
+                # Show preview of selected example
+                with gr.Row():
+                    with gr.Column():
+                        gr.Markdown("### Selected Example Preview")
+                        example_description = gr.Markdown("**Description**: " + example_train["description"])
+                        with gr.Row():
+                            with gr.Column():
+                                gr.Markdown("#### Candidate Hypothesis")
+                                example_rule = gr.Code(value=example_train["rule"])
+                            with gr.Column():
+                                gr.Markdown("#### Validation Program")
+                                example_validation = gr.Code(value=example_train["validation"])
+
+                        with gr.Row():
+                            with gr.Column():
+                                gr.Markdown("#### Validation Examples")
+                                example_predicates = gr.Markdown(f"""
+                                **Positive Examples**: `{example_train["pos_pred"]}`  
+                                **Negative Examples**: `{example_train["neg_pred"]}`
+                                """)
+
+                # Load button for the selected example
+                load_button = gr.Button("Load Selected Example", variant="secondary")
+            gr.Markdown("### Citation")
+
+            gr.Markdown("""
+            If you use Symbolic Judge in your work, please cite:
+            ```
+            @misc{anonymous2025slr,
+                  title={Verifiable Rewards for Scalable Logical Reasoning},
+                  author={Anonymous},
+                  year={2025},
+            }
+            ```
+            """)
+
+            # Set up event handlers for the example selection
+            def update_example_preview(selection):
+                selected_example = next((ex[1] for ex in examples if ex[0] == selection), example_train)
+                return (
+                    "**Description**: " + selected_example["description"],
+                    selected_example["rule"],
+                    selected_example["validation"],
+                    f"""
+                    **Positive Examples**: `{selected_example["pos_pred"]}`  
+                    **Negative Examples**: `{selected_example["neg_pred"]}`
+                    """
+                )
+
+            example_radio.change(
+                fn=update_example_preview,
+                inputs=[example_radio],
+                outputs=[example_description, example_rule, example_validation, example_predicates]
+            )
+
+            # Event handler for the load button
+            def load_selected_example(selection):
+                selected_example = next((ex[1] for ex in examples if ex[0] == selection), example_train)
+                return load_example(selected_example)
+
+            load_button.click(
+                fn=load_selected_example,
+                inputs=[example_radio],
+                outputs=[prediction_input, references_input, pos_pred_input, neg_pred_input]
+            )
+
+            # Set up the evaluate button
+            eval_button.click(
+                fn=evaluate_fn,
+                inputs=[prediction_input, references_input, pos_pred_input, neg_pred_input],
+                outputs=[accuracy_output, partial_score_output, syntax_score_output, error_output]
+            )
+
+        with gr.Tab("Documentation"):
+            gr.Markdown(readme_content)
+
+    return demo
+
+# Use a local path instead of a module name
+module = evaluate.load("LG-Anonym/VerifiableRewardsForScalableLogicalReasoning")
+create_interface(module).launch()

packages.txt

@@ -0,0 +1 @@
+swi-prolog

requirements.txt

@@ -0,0 +1,5 @@
+git+https://github.com/huggingface/evaluate@main
+evaluate
+gradio>=5.34.2
+datasets
+tqdm