README.md

metadata

title: MLRC-BENCH
emoji: 📊
colorFrom: green
colorTo: blue
sdk: streamlit
sdk_version: 1.39.0
app_file: app.py
pinned: false
license: cc-by-4.0

Overview

This application provides a visual leaderboard for comparing AI model performance on challenging Machine Learning Research Competition problems. It uses Streamlit to create an interactive web interface with filtering options, allowing users to select specific models and tasks for comparison.

The leaderboard uses the MLRC-BENCH benchmark, which measures what percentage of the top human-to-baseline performance gap an agent can close. Success is defined as achieving at least 5% of the margin by which the top human solution surpasses the baseline.

Key Features

• Interactive Filtering: Select specific model types and tasks to focus on
• Customizable Metrics: Compare models using "Margin to Human" performance scores
• Hierarchical Table Display: Fixed columns with scrollable metrics section
• Conditional Formatting: Visual indicators for positive/negative values
• Model Type Color Coding: Different colors for Open Source, Open Weights, and Closed Source models
• Medal Indicators: Top-ranked models receive gold, silver, and bronze medals
• Task Descriptions: Detailed explanations of what each task measures

Project Structure

The codebase follows a modular architecture for improved maintainability and separation of concerns:

app.py (main entry point)
├── requirements.txt
└── src/
    ├── app.py (main application logic)
    ├── components/
    │   ├── header.py (header and footer components)
    │   ├── filters.py (filter selection components)
    │   ├── leaderboard.py (leaderboard table component)
    │   └── tasks.py (task descriptions component)
    ├── data/
    │   ├── processors.py (data processing utilities)
    │   └── metrics/
    │       └── margin_to_human.json (metric data file)
    ├── styles/
    │   ├── base.py (combined styles)
    │   ├── components.py (component styling)
    │   ├── tables.py (table-specific styling)
    │   └── theme.py (theme definitions)
    └── utils/
        ├── config.py (configuration settings)
        └── data_loader.py (data loading utilities)

Module Descriptions

Core Files

• app.py (root): Simple entry point that imports and calls the main function
• src/app.py: Main application logic, coordinates the overall flow

Components

• header.py: Manages the page header, section headers, and footer components
• filters.py: Handles metric, task, and model type selection interfaces
• leaderboard.py: Renders the custom HTML leaderboard table
• tasks.py: Renders the task descriptions section

Data Processing

• processors.py: Contains utilities for data formatting and styling
• data_loader.py: Functions for loading and processing metric data

Styling

• theme.py: Base theme definitions and color schemes
• components.py: Styling for UI components (buttons, cards, etc.)
• tables.py: Styling for tables and data displays
• base.py: Combines all styles for application-wide use

Configuration

• config.py: Contains all configuration settings including themes, metrics, and model categorizations

Benefits of Modular Architecture

The modular structure provides several advantages:

1. Improved Code Organization: Code is logically separated based on functionality
2. Better Separation of Concerns: Each module has a clear, single responsibility
3. Enhanced Maintainability: Changes to one aspect don't require modifying the entire codebase
4. Simplified Testing: Components can be tested independently
5. Easier Collaboration: Multiple developers can work on different parts simultaneously
6. Cleaner Entry Point: Main app file is simple and focused

Installation & Setup

1. Clone the repository

   git clone <repository-url>
   cd model-capability-leaderboard
   

2. Install the required dependencies

   pip install -r requirements.txt
   

3. Run the application

   streamlit run app.py
   

Extending the Application

Adding New Metrics

To add a new metric:

1. Create a new JSON data file in the src/data/metrics/ directory (e.g., src/data/metrics/new_metric.json)

2. Update metrics_config in src/utils/config.py:

   metrics_config = {
       "Margin to Human": { ... },
       "New Metric Name": {
           "file": "src/data/metrics/new_metric.json",
           "description": "Description of the new metric",
           "min_value": 0,
           "max_value": 100,
           "color_map": "viridis"
       }
   }
   

3. Ensure your metric JSON file follows the same format as existing metrics:

   {
     "task-name": {
       "model-name-1": value,
       "model-name-2": value
     },
     "another-task": {
       "model-name-1": value,
       "model-name-2": value
     }
   }
   

Adding New Model Types

To add new model types:

1. Update model_categories in src/utils/config.py:

   model_categories = {
       "Existing Model": "Category",
       "New Model Name": "New Category"
   }
   

Modifying the UI Theme

To change the theme colors:

1. Update the dark_theme dictionary in src/utils/config.py

Adding New Components

To add new visualization components:

1. Create a new file in the src/components/ directory
2. Import and use the component in src/app.py

Data Format

The application uses JSON files for metric data. The expected format is:

{
  "task-name": {
    "model-name-1": value,
    "model-name-2": value
  },
  "another-task": {
    "model-name-1": value,
    "model-name-2": value
  }
}

Testing

This modular structure makes it easier to write focused unit tests:

# Example test for data_loader.py
def test_process_data():
    test_data = {"task": {"model": 0.5}}
    df = process_data(test_data)
    assert "Task" in df.columns
    assert df.loc["model", "Task"] == 0.5

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Contact

For any questions or feedback, please contact [email protected].