README.md

---
library_name: transformers
tags:
- llm
- code
- nlp
license: apache-2.0
language:
- en
base_model:
- meta-llama/Llama-3.2-3B-Instruct
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This model is a transformer-based neural network designed to generate safe and efficient task plans for robotic operations. 
It translates human instructions into detailed task plans in JSON format, ensuring that tasks are executed safely and efficiently. 
The model incorporates spatial relationships, safety constraints, and performance metrics to provide comprehensive and contextually relevant task plans for robotic systems.


## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->
This model is a transformer-based neural network specifically designed to generate safe and efficient task plans for robotic operations. It takes human instructions as input and outputs detailed task plans in JSON format, which include spatial relationships, task planning steps, and performance metrics. 
The model is particularly useful for applications where precise and safe robotic manipulation is required, such as in manufacturing, service robotics, and automation.

- **Developed by:** Shaanxi University of Science and Technology/Fehead
- **Funded by [optional]:** Shaanxi Provincial Department of Education Service Local Special Plan Project (22JC019)
- **Shared by [optional]:** Shaanxi University of Science and Technology/Fehead
- **Model type:** Transformer
- **Language(s) (NLP):** English
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** meta-llama/Llama-3.2-3B-Instruct

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

### Direct Use

The model can be used directly for generating task plans for robotic operations based on human instructions. This is particularly useful in scenarios where precise and safe robotic manipulation is required. The model takes a human instruction as input and outputs a detailed task plan in JSON format, which includes spatial relationships, task planning steps, and performance metrics. This makes it suitable for applications in:
Manufacturing: Automating repetitive tasks and improving efficiency.
Service Robotics: Assisting in tasks such as cleaning, object retrieval, and placement.
Automation: Enhancing the capabilities of automated systems in various industries.

### Downstream Use [optional]

The model can be fine-tuned or integrated into larger systems for more specialized applications. For example:
Custom Task Planning: Fine-tuning the model on specific tasks or environments to improve accuracy and relevance.
Robotic Control Systems: Integrating the model into robotic control systems to provide real-time task planning and execution.
Human-Robot Interaction: Enhancing interaction systems where robots need to understand and execute human instructions accurately.

### Out-of-Scope Use

While the model is designed for generating safe and efficient task plans, it is important to note its limitations and potential misuse:
Misuse: The model should not be used in scenarios where the safety constraints are not applicable or where the environment is significantly different from the training data.
Limitations: The model's performance may degrade in environments with high variability or in tasks that require highly specialized knowledge not covered in the training data.
Ethical Considerations: Users should ensure that the model is used ethically and responsibly, avoiding any scenarios that could lead to harm or unethical behavior.

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

To get started with the model, you can use the following code to load the model, generate task plans based on human instructions, and parse the output. This example demonstrates how to run the fine-tuned model locally.
Prerequisites
Ensure you have the necessary dependencies installed. You can install them using pip:

pip install transformers torch

Below is an example of how to run the fine-tuned model locally to generate task plans based on human instructions:

import os
import json
import torch
from textwrap import dedent
from typing import Dict, List, Optional, Union, Any
from functools import lru_cache

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    pipeline,
    Pipeline
)

class ModelManager:
    def __init__(self, model_name: str):
        self.model_name = model_name
        self.tokenizer = None
        self.model = None
        self.pipe = None
        
    @lru_cache(maxsize=1)
    def get_tokenizer(self):
        """Load tokenizer if not already loaded."""
        if self.tokenizer is None:
            self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)
        return self.tokenizer

    def load_model(self):
        """Load the model with proper error handling and performance optimizations."""
        try:
            if self.model is None:
                self.model = AutoModelForCausalLM.from_pretrained(
                    self.model_name,
                    torch_dtype=torch.float16,
                    device_map="auto",
                    # Add performance optimizations
                    low_cpu_mem_usage=True,
                    offload_folder="offload",
                )
                
                tokenizer = self.get_tokenizer()
                current_embed_size = self.model.get_input_embeddings().weight.size(0)
                if len(tokenizer) != current_embed_size:
                    self.model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=8)
                
            return self.model
        except Exception as e:
            raise RuntimeError(f"Failed to load model: {str(e)}") from e

    def get_pipeline(self) -> Pipeline:
        """Get or create the generation pipeline with optimized settings."""
        if self.pipe is None:
            model = self.load_model()
            tokenizer = self.get_tokenizer()
            self.pipe = pipeline(
                task="text-generation",
                model=model,
                tokenizer=tokenizer,
                max_new_tokens=1000,  # Increased for longer responses
                return_full_text=False,
                # Add performance settings
                batch_size=1,
                num_workers=4,
            )
        return self.pipe

    def create_prompt(self, user_input: str) -> str:
        """Create a formatted prompt for the model."""
        prompt = dedent(f"""
        Instruction:
        {user_input}
        
        Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.
        """)

        messages = [
            {
                "role": "system",
                "content": "Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.",
            },
            {"role": "user", "content": prompt},
        ]
        
        return self.get_tokenizer().apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )

    def parse_json_response(self, text: str) -> Dict[str, Any]:
        """Parse the model output into a JSON object."""
        try:
            # Clean up the text to extract just the JSON part
            # Find the first '{' and last '}'
            start_idx = text.find('{')
            end_idx = text.rfind('}') + 1
            
            if start_idx == -1 or end_idx == 0:
                raise ValueError("No JSON object found in response")
                
            json_str = text[start_idx:end_idx]
            
            # Parse the JSON
            return json.loads(json_str)
        except json.JSONDecodeError as e:
            raise ValueError(f"Failed to parse JSON: {str(e)}\nOriginal text: {text}")

    def generate(self, prompt: str) -> Dict[str, Any]:
        """Generate and parse response for a given prompt."""
        try:
            pipe = self.get_pipeline()
            outputs = pipe(prompt)
            raw_response = outputs[0]["generated_text"]
            # return self.parse_json_response(raw_response)
            return raw_response
        except Exception as e:
            raise RuntimeError(f"Generation failed: {str(e)}") from e

def save_response(response: Dict[str, Any], filename: str = "response.json"):
    """Save the response to a JSON file."""
    with open(filename, 'w') as f:
        json.dump(response, f, indent=2)

def main():
    model_name = "gabrielkyebambo/Llama-3.2-3B-Instruct-TaskPlans"
    manager = ModelManager(model_name)
    
    try:
        while True:
            user_input = input('Enter prompt (or "quit" to exit): ')
            if user_input.lower() == 'quit':
                break
                
            prompt = manager.create_prompt(user_input)
            print(f'\nFormatted prompt:\n{prompt}\n')
            
            response = manager.generate(prompt)
            
            # Save response to file
            save_response(response)
            
            # Pretty print the response
            print("\nGenerated task plan (formatted):")
            print(json.dumps(response, indent=2))
            
    except KeyboardInterrupt:
        print("\nExiting gracefully...")
    except Exception as e:
        print(f"An error occurred: {str(e)}")

if __name__ == "__main__":
    main()

## Training Details

### Training Data

The model was trained on a dataset consisting of 28,715 human instructions paired with corresponding task plans. The dataset is divided into training, validation, and test sets as follows:
Training set size: 22,972 examples
Validation set size: 4,594 examples
Test set size: 1,149 examples

Each example in the dataset includes the following features:
instruction: Human instruction in natural language.
task_plan: Corresponding task plan in JSON format.
text: Combined text of instruction and task plan.
token_count: Number of tokens in the text.

### Training Procedure

Preprocessing
Tokenization: Human instructions and task plans were tokenized using the Hugging Face tokenizer.
Normalization: Text data was normalized to ensure consistency.
Augmentation: Additional synthetic data was generated to cover edge cases and improve robustness.

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

- **Training regime:**
- fp16 mixed precision
Batch size: 4 (per device)
Evaluation batch size: 1 (per device)
Gradient accumulation steps: 2
Optimizer: paged_adamw_8bit
Learning rate: 3e-4
Number of epochs: 3
Max sequence length: 1024
Evaluation strategy: steps
Evaluation steps: 0.2 (20% of training steps)
Save steps: 0.2 (20% of training steps)
Logging steps: 10
Warmup ratio: 0.1
Save total limit: 2
Learning rate scheduler: cosine
Reporting: tensorboard
Save safetensors: True
Max gradient norm: 0.5
Dataset kwargs:
add_special_tokens: False
append_concat_token: False

#### Speeds, Sizes, Times [optional]

Training time: 27 hours
Hardware used: Single GPU (4060 TI 16GB)
Checkpoint size: [Specify Checkpoint Size]
Total parameters: 3,261,402,112
Trainable parameters: 48,627,712 (1.4910% of total parameters)

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

Testing Data
The model was evaluated on a separate test dataset consisting of 1,149 human instructions and corresponding task plans.

Energy consumption: [Specify Average Energy Consumption]

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->
Metrics
Accuracy: 98
Safety Compliance: 95
Efficiency Metrics: 97


[More Information Needed]

### Results

Human Instruction: "go to the dining room, pick up the mug from the dining table, then take it to the kitchen and put it inside the sink"

Formatted prompt:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 02 Feb 2025

Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.<|eot_id|><|start_header_id|>user<|end_header_id|>

Instruction:
go to the dinning room, pick up the mug from the dinning table, then take it to the kitchen and put it inside the sink

Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{
  "spatial_relationships": {
    "object": "mug",
    "source": {
      "location": "dining room",
      "container_or_surface": "dining table",
      "height": "0.05m",
      "weight_range": [0.2, 0.3],
      "grip_type": "power"
    },
    "destination": {
      "location": "kitchen",
      "container_or_surface": "sink",
      "height": "0.95m",
      "weight_range": [0.1, 0.2],
      "grip_type": "power"
    },
    "navigation_constraints": {
      "min_obstacle_distance": "0.7m",
      "max_velocity": "1.0m/s",
      "max_acceleration": "1.0m/s²"
    },
    "task_constraints": {
      "success_criteria": [
        "navigation_completed_safely",
        "object_grasped_securely",
        "object_transported_safely",
        "object_placed_correctly"
      ],
      "safety_constraints": {
        "collision_avoidance": "active",
        "force_limits": {
          "grasp_force": "24.08N",
          "transport_force": "17.17N",
          "placement_force": "2.45N"
        },
        "height_constraints": {
          "min_height": "0.05m",
          "max_height": "0.95m"
        },
        "orientation_constraints": {
          "object_stability": "high",
          "table_surface_stability": "high"
        }
      }
    },
    "performance_metrics": {
      "success_criteria": [
        "navigation_completed_safely",
        "object_grasped_securely",
        "object_transported_safely",
        "object_placed_correctly"
      ],
      "safety_metrics": {
        "collision_avoidance": "verified",
        "force_limits_maintained": "verified",
        "height_constraints_maintained": "verified",
        "orientation_constraints_maintained": "verified"
      },
      "estimated_metrics": {
        "total_distance": "13.6m",
        "estimated_time": "59s",
        "energy_consumption": "18.3Wh"
      },
      "critical_monitoring": [
        "object_tracking",
        "force_feedback",
        "obstacle_detection"
      ]
    }
  }
}

[More Information Needed]

#### Summary
This model provides a robust solution for generating task plans for robotic operations, ensuring safety and efficiency. It is suitable for applications in manufacturing, service robotics, and automation. For more detailed information, refer to the full model card.
The dataset for the model is currently being expanded to include a broader range of human instructions for robotic operations. This expansion aims to enhance the model's ability to handle more diverse and complex tasks in human-robot collaboration scenarios. 
The new dataset will cover various manufacturing tasks, including assembly, disassembly, and maintenance operations, to improve the model's generalization and adaptability

## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** NVIDIA 4060 TI 16GB GPU.
- **Hours used:** > 27 hours
- **Cloud Provider:** Trained locally
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

@article{qi2024safety,
  title={Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs},
  author={Qi, Yong and Kyebambo, Gabriel and Xie, Siyuan and Shen, Wei and Wang, Shenghui and Xie, Bitao and He, Bin and Wang, Zhipeng and Jiang, Shuo},
  journal={arXiv preprint arXiv:2405.17846},
  year={2024}
}

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[email protected]