Update README.md

README.md

@@ -1,12 +1,23 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- service robotics
+- llm
+- taskplan generation
+- llama instruct taskplans
+license: apache-2.0
+language:
+- en
+base_model:
+- meta-llama/Llama-3.2-3B-Instruct
 ---
 
 # Model Card for Model ID
 
 <!-- Provide a quick summary of what the model is/does. -->
-
+This model is a transformer-based neural network designed to generate safe and efficient task plans for robotic operations. 
+It translates human instructions into detailed task plans in JSON format, ensuring that tasks are executed safely and efficiently. 
+The model incorporates spatial relationships, safety constraints, and performance metrics to provide comprehensive and contextually relevant task plans for robotic systems.
 
 
 ## Model Details
@@ -14,16 +25,16 @@ tags: []
 ### Model Description
 
 <!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+This model is a transformer-based neural network specifically designed to generate safe and efficient task plans for robotic operations. It takes human instructions as input and outputs detailed task plans in JSON format, which include spatial relationships, task planning steps, and performance metrics. 
+The model is particularly useful for applications where precise and safe robotic manipulation is required, such as in manufacturing, service robotics, and automation.
+
+- **Developed by:** Shaanxi University of Science and Technology/Fehead
+- **Funded by [optional]:** Shaanxi Provincial Department of Education Service Local Special Plan Project (22JC019)
+- **Shared by [optional]:** Shaanxi University of Science and Technology/Fehead
+- **Model type:** Transformer
+- **Language(s) (NLP):** English
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Finetuned from model [optional]:** meta-llama/Llama-3.2-3B-Instruct
 
 ### Model Sources [optional]
 
@@ -35,23 +46,26 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+The model can be used directly for generating task plans for robotic operations based on human instructions. This is particularly useful in scenarios where precise and safe robotic manipulation is required. The model takes a human instruction as input and outputs a detailed task plan in JSON format, which includes spatial relationships, task planning steps, and performance metrics. This makes it suitable for applications in:
+Manufacturing: Automating repetitive tasks and improving efficiency.
+Service Robotics: Assisting in tasks such as cleaning, object retrieval, and placement.
+Automation: Enhancing the capabilities of automated systems in various industries.
 
 ### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+The model can be fine-tuned or integrated into larger systems for more specialized applications. For example:
+Custom Task Planning: Fine-tuning the model on specific tasks or environments to improve accuracy and relevance.
+Robotic Control Systems: Integrating the model into robotic control systems to provide real-time task planning and execution.
+Human-Robot Interaction: Enhancing interaction systems where robots need to understand and execute human instructions accurately.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+While the model is designed for generating safe and efficient task plans, it is important to note its limitations and potential misuse:
+Misuse: The model should not be used in scenarios where the safety constraints are not applicable or where the environment is significantly different from the training data.
+Limitations: The model's performance may degrade in environments with high variability or in tasks that require highly specialized knowledge not covered in the training data.
+Ethical Considerations: Users should ensure that the model is used ethically and responsibly, avoiding any scenarios that could lead to harm or unethical behavior.
 
 [More Information Needed]
 
@@ -69,21 +83,187 @@ Users (both direct and downstream) should be made aware of the risks, biases and
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+To get started with the model, you can use the following code to load the model, generate task plans based on human instructions, and parse the output. This example demonstrates how to run the fine-tuned model locally.
+Prerequisites
+Ensure you have the necessary dependencies installed. You can install them using pip:
+
+pip install transformers torch
+
+Below is an example of how to run the fine-tuned model locally to generate task plans based on human instructions:
+
+import os
+import json
+import torch
+from textwrap import dedent
+from typing import Dict, List, Optional, Union, Any
+from functools import lru_cache
+
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    pipeline,
+    Pipeline
+)
+
+class ModelManager:
+    def __init__(self, model_name: str):
+        self.model_name = model_name
+        self.tokenizer = None
+        self.model = None
+        self.pipe = None
+        
+    @lru_cache(maxsize=1)
+    def get_tokenizer(self):
+        """Load tokenizer if not already loaded."""
+        if self.tokenizer is None:
+            self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)
+        return self.tokenizer
+
+    def load_model(self):
+        """Load the model with proper error handling and performance optimizations."""
+        try:
+            if self.model is None:
+                self.model = AutoModelForCausalLM.from_pretrained(
+                    self.model_name,
+                    torch_dtype=torch.float16,
+                    device_map="auto",
+                    # Add performance optimizations
+                    low_cpu_mem_usage=True,
+                    offload_folder="offload",
+                )
+                
+                tokenizer = self.get_tokenizer()
+                current_embed_size = self.model.get_input_embeddings().weight.size(0)
+                if len(tokenizer) != current_embed_size:
+                    self.model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=8)
+                
+            return self.model
+        except Exception as e:
+            raise RuntimeError(f"Failed to load model: {str(e)}") from e
+
+    def get_pipeline(self) -> Pipeline:
+        """Get or create the generation pipeline with optimized settings."""
+        if self.pipe is None:
+            model = self.load_model()
+            tokenizer = self.get_tokenizer()
+            self.pipe = pipeline(
+                task="text-generation",
+                model=model,
+                tokenizer=tokenizer,
+                max_new_tokens=1000,  # Increased for longer responses
+                return_full_text=False,
+                # Add performance settings
+                batch_size=1,
+                num_workers=4,
+            )
+        return self.pipe
+
+    def create_prompt(self, user_input: str) -> str:
+        """Create a formatted prompt for the model."""
+        prompt = dedent(f"""
+        Instruction:
+        {user_input}
+        
+        Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.
+        """)
+
+        messages = [
+            {
+                "role": "system",
+                "content": "Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.",
+            },
+            {"role": "user", "content": prompt},
+        ]
+        
+        return self.get_tokenizer().apply_chat_template(
+            messages, tokenize=False, add_generation_prompt=True
+        )
+
+    def parse_json_response(self, text: str) -> Dict[str, Any]:
+        """Parse the model output into a JSON object."""
+        try:
+            # Clean up the text to extract just the JSON part
+            # Find the first '{' and last '}'
+            start_idx = text.find('{')
+            end_idx = text.rfind('}') + 1
+            
+            if start_idx == -1 or end_idx == 0:
+                raise ValueError("No JSON object found in response")
+                
+            json_str = text[start_idx:end_idx]
+            
+            # Parse the JSON
+            return json.loads(json_str)
+        except json.JSONDecodeError as e:
+            raise ValueError(f"Failed to parse JSON: {str(e)}\nOriginal text: {text}")
+
+    def generate(self, prompt: str) -> Dict[str, Any]:
+        """Generate and parse response for a given prompt."""
+        try:
+            pipe = self.get_pipeline()
+            outputs = pipe(prompt)
+            raw_response = outputs[0]["generated_text"]
+            # return self.parse_json_response(raw_response)
+            return raw_response
+        except Exception as e:
+            raise RuntimeError(f"Generation failed: {str(e)}") from e
+
+def save_response(response: Dict[str, Any], filename: str = "response.json"):
+    """Save the response to a JSON file."""
+    with open(filename, 'w') as f:
+        json.dump(response, f, indent=2)
+
+def main():
+    model_name = "gabrielkyebambo/Llama-3.2-3B-Instruct-TaskPlans"
+    manager = ModelManager(model_name)
+    
+    try:
+        while True:
+            user_input = input('Enter prompt (or "quit" to exit): ')
+            if user_input.lower() == 'quit':
+                break
+                
+            prompt = manager.create_prompt(user_input)
+            print(f'\nFormatted prompt:\n{prompt}\n')
+            
+            response = manager.generate(prompt)
+            
+            # Save response to file
+            save_response(response)
+            
+            # Pretty print the response
+            print("\nGenerated task plan (formatted):")
+            print(json.dumps(response, indent=2))
+            
+    except KeyboardInterrupt:
+        print("\nExiting gracefully...")
+    except Exception as e:
+        print(f"An error occurred: {str(e)}")
+
+if __name__ == "__main__":
+    main()
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was trained on a dataset consisting of 28,715 human instructions paired with corresponding task plans. The dataset is divided into training, validation, and test sets as follows:
+Training set size: 22,972 examples
+Validation set size: 4,594 examples
+Test set size: 1,149 examples
 
-[More Information Needed]
+Each example in the dataset includes the following features:
+instruction: Human instruction in natural language.
+task_plan: Corresponding task plan in JSON format.
+text: Combined text of instruction and task plan.
+token_count: Number of tokens in the text.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Preprocessing
+Tokenization: Human instructions and task plans were tokenized using the Hugging Face tokenizer.
+Normalization: Text data was normalized to ensure consistency.
+Augmentation: Additional synthetic data was generated to cover edge cases and improve robustness.
 
 #### Preprocessing [optional]
 
@@ -92,11 +272,36 @@ Use the code below to get started with the model.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:**
+- fp16 mixed precision
+Batch size: 4 (per device)
+Evaluation batch size: 1 (per device)
+Gradient accumulation steps: 2
+Optimizer: paged_adamw_8bit
+Learning rate: 3e-4
+Number of epochs: 3
+Max sequence length: 1024
+Evaluation strategy: steps
+Evaluation steps: 0.2 (20% of training steps)
+Save steps: 0.2 (20% of training steps)
+Logging steps: 10
+Warmup ratio: 0.1
+Save total limit: 2
+Learning rate scheduler: cosine
+Reporting: tensorboard
+Save safetensors: True
+Max gradient norm: 0.5
+Dataset kwargs:
+add_special_tokens: False
+append_concat_token: False
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+Training time: 27 hours
+Hardware used: Single GPU (4060 TI 16GB)
+Checkpoint size: [Specify Checkpoint Size]
+Total parameters: 3,261,402,112
+Trainable parameters: 48,627,712 (1.4910% of total parameters)
 
 [More Information Needed]
 
@@ -108,7 +313,10 @@ Use the code below to get started with the model.
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
+Testing Data
+The model was evaluated on a separate test dataset consisting of 1,149 human instructions and corresponding task plans.
+
+Energy consumption: [Specify Average Energy Consumption]
 
 [More Information Needed]
 
@@ -121,16 +329,110 @@ Use the code below to get started with the model.
 #### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
+Metrics
+Accuracy: 98
+Safety Compliance: 95
+Efficiency Metrics: 97
+
 
 [More Information Needed]
 
 ### Results
 
+Human Instruction: "go to the dining room, pick up the mug from the dining table, then take it to the kitchen and put it inside the sink"
+
+Formatted prompt:
+<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+Cutting Knowledge Date: December 2023
+Today Date: 02 Feb 2025
+
+Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+Instruction:
+go to the dinning room, pick up the mug from the dinning table, then take it to the kitchen and put it inside the sink
+
+Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+{
+  "spatial_relationships": {
+    "object": "mug",
+    "source": {
+      "location": "dining room",
+      "container_or_surface": "dining table",
+      "height": "0.05m",
+      "weight_range": [0.2, 0.3],
+      "grip_type": "power"
+    },
+    "destination": {
+      "location": "kitchen",
+      "container_or_surface": "sink",
+      "height": "0.95m",
+      "weight_range": [0.1, 0.2],
+      "grip_type": "power"
+    },
+    "navigation_constraints": {
+      "min_obstacle_distance": "0.7m",
+      "max_velocity": "1.0m/s",
+      "max_acceleration": "1.0m/s²"
+    },
+    "task_constraints": {
+      "success_criteria": [
+        "navigation_completed_safely",
+        "object_grasped_securely",
+        "object_transported_safely",
+        "object_placed_correctly"
+      ],
+      "safety_constraints": {
+        "collision_avoidance": "active",
+        "force_limits": {
+          "grasp_force": "24.08N",
+          "transport_force": "17.17N",
+          "placement_force": "2.45N"
+        },
+        "height_constraints": {
+          "min_height": "0.05m",
+          "max_height": "0.95m"
+        },
+        "orientation_constraints": {
+          "object_stability": "high",
+          "table_surface_stability": "high"
+        }
+      }
+    },
+    "performance_metrics": {
+      "success_criteria": [
+        "navigation_completed_safely",
+        "object_grasped_securely",
+        "object_transported_safely",
+        "object_placed_correctly"
+      ],
+      "safety_metrics": {
+        "collision_avoidance": "verified",
+        "force_limits_maintained": "verified",
+        "height_constraints_maintained": "verified",
+        "orientation_constraints_maintained": "verified"
+      },
+      "estimated_metrics": {
+        "total_distance": "13.6m",
+        "estimated_time": "59s",
+        "energy_consumption": "18.3Wh"
+      },
+      "critical_monitoring": [
+        "object_tracking",
+        "force_feedback",
+        "obstacle_detection"
+      ]
+    }
+  }
+}
+
 [More Information Needed]
 
 #### Summary
-
-
+This model provides a robust solution for generating task plans for robotic operations, ensuring safety and efficiency. It is suitable for applications in manufacturing, service robotics, and automation. For more detailed information, refer to the full model card.
+The dataset for the model is currently being expanded to include a broader range of human instructions for robotic operations. This expansion aims to enhance the model's ability to handle more diverse and complex tasks in human-robot collaboration scenarios. 
+The new dataset will cover various manufacturing tasks, including assembly, disassembly, and maintenance operations, to improve the model's generalization and adaptability
 
 ## Model Examination [optional]
 
@@ -144,9 +446,9 @@ Use the code below to get started with the model.
 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
+- **Hardware Type:** NVIDIA 4060 TI 16GB GPU.
+- **Hours used:** > 27 hours
+- **Cloud Provider:** Trained locally
 - **Compute Region:** [More Information Needed]
 - **Carbon Emitted:** [More Information Needed]
 
@@ -174,7 +476,12 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 **BibTeX:**
 
-[More Information Needed]
+@article{qi2024safety,
+  title={Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs},
+  author={Qi, Yong and Kyebambo, Gabriel and Xie, Siyuan and Shen, Wei and Wang, Shenghui and Xie, Bitao and He, Bin and Wang, Zhipeng and Jiang, Shuo},
+  journal={arXiv preprint arXiv:2405.17846},
+  year={2024}
+}
 
 **APA:**
 
@@ -196,4 +503,4 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Contact
 
-[More Information Needed]
+[email protected]