Update README.md

README.md

@@ -1,76 +1,306 @@
+---
+license: cc
+datasets:
+- vector-institute/newsmediabias-plus
+language:
+- en
+metrics:
+- accuracy
+- precision
+base_model:
+- meta-llama/Llama-3.2-11B-Vision-Instruct
+---
 
-# Llama3.2 NLP Bias Classifier
 
-This model merges the base Llama-3.2 architecture with a custom adapter to classify text for disinformation likelihood, leveraging NLP techniques for high accuracy in distinguishing manipulative content from unbiased sources. It focuses on detecting rhetorical techniques commonly used in disinformation, offering both 'Likely' and 'Unlikely' classifications based on structured indicators.
+# LLaMA 3.2 Multimodal News Media Bias Detector
+
+
+## Model Overview
+
+The LLaMA 3.2 Multimodal News Media Bias Detector is a fine-tuned version of the LLaMA 2 Vision-Instruct model, designed to assess potential disinformation in news media articles by analyzing both text and image content. The model identifies the presence of rhetorical techniques commonly associated with disinformation and classifies articles as 'Likely' or 'Unlikely' to be disinformation.
 
 ## Model Details
 
-- **Base Model**: [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Lla
-- **Deployment Environment**: Configured for GPU (CUDA) support.
-- **Training Data**: [vector-institute/newsmediabias-plus](https://huggingface.co/datasets/vector-institute/newsmediabias-plus)
-- **Sampled Data to test inference**: [sample data](https://huggingface.co/vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector/blob/main/sampled-data/sample_dataset.csv)
+- **Base Model**: [meta-llama/Llama-3.2-11B-Vision-Instruct](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct)
+- **Architecture**: Multimodal (Text and Image)
+- **Model Size**: 11B parameters
+- **Quantization**: 4-bit quantization (`nf4`) using `bitsandbytes` for efficient inference
+- **Framework**: [Hugging Face Transformers](https://github.com/huggingface/transformers)
+- **Fine-tuning**: The model was fine-tuned using the [PEFT](https://github.com/huggingface/peft) library with LoRA adapters
+
+## Intended Use
+
+### Primary Use Cases
+
+- **Disinformation Detection**: Analyze news articles and accompanying images to detect potential disinformation.
+- **Media Analysis**: Assist researchers and analysts in studying bias and rhetorical techniques in media content.
+- **Educational Tool**: Serve as a resource for learning about multimodal models and disinformation detection.
+
+### Limitations
+
+- **Not a Fact-Checker**: The model does not verify factual accuracy but assesses rhetorical techniques.
+- **Bias in Training Data**: The model's judgments may reflect biases present in the training data.
+- **Language and Domain Limitations**: Primarily trained on English-language news articles; performance may degrade on other languages or domains.
+
+## How to Use
+First access through huggingface cli
+
+``` ! huggingface-cli login ```
+
+**Sampled Data Usage**
+
+Please use the sampled data available at either of the following repositories for testing or development purposes with the LLaMA 3.2 Multimodal News Media Bias Detector:
+
+- [Sampled Data on Hugging Face - LLaMA 3.2 Multimodal News Media Bias Detector](https://huggingface.co/vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector/tree/main/sampled-data)
+- [NewsMediaBias-Plus Dataset on Hugging Face](https://huggingface.co/datasets/vector-institute/newsmediabias-plus)
+
+These datasets are pre-configured to be compatible with the model and include a variety of news articles and images that have been annotated for potential disinformation, allowing for effective model evaluation and demonstration.
+
 
-## Model Usage
+### Installation
+
+Ensure you have the required libraries installed:
+
+```
+pip install transformers accelerate bitsandbytes
+```
+```
+### Loading the Model
 
-```python
 import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig
+
+merged_model_save_path= "vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector"
+# Load quantization config if used during saving
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+
+# Load the merged model
+model = AutoModelForVision2Seq.from_pretrained(
+    merged_model_save_path,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    quantization_config=bnb_config  
+)
+model.eval()
+
+# Load the processor
+processor = AutoProcessor.from_pretrained(merged_model_save_path)
+
 import pandas as pd
+import os
+from PIL import Image
 
-# Load tokenizer and model
-tokenizer = AutoTokenizer.from_pretrained("save_directory")
-model = AutoModelForCausalLM.from_pretrained("save_directory", torch_dtype=torch.float16).to(device)
-```
+# Path to your dataset CSV and image folder
+dataset_csv_path = 'sample_dataset.csv'
+image_folder_path = 'sampled_images'
 
-### Generating Predictions
+# Load the DataFrame
+df = pd.read_csv(dataset_csv_path)
 
-The model evaluates text for disinformation by identifying rhetorical techniques. To classify input text, use the `generate_response` function:
+# Function to prepare samples from the DataFrame
+def prepare_samples_from_dataframe(df, image_folder_path):
+    samples = []
+    for index, row in df.iterrows():
+        unique_id = row['unique_id']  # Replace with the column that contains the image identifier
+        text_content = row['first_paragraph']  # Replace with your text column name
 
-```python
-def generate_response(model, prompt):
-    inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=1024).to(device)
-    outputs = model.generate(inputs['input_ids'], max_new_tokens=50, temperature=0.7, top_p=0.95)
-    return tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
-```
+        # Find the image file
+        possible_extensions = ['jpg', 'jpeg', 'png']
+        image_path = None
+        for ext in possible_extensions:
+            img_path = os.path.join(image_folder_path, f"{unique_id}.{ext}")
+            if os.path.exists(img_path):
+                image_path = img_path
+                break
 
-## Dataset and Evaluation
+        if image_path is None:
+            print(f"No image found for ID {unique_id}")
+            continue
 
-- **Input Dataset**: Sample data from `sample_dataset.csv` containing balanced examples of 'Likely' and 'Unlikely' disinformation.
-- **Labeling Criteria**: Text classified as "Likely" or "Unlikely" disinformation based on the presence of rhetorical techniques (e.g., exaggeration, emotional appeal).
-- **Metrics**: Precision, recall, F1 score, and accuracy, computed with `sklearn.metrics`.
+        image = Image.open(image_path).convert("RGB")
+        max_size = (224, 224)
+        image.thumbnail(max_size, Image.Resampling.LANCZOS)
 
-### Model Performance
+        # Prepare the prompt text
+        sample_text = (
+            "Assess the text and image below for potential disinformation (try finding deliberately misleading or biased information) by identifying the presence of rhetorical techniques listed.\n"
+            "If you find any of the listed rhetorical techniques, then the article is likely disinformation; if not, it is likely not disinformation.\n\n"
+            "Rhetorical Techniques Checklist:\n"
+            "- Emotional Appeal: Uses language that intentionally invokes extreme emotions like fear or anger, aiming to distract from lack of factual backing.\n"
+            "- Exaggeration and Hyperbole: Makes claims that are unsupported by evidence, or presents normal situations as extraordinary to manipulate perceptions.\n"
+            "- Bias and Subjectivity: Presents information in a way that unreasonably favors one perspective, omitting key facts that might provide balance.\n"
+            "- Repetition: Uses repeated messaging of specific points or misleading statements to embed a biased viewpoint in the reader's mind.\n"
+            "- Specific Word Choices: Employs emotionally charged or misleading terms to sway opinions subtly, often in a manipulative manner.\n"
+            "- Appeals to Authority: References authorities who lack relevant expertise or cites sources that do not have the credentials to be considered authoritative in the context.\n"
+            "- Lack of Verifiable Sources: Relies on sources that either cannot be verified or do not exist, suggesting a fabrication of information.\n"
+            "- Logical Fallacies: Engages in flawed reasoning such as circular reasoning, strawman arguments, or ad hominem attacks that undermine logical debate.\n"
+            "- Conspiracy Theories: Propagates theories that lack proof and often contain elements of paranoia or implausible scenarios as facts.\n"
+            "- Inconsistencies and Factual Errors: Contains multiple contradictions or factual inaccuracies that are easily disprovable, indicating a lack of concern for truth.\n"
+            "- Selective Omission: Deliberately leaves out crucial information that is essential for a fair understanding of the topic, skewing perception.\n"
+            "- Manipulative Framing: Frames issues in a way that leaves out alternative perspectives or possible explanations, focusing only on aspects that support a biased narrative.\n\n"
+            f"{text_content}\n\n"
+            "Please **only** provide your answer in the format: 'Classification: Likely' or 'Classification: Unlikely'. Do not include any additional text or explanation."
+        )
 
+        sample = {
+            "unique_id": unique_id,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": [
+                        {
+                            "type": "text",
+                            "text": sample_text,
+                        },
+                        {
+                            "type": "image",
+                            "image": image,
+                        }
+                    ],
+                }
+            ]
+        }
 
-| Label          | Precision | Recall | F1 Score |
-|----------------|-----------|--------|----------|
-| Unlikely (0)   | 78%       | 82%    | 79.95%   |
-| Likely (1)     | 81%       | 85%    | 82.95%   |
-| **Accuracy**   |           |        | 87%      |
+        samples.append(sample)
 
-### Example Classification
+    return samples
 
-```plaintext
-Example 1:
-Text: "This new vaccine causes severe side effects in a majority of patients, which is something the authorities don’t want you to know."
-Actual Label: Likely (1)
-Model Prediction: Likely (1)
-```
+import re
+
+def extract_assistant_reply(generated_text):
+    lines = generated_text.strip().split('\n')
+    for idx, line in enumerate(lines):
+        if line.strip().lower() == 'assistant':
+            assistant_reply = '\n'.join(lines[idx+1:]).strip()
+            return assistant_reply
+    return lines[-1].strip()
+
+def extract_classification(assistant_reply):
+    match = re.search(r'Classification:\s*(Likely|Unlikely)', assistant_reply, re.IGNORECASE)
+    if match:
+        return match.group(1).capitalize()
+    else:
+        # Attempt to find 'Likely' or 'Unlikely' anywhere in the reply
+        match = re.search(r'\b(Likely|Unlikely)\b', assistant_reply, re.IGNORECASE)
+        if match:
+            return match.group(1).capitalize()
+        else:
+            return 'Unknown'
+
+def generate_prediction(sample):
+    # Remove the model.to(device) line
+    # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    # model.to(device)
 
-## Limitations and Future Work
+    # Prepare the input texts and images
+    texts = processor.apply_chat_template(sample["messages"], tokenize=False)
+    image_input = sample["messages"][0]['content'][1]['image']
 
-- **False Positives**: May misclassify subjective statements lacking explicit disinformation techniques.
-- **Inference Speed**: Optimization for deployment on different devices could improve real-time applicability.
+    # Prepare the inputs without moving them to a device
+    inputs = processor(text=texts, images=image_input, return_tensors="pt", padding=True)
 
-## Citation
+    # Ensure that the inputs are on the same device as the model's embeddings
+    # This is handled internally by the model when using device_map="auto"
+
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=20,
+            do_sample=False,
+            num_beams=1,
+        )
+
+    generated_texts = processor.batch_decode(outputs, skip_special_tokens=True)
+    assistant_reply = extract_assistant_reply(generated_texts[0])
+
+    return assistant_reply
+
+# Prepare the samples from the DataFrame
+samples = prepare_samples_from_dataframe(df, image_folder_path)
+
+# Limit to a subset if desired
+samples_to_infer = samples[:5]  # For example, take the first 5 samples
+
+# Run inference and collect results
+results = []
+
+for sample in samples_to_infer:
+    assistant_reply = generate_prediction(sample)
+    predicted_label = extract_classification(assistant_reply)
+
+    # Collect results
+    result = {
+        'unique_id': sample['unique_id'],
+        'assistant_reply': assistant_reply,
+        'predicted_label': predicted_label,
+        # Add any other fields you need from the sample or DataFrame
+    }
+    results.append(result)
+
+    # Display the results
+    print(f"Sample ID: {sample['unique_id']}")
+    print("Assistant's Reply:")
+    print(assistant_reply)
+    print(f"Predicted Label: {predicted_label}")
+    print("-" * 50)
+
+results_df = pd.DataFrame(results)
+# Save to CSV if desired
+results_df.to_csv('inference_results.csv', index=False)
+```
+### Example Output
 
-If you use this model, please cite our work as follows:
 ```
-@inproceedings{Raza2024LlamaBiasClassifier,
-  title={Llama3.2 NLP Bias Classifier for Disinformation Detection},
-  author={Shaina Raza},
-  year={2024}
-}
+Assistant's Reply:
+Classification: Likely
 ```
 
-For more information, contact Shaina Raza, PhD at [email protected]
+## Training Data
+
+The model is fine-tuned on [NewsMediaBias-Plus Dataset on Hugging Face](https://huggingface.co/datasets/vector-institute/newsmediabias-plus)
+
+
+## Training Procedure
+
+- **Fine-Tuning**: The model was fine-tuned using the PEFT library with LoRA adapters to efficiently adapt the base model to the disinformation detection task.
+- **Quantization**: 4-bit quantization was applied using `bitsandbytes` to optimize inference performance.
+- **Hyperparameters**: Standard hyperparameters were used, with adjustments made for optimal performance on the custom dataset.
+
+## Evaluation
+
+The model was evaluated on a balanced test set with the following metrics:
+
+- **Accuracy**: *88%*
+- **Precision**: *78%*
+- **Recall**: *75%*
+
+*Note: Replace the above metrics with actual results from your evaluations.*
+
+## Limitations and Biases
+
+- **Biases in Data**: The model's performance and judgments are influenced by the biases present in the training data.
+- **False Positives/Negatives**: The model may incorrectly classify some articles due to subtle rhetorical techniques or lack thereof.
+- **Ethical Considerations**: Users should be cautious when interpreting the model's outputs and consider them as part of a broader analysis.
+
+## Ethical Considerations
+
+- **User Responsibility**: The model should not be used as the sole basis for making critical decisions.
+- **Data Privacy**: Ensure compliance with data privacy laws when using the model with user data.
+- **Misuse Prevention**: The model should not be used to unjustly label or discriminate against individuals or groups.
+
+
+
+## License
+
+This model is licensed under CC BY-NC-SA 4.0.
+
+Developed by Dr. Shaina Raza at the Vector Institute. Contact: [[email protected]](mailto:[email protected])
+
+**Disclaimer**: The model is provided "as is" without any warranty. The developers are not responsible for any consequences arising from the use of this model.
+
+```