tags:
- model_hub_mixin
- pytorch_model_hub_mixin
license: other
Prompt Task/Complexity Classifier
Model Overview
This is a multi-headed model which classifies English text prompts across task types and complexity dimensions. Tasks are classified across 11 common categories. Complexity is evaluated across 6 dimensions and ensembled to create an overall complexity score. Further information on the taxonomies can be found below.
This model is ready for commercial use.
Task types:
- Open QA: A question where the response is based on general knowledge
- Closed QA: A question where the response is based on text/data provided with the prompt
- Summarization
- Text Generation
- Code Generation
- Chatbot
- Classification
- Rewrite
- Brainstorming
- Extraction
- Other
Complexity dimensions:
- Overall Complexity Score: The weighted sum of the complexity dimensions. Calculated as 0.35*CreativityScore + 0.25*ReasoningScore + 0.15*ConstraintScore + 0.15*DomainKnowledgeScore + 0.05*ContextualKnowledgeScore + 0.05*NumberOfFewShots
- Creativity: The level of creativity needed to respond to a prompt. Score range of 0-1, with a higher score indicating more creativity.
- Reasoning: The extent of logical or cognitive effort required to respond to a prompt. Score range of 0-1, with a higher score indicating more reasoning
- Contextual Knowledge: The background information necessary to respond to a prompt. Score range of 0-1, with a higher score indicating more contextual knowledge required outside of prompt.
- Domain Knowledge: The amount of specialized knowledge or expertise within a specific subject area needed to respond to a prompt. Score range of 0-1, with a higher score indicating more domain knowledge is required.
- Constraints: The number of constraints or conditions provided with the prompt. Score range of 0-1, with a higher score indicating more constraints in the prompt.
- Number of Few Shots: The number of examples provided with the prompt. Score range of 0-n, with a higher score indicating more examples provided in the prompt.
License
This model is released under the NVIDIA Open Model License Agreement.
References
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- Training language models to follow instructions with human feedback
Model Architecture
The model architecture uses a DeBERTa backbone and incorporates multiple classification heads, each dedicated to a task categorization or complexity dimension. This approach enables the training of a unified network, allowing it to predict simultaneously during inference. Deberta-v3-base can theoretically handle up to 12k tokens, but default context length is set at 512 tokens.
How to Use in NVIDIA NeMo Curator
NeMo Curator improves generative AI model accuracy by processing text, image, and video data at scale for training and customization. It also provides pre-built pipelines for generating synthetic data to customize and evaluate generative AI systems.
The inference code for this model is available through the NeMo Curator GitHub repository. Check out this example notebook to get started.
How to Use in Transformers
To use the prompt task and complexity classifier, use the following code:
# TODO
Input & Output
Input
- Input Type: Text
- Input Format: String
- Input Parameters: 1D
- Other Properties Related to Input: Token Limit of 512 tokens
Output
- Output Type: Text/Numeric Classifications
- Output Format: String & Numeric
- Output Parameters: 1D
- Other Properties Related to Output: None
Examples
Prompt: Write a mystery set in a small town where an everyday object goes missing, causing a ripple of curiosity and suspicion. Follow the investigation and reveal the surprising truth behind the disappearance.
| Task | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
|---|---|---|---|---|---|---|---|
| Text Generation | 0.472 | 0.867 | 0.056 | 0.048 | 0.226 | 0.785 | 0 |
Prompt: Antibiotics are a type of medication used to treat bacterial infections. They work by either killing the bacteria or preventing them from reproducing, allowing the body’s immune system to fight off the infection. Antibiotics are usually taken orally in the form of pills, capsules, or liquid solutions, or sometimes administered intravenously. They are not effective against viral infections, and using them inappropriately can lead to antibiotic resistance. Explain the above in one sentence.
| Task | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
|---|---|---|---|---|---|---|---|
| Summarization | 0.133 | 0.003 | 0.014 | 0.003 | 0.644 | 0.211 | 0 |
Software Integration
- Runtime Engine: Python 3.10 and NeMo Curator
- Supported Hardware Microarchitecture Compatibility: NVIDIA GPU, Volta™ or higher (compute capability 7.0+), CUDA 12 (or above)
- Preferred/Supported Operating System(s): Ubuntu 22.04/20.04
Model Version
Prompt Task and Complexity Classifier v1.1
Training, Testing, and Evaluation Datasets
Training Data
- 4024 English prompts with task distribution outlined below
- Prompts were annotated by humans according to task and complexity taxonomies
Task distribution:
| Task | Count |
|---|---|
| Open QA | 1214 |
| Closed QA | 786 |
| Text Generation | 480 |
| Chatbot | 448 |
| Classification | 267 |
| Summarization | 230 |
| Code Generation | 185 |
| Rewrite | 169 |
| Other | 104 |
| Brainstorming | 81 |
| Extraction | 60 |
| Total | 4024 |
Evaluation
For evaluation, Top-1 accuracy metric was used, which involves matching the category with the highest probability to the expected answer. Additionally, n-fold cross-validation was used to produce n different values for this metric to verify the consistency of the results. The table below displays the average of the top-1 accuracy values for the N folds calculated for each complexity dimension separately.
| Task Accuracy | Creative Accuracy | Reasoning Accuracy | Contextual Accuracy | FewShots Accuracy | Domain Accuracy | Constraint Accuracy | |
|---|---|---|---|---|---|---|---|
| Average of 10 Folds | 0.981 | 0.996 | 0.997 | 0.981 | 0.979 | 0.937 | 0.991 |
Inference
- Engine: PyTorch
- Test Hardware: A10G
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report security vulnerabilities or NVIDIA AI Concerns here.