Update README.md

README.md

@@ -1,199 +1,153 @@
 ---
+language:
+- en
 library_name: transformers
-tags: []
+license: cc-by-4.0
+tags:
+- kl3m
+- kl3m-004
+- correction
+- legal
+- financial
+- enterprise
+- slm
+date: '2024-02-20T00:00:00.000Z'
+pipeline_tag: text-generation
+widget:
+ - text: "Tne Uni+ed 5tates is nct responsib|e for the<|sep|>"
+ - temperature: 0.3
+ - do_sample: True
 ---
 
-# Model Card for Model ID
+# kl3m-004-correction-001 Model
 
-<!-- Provide a quick summary of what the model is/does. -->
+kl3m-004-correction-001 is a small, ~500M parameter language model model designed to assist in the correction of common typing, spelling,
+OCR, and format issues in English text, especially in the financial and legal domains.
 
+Notably, this model has been trained on the [alea-institute/kl3m-004-char-8k-cased](https://huggingface.co/alea-institute/kl3m-004-char-8k-cased) tokenizer, which
+is a BPE tokenizer trained with a 3-character maximum token constraint.  
 
+This model was originally trained 3 days on 1xRTX3090, and a large ~3B parameter MoE is pending release.
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## How to Get Started with the Model
 
-Use the code below to get started with the model.
+## Getting Started
 
-[More Information Needed]
+Simply prompt the model with the original text, followed by the `<|sep|>` token, and wait for stop token (`<|end|>`) generation.  You can use `pipeline` to handle this for you.
 
-## Training Details
 
-### Training Data
+### Deterministic
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+In many situations, deterministic correction (i.e., most probable logit sequence) is fine.
 
-[More Information Needed]
+```python
+from transformers import pipeline
+p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cpu')
 
-### Training Procedure
+text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+correction = p(text + "<|sep|>", max_new_tokens=512, return_full_text=False)[0]['generated_text']
 
-#### Preprocessing [optional]
+# Output: The United States is not responsible for such production
+```
 
-[More Information Needed]
+### Sampled with Frequency Weighting
 
+In other situations, it can be useful to generate multiple corrections with a sampler and evaluate the distribution.  For example:
+* using a string or token-based distance metric to score or rank corrections
+* showing multiple suggestions to a user with frequency-weighted order
 
-#### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+```python
+from transformers import pipeline
+from collections import Counter
 
-#### Speeds, Sizes, Times [optional]
+p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cuda')
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"
 
-[More Information Needed]
+corrections = Counter(
+  [
+    g['generated_text']
+    for g in p(
+      text + "<|sep|>",
+      max_new_tokens=512,
+      return_full_text=False,
+      temperature=0.5,
+      # top_p, top_k, custom sampler, etc.
+      do_sample=True,
+      num_return_sequences=10
+    )
+  ]
+).most_common(3)
 
-## Evaluation
+# Output: [('The United States is not responsible for such production', 7), ('the United States is not responsible for such production', 3)]
+```
 
-<!-- This section describes the evaluation protocols and provides the results. -->
 
-### Testing Data, Factors & Metrics
+## Source
 
-#### Testing Data
+[https://github.com/alea-institute/kl3m-model-research](https://github.com/alea-institute/kl3m-model-research)
 
-<!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+## Training Data
+This model was trained on a dataset generated with the KL3M data collection and the [https://github.com/alea-institute/alea-data-generator](alea-data-generator) library, which
+can create realistic synthetic samples using traditional (non-generative) techniques.  
 
-#### Factors
+The source code to retrieve and process this dataset is available here:
+[https://github.com/alea-institute/kl3m-data](https://github.com/alea-institute/kl3m-data)
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+Some pre-tokenized subsets of the KL3M data collection are available on Hugging Face:
+[https://huggingface.co/datasets?sort=most_rows&search=kl3m-data](https://huggingface.co/datasets?sort=most_rows&search=kl3m-data)
 
-[More Information Needed]
+Complete, raw data is available upon request at this time via S3 under a Requester Pays model.  We are actively working on a
+zero-cost distribution model as soon as we can obtain additional support.
 
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
+## Model Details
 
-## Citation [optional]
+### Summary
+- **Architecture**: LlamaForCausalLM
+- **Parameters**: 478.2M
+- **Context Window**: 512 tokens (no ROPE)
+- **Language(s)**: Primarily English
+- **Tokenizer**: kl3m-004-char-8k-cased BPE tokenizer (8K tokens, between 1-3 characters each)
+- **Developed by**: [ALEA Institute](https://aleainstitute.ai)
+- **License**: [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)
+- **Hardware Requirements**: Runs real-time in fp32 on CPU or consumer NV/AMD GPUs
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Key Features
 
-**BibTeX:**
+- **Clean Training Data**: Built on what was originally referred to as the Kelvin Legal DataPack, ensuring all training data is ethically sourced and legally permissible.
+- **Low Toxicity**: [Empirically lower toxicity and bias](https://github.com/alea-institute/kl3m-toxicity)
+- **Enterprise Focus**: Specifically designed for legal, regulatory, and financial workflows.
+- **Efficient Deployment**: Optimized for real-time inference on consumer hardware.
 
-[More Information Needed]
+## Use Cases
 
-**APA:**
+- Correcting common typing or spelling errors
+- Correcting common OCR errors
+- Correcting common formatting errors 
 
-[More Information Needed]
+## License
 
-## Glossary [optional]
+Model weights are released under the CC-BY 4.0 License.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+## Contact
 
-[More Information Needed]
+The KL3M model family is now maintained by the [ALEA Institute](https://aleainstitute.ai). For technical support, collaboration opportunities, or general inquiries:
+ 
+- GitHub: https://github.com/alea-institute/kl3m-model-research
+- Email: [email protected]
+- Website: https://aleainstitute.ai
 
-## More Information [optional]
 
-[More Information Needed]
+## Citation
 
-## Model Card Authors [optional]
+Tokenizer, dataset, and model publications are pending.
 
-[More Information Needed]
+## Contact
 
-## Model Card Contact
+For any questions, please contact [ALEA Institute](https://aleainstitute.ai) at [[email protected]](mailto:[email protected]) or
+create an issue on this repository or [GitHub](https://github.com/alea-institute/kl3m-model-research).
 
-[More Information Needed]
+![https://aleainstitute.ai](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)