Update README.md

README.md

@@ -1,199 +1,231 @@
 ---
-library_name: transformers
-tags: []
+license: other
+license_name: nvidia-open-model-license
+license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
 ---
 
-# Model Card for Model ID
+# Model Overview
 
-<!-- Provide a quick summary of what the model is/does. -->
 
+[[**Github**](https://github.com/NVlabs/RADIO)]  [[**CVPR 2025**](https://arxiv.org/abs/2412.07679)]  [[**CVPR 2024**](https://arxiv.org/abs/2312.06709)]
 
 
-## Model Details
+## Description
 
-### Model Description
+This model performs visual feature extraction.
+For instance, RADIO generates image embeddings that can be used by a downstream model to classify images.
 
-<!-- Provide a longer summary of what this model is. -->
+C-RADIOv2 models are available in multiple sizes:
+* Base (90M parameters).
+* Large (320M parameters).
+* Huge (653M parameters).
+* Gigantic (1.1B parameters).
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+C-RADIOv2 was trained for 1M steps (400k more steps than v1), using inverse frequency sampling for data balancing, and [PHI Standardization](https://arxiv.org/abs/2410.01680) for teacher distribution balancing.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+This model is ready for commercial/non-commercial use.
 
-### Model Sources [optional]
+### License/Terms of Use
 
-<!-- Provide the basic links for the model. -->
+GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+## Deployment Geography
 
-## Uses
+Global.
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+## Use Case
 
-### Direct Use
+The embeddings generated by this model are expected to be used by a downstream application.
+For example:
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+* Image-level understanding (image classification, curation, etc.).
+* Dense processing (semantic segmentation, depth estimation, etc.).
+* Integration into a Vision-Language Model.
 
-[More Information Needed]
+## Release Date
 
-### Downstream Use [optional]
+Huggingface: 03/26/2025 via [RADIO Collection of Models](https://huggingface.co/collections/nvidia/radio-669f77f1dd6b153f007dd1c6).
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+## References
 
-[More Information Needed]
+* \[CVPR 2025\] [**RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models**](https://arxiv.org/abs/2412.07679)
+* \[CVPR 2024\] [**AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One**](https://arxiv.org/abs/2312.06709)
 
-### Out-of-Scope Use
+## Model Architecture
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+**Architecture Type:** Neural Network  <br>
+**Network Architecture:** Vision Transformer <br>
 
-[More Information Needed]
+## Input
 
-## Bias, Risks, and Limitations
+**Input Type(s):** Image <br>
+**Input Format(s):** Red, Green, Blue (RGB) <br>
+**Input Parameters:** Two Dimensional (2D) <br>
+**Other Properties Related to Input:** Image resolutions up to 2048x2028 in increments of 16 pixels <br>
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## Output
 
-[More Information Needed]
+**Output Type(s):** Embeddings <br>
+**Output Format:** Tensor <br>
+**Output Parameters:** 2D <br>
+**Other Properties Related to Output:** Downstream model required to leverage image features <br>
 
-### Recommendations
+## Usage:
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+RADIO will return a tuple with two tensors.
+The `summary` is similar to the `cls_token` in ViT and is meant to represent the general concept of the entire image.
+It has shape `(B,C)` with `B` being the batch dimension, and `C` being some number of channels.
+The `spatial_features` represent more localized content which should be suitable for dense tasks such as semantic segmentation, or for integration into an LLM.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, CLIPImageProcessor
 
-## How to Get Started with the Model
+hf_repo = "nvidia/C-RADIOv2-L"
 
-Use the code below to get started with the model.
+image_processor = CLIPImageProcessor.from_pretrained(hf_repo)
+model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)
+model.eval().cuda()
 
-[More Information Needed]
+image = Image.open('./assets/radio.png').convert('RGB')
+pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values
+pixel_values = pixel_values.cuda()
 
-## Training Details
+summary, features = model(pixel_values)
+```
 
-### Training Data
+Spatial features have shape `(B,T,D)` with `T` being the flattened spatial tokens, and `D` being the channels for spatial features. Note that `C!=D` in general.
+Converting to a spatial tensor format can be done using the downsampling size of the model, combined with the input tensor shape. For RADIO, the patch size is 16.
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+```Python
+from einops import rearrange
+spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)
+```
 
-[More Information Needed]
+The resulting tensor will have shape `(B,D,H,W)`, as is typically seen with computer vision models.
 
-### Training Procedure
+## Software Integration
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+**Runtime Engine(s):**
+* TAO- 24.10 <br>
 
-#### Preprocessing [optional]
+**Supported Hardware Microarchitecture Compatibility:** <br>
+* NVIDIA Ampere <br>
+* NVIDIA Blackwell <br>
+* NVIDIA Jetson  <br>
+* NVIDIA Hopper <br>
+* NVIDIA Lovelace <br>
+* NVIDIA Pascal <br>
+* NVIDIA Turing <br>
+* NVIDIA Volta <br>
 
-[More Information Needed]
+**[Preferred/Supported] Operating System(s):** <br>
+* Linux
+* Linux 4 Tegra
+* QNX
+* Windows
 
+## Model Version(s)
 
-#### Training Hyperparameters
+* C-RADIOv2-B (90M parameters).
+* C-RADIOv2-L (320M parameters).
+* C-RADIOv2-H (653M parameters).
+* C-RADIOv2-G (1.8B parameters).
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+**Links:**
 
-#### Speeds, Sizes, Times [optional]
+* https://huggingface.co/nvidia/C-RADIOv2-B
+* https://huggingface.co/nvidia/C-RADIOv2-L
+* https://huggingface.co/nvidia/C-RADIOv2-H
+* https://huggingface.co/nvidia/C-RADIOv2-g
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+# Training and Evaluation Datasets
 
-[More Information Needed]
+## Training Dataset
 
-## Evaluation
+NV-CC-Img-Text-Dataset  <br>
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+### Data Collection Method by dataset
 
-### Testing Data, Factors & Metrics
+* Automated <br>
 
-#### Testing Data
+### Labeling Method by dataset
 
-<!-- This should link to a Dataset Card if possible. -->
+* Not Applicable (no labels are needed)
 
-[More Information Needed]
+### Properties
 
-#### Factors
+* 700 Million Images <br>
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+## Evaluation Dataset
 
-[More Information Needed]
+**Link:** [ImageNet](https://www.image-net.org/)  <br>
 
-#### Metrics
+### Data Collection Method by dataset
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+* Automated <br>
 
-[More Information Needed]
+### Labeling Method by dataset
 
-### Results
+* Human <br>
 
-[More Information Needed]
+**Properties:** This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images.<br>
 
-#### Summary
+## Inference
 
+**Engine:** PyTorch <br>
+**Test Hardware:** A100 <br>
 
+## Ethical Considerations
 
-## Model Examination [optional]
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
 
-<!-- Relevant interpretability work for the model goes here -->
+For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards below.
 
-[More Information Needed]
+Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
 
-## Environmental Impact
+### Bias
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Field                                                                                               |  Response
+:---------------------------------------------------------------------------------------------------|:---------------
+Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing:  |  None
+Measures taken to mitigate against unwanted bias:                                                   |  None
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+### Explainability
 
-## Technical Specifications [optional]
+Field                                                                                                  |  Response
+:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------
+Intended Application & Domain:                                                                   |  Visual Feature Extraction
+Model Type:                                                                                            |  Vision Transformer
+Intended Users:                                                                                        |  Developers of downstream vision applications
+Output:                                                                                                |  Image embeddings
+Describe how the model works:                                                                          |  The model takes an image as input, processes the image through multiple transformer blocks, and outputs summary and patch embeddings.
+Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:  |  Not Applicable
+Technical Limitations:                                                                                 |  This model generates image embeddings that can be used by a downstream model to, for example, classify images. The downstream model must be trained to leverage the visual embeddings.
+Verified to have met prescribed NVIDIA quality standards:  |  Yes
+Performance Metrics:                                                                                   |  Image classification accuracy, semantic segmentation mean-over-intersection.
+Potential Known Risks:                                                                                 |  This model is only tested on input resolutions ranging from 256 to 2048, in increments of 16 pixels. Additionally, the generated embeddings might fail to disambiguate differences that appear evident to humans (e.g. two images showing different breeds of dogs might in fact produce very similar embeddings). Domain-specific evaluation is required for the target application.
+Licensing:                                                                                             |  [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf)
 
-### Model Architecture and Objective
 
-[More Information Needed]
+### Privacy
 
-### Compute Infrastructure
+Field                                                                                                                              |  Response
+:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
+Generatable or reverse engineerable personal data?                                                     |  None
+Personal data used to create this model?                                                                                       |  None
+How often is dataset reviewed?                                                                                                     |  Before Every Release
+Is there provenance for all datasets used in training?                                                                                |  Yes
+Does data labeling (annotation, metadata) comply with privacy laws?                                                                |  Yes
+Is data compliant with data subject requests for data correction or removal, if such a request was made?                           |  Yes
 
-[More Information Needed]
+### Safety
 
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+Field                                               |  Response
+:---------------------------------------------------|:----------------------------------
+Model Application(s):                               |  Generation of visual embeddings
+Describe the life critical impact (if present).   |  Not Applicable
+Use Case Restrictions:                              |  Abide by NVIDIA Open Model License Agreement
+Model and dataset restrictions:            |  The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.