Prisma-Multimodal/sae-top_k-64-cls_only-layer_1-hook_resid_post

CLIP Sparse Autoencoder Checkpoint

This model is a sparse autoencoder trained on CLIP's internal representations.

Model Details

Architecture

• Layer: 1
• Layer Type: hook_resid_post
• Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• Dictionary Size: 49152
• Input Dimension: 768
• Expansion Factor: 64
• CLS Token Only: True

Training

• Training Images: 206843904
• Learning Rate: 0.0002
• L1 Coefficient: 0.3000
• Batch Size: 4096
• Context Size: 1

Performance Metrics

Sparsity

• L0 (Active Features): 64
• Dead Features: 29224
• Mean Log10 Feature Sparsity: -9.3313
• Features Below 1e-5: 48808
• Features Below 1e-6: 42124
• Mean Passes Since Fired: 15533.3896

Reconstruction

• Explained Variance: 0.9689
• Explained Variance Std: 0.0180
• MSE Loss: 0.0000
• L1 Loss: 0
• Overall Loss: 0.0000

Training Details

• Training Duration: 17941.6724 seconds
• Final Learning Rate: 0.0002
• Warm Up Steps: 200
• Gradient Clipping: 1

Additional Information

• Weights & Biases Run: https://wandb.ai/perceptual-alignment/clip/runs/yrtlmomi
• Original Checkpoint Path: /network/scratch/s/sonia.joseph/checkpoints/clip-b
• Random Seed: 42