Update README.md

README.md

@@ -18,97 +18,78 @@ pinned: false
 <!-- The classes below are necessary for correct rendering -->
 <div class="lg:col-span-3">
   <img src="https://raw.githubusercontent.com/NCAI-Research/CALM/main/assets/logo.png" width="380" alt="CALM Logo" />
-  <p class="mb-2" style="font-size:64px">
+  <p class="h1 mb-2">
     CALM: Collaborative Arabic Language Model
   </p>
   <p class="mb-2">
-    The CALM project is joint effort lead by <u><a href="https://sdaia.gov.sa/ncai/?Lang=en">NCAI</a></u> in collaboration with <u><a href="https://yandex.com/">Yandex</a> and <a href="https://huggingface.co/">HuggingFace</a></u> to train an Arabic language model with volunteers from around the globe. The project is an adaptation of the framework proposed at the NeurIPS 2021 demonstration: <u><a href="https://huggingface.co/training-transformers-together">Training Transformers Together</a></u>.
-    TODO
-    In this demo, we train a model similar to <u><a target="_blank" href="https://openai.com/blog/dall-e/">OpenAI DALL-E</a></u> —
-    a Transformer "language model" that generates images from text descriptions.
-    Training happens collaboratively — volunteers from all over the Internet contribute to the training using hardware available to them.
-    We use <u><a target="_blank" href="https://laion.ai/laion-400-open-dataset/">LAION-400M</a></u>,
-    the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on
-    the <u><a target="_blank" href="https://github.com/lucidrains/DALLE-pytorch">dalle‑pytorch</a></u> implementation
-    by <u><a target="_blank" href="https://github.com/lucidrains">Phil Wang</a></u> with a few tweaks to make it communication-efficient.
+    The CALM project is joint effort lead by <u><a target="_blank"  href="https://sdaia.gov.sa/ncai/?Lang=en">NCAI</a></u> in collaboration with 
+    <u><a target="_blank"  href="https://yandex.com/">Yandex</a> and <a href="https://huggingface.co/">HuggingFace</a></u> to train an Arabic language model with 
+    volunteers from around the globe. The project is an adaptation of the framework proposed at the NeurIPS 2021 demonstration: 
+    <u><a target="_blank"  href="https://huggingface.co/training-transformers-together">Training Transformers Together</a></u>.
   </p>
   <p class="mb-2">
-    See details about how to join and how it works on <u><a target="_blank" href="https://training-transformers-together.github.io/">our website</a></u>.
+  One of the main obstacles facing many researchers in the Arabic NLP community is the lack of computing resources that are needed for training large models. Models with 
+  leading performane on Arabic NLP tasks, such as <a target="_blank" href="https://github.com/aub-mind/arabert">AraBERT</a>, 
+  <a href="https://github.com/CAMeL-Lab/CAMeLBERT" target="_blank" >CamelBERT</a>, 
+  <a href="https://huggingface.co/aubmindlab/araelectra-base-generator" target="_blank" >AraELECTRA</a>, and <a href="https://huggingface.co/qarib">QARiB</a>, 
+  took days to train on TPUs. In the spirit of democratization of AI and community enabling, a core value at NCAI, CALM aims to demonstrate the effectiveness 
+  of collaborative training and form a community of volunteers for ANLP researchers with basic level cloud GPUs who wish to train their own models collaboratively.
   </p>
   <p class="mb-2">
-    This organization gathers people participating in the collaborative training and provides links to the necessary resources:
+    CALM trains a single BERT model on a dataset that combines MSA, Oscar and Arabic Wikipedia, and dialectal data for the gulf region from existing open source datasets. 
+    Each volunteer GPU trains the model locally at its own pace on a portion of the dataset while another portion is being streamed in the background to reduces local 
+    memory consumption. Computing the gradients and aggregating them is performed in a distributed manner, based on the computing abilities of each participating 
+    volunteer. Details of the distributed training process are further described in the paper 
+    <a target="_blank"  href="https://papers.nips.cc/paper/2021/hash/41a60377ba920919939d83326ebee5a1-Abstract.html">Deep Learning in Open Collaborations</a>.
   </p>
+  
+  <p class="h2 mb-2">
+    How to participate in training?
+  </p>
+  <p class="mb-2">
+  To join the collaborative training, all you have to do is to keep a notebook running for at <b>least 15 minutes</b>, you're free to close it after that and join again 
+  in another time. There are few steps before running the notebook:
+  </p>
+  
   <ul class="mb-2">
-    <li>👉 Starter kits for <u><a target="_blank" href="https://colab.research.google.com/drive/1BqTWcfsvNQwQqqCRKMKp1_jvQ5L1BhCY?usp=sharing">Google Colab</a></u>    and <u><a target="_blank" href="https://www.kaggle.com/yhn112/training-transformers-together/">Kaggle</a></u> (easy way to join the training)</li>
-    <li>👉 <u><a target="_blank" href="https://huggingface.co/spaces/training-transformers-together/Dashboard">Dashboard</a></u> (the current training state: loss, number of peers, etc.)</li>
-    <li>👉 <u><a target="_blank" href="https://colab.research.google.com/drive/1Vkb-4nhEEH1a5vrKtpL4MTNiUTPdpPUl?usp=sharing">Colab notebook for running inference</a></u>
-    <li>👉 <u><a target="_blank" href="https://huggingface.co/training-transformers-together/dalle-demo-v1">Model weights</a></u> (the latest checkpoint)</li></li>
-    <li>👉 Weights & Biases plots for <u><a target="_blank" href="https://wandb.ai/learning-at-home/dalle-hivemind/runs/3l7q56ht">aux peers</a></u> (aggregating the metrics) and actual <u><a target="_blank" href="https://wandb.ai/learning-at-home/dalle-hivemind-trainers">trainers</a></u> (contributing with their GPUs)</li>
-    <li>👉 <u><a target="_blank" href="https://github.com/learning-at-home/dalle-hivemind">Code</a></u></li>
-    <li>👉 <u><a target="_blank" href="https://huggingface.co/datasets/laion/laion_100m_vqgan_f8">Dataset</a></u></li>
+    <li>👉 Create an account on <a href="https://huggingface.co">Huggingface</a>.</li>
+    <li>👉 Join the <a href="https://huggingface.co/CALM">NCAI-CALM Organization</a> on Huggingface through the invitation link shared with you by email.</li>
+    <li>👉 Get your Access Token, it's later required in the notebook.
+      <ul class="mb-2">
+        <li>👉 Go to your <a href="https://huggingface.co">HF account</a></li>
+        <li>👉 Go to Settings ⇒ Access Tokens</li>
+        <li>👉 Generate a new Access Token and enter any name for "what's this token for"</li>
+        <li>👉 Select <code>read</code> role</li>
+        <li>👉 Copy your access token</li>
+        <li>👉 Paste it in the execution prompt in the notebook</li>
+      </ul>
+    </li>
   </ul>
+  
+  <p class="h2 mb-2">
+    Start training
+  </p>
+  <p class="mb-2">Pick one of the following methods to run the training code.
+  <br /><em>NOTE: Kaggle gives you around 40 hrs per week of GPU time, so it's preferred over Colab, unless you have Colab Pro or Colab Pro+.</em></p>
+  <ul class="mb-2">
+    <li><span><a href="https://www.kaggle.com/prmais/volunteer-gpu-notebook">
+    <img style="display:inline;margin:0px" src="https://img.shields.io/badge/kaggle-Open%20in%20Kaggle-blue.svg"/>
+    </a></span> <b> (recommended)</b> <br />
+    </li>
+    <li> <span><a href="https://colab.research.google.com/github/NCAI-Research/CALM/blob/main/notebooks/volunteer-gpu-notebook.ipynb">
+    <img style="display:inline;margin:0px" src="https://colab.research.google.com/assets/colab-badge.svg"/>
+    </a></span>
+    </li>
+    <li>Running locally
+    <br />If you have additional local computing GPUs, please visit our discord channel for instructions to set it.
+    </li>
+  </ul>
+  
+  <p class="h2 mb-2">
+    Issues or questions?
+  </p>
+  
   <p class="mb-2">
-    Feel free to reach us on <u><a target="_blank" href="https://discord.gg/uGugx9zYvN">Discord</a></u> if you have any questions 🙂
+    Feel free to reach us on <u><a target="_blank" href="https://discord.gg/vRNN9ua2">Discord</a></u> if you have any questions 🙂
   </p>
 </div>
-
-
-<div class="lg:col-span-3">
-
-
-
-<!-- this is actually not markdown, please use regular HTML -->
-
-# 
-
-
-Once of the main obstacles facing many researchers in the Arabic NLP community is the lack of computing resources that are needed for training large models. Models with leading performane on Arabic NLP tasks, such as <a href="https://github.com/aub-mind/arabert">AraBERT</a>, <a href="https://github.com/CAMeL-Lab/CAMeLBERT">CamelBERT</a>, <a href="https://huggingface.co/aubmindlab/araelectra-base-generator">AraELECTRA</a>, and <a href="https://huggingface.co/qarib">QARiB</a>, took days to train on TPUs. In the spirit of democratization of AI and community enabling, a core value at NCAI, CALM aims to demonstrate the effectiveness of collaborative training and form a community of volunteers for ANLP researchers with basic level cloud GPUs who wish to train their own models collaboratively.
-
-CALM trains a single BERT model on a dataset that combines MSA, Oscar and Arabic Wikipedia, and dialectal data for the gulf region from existing open source datasets. Each volunteer GPU trains the model locally at its own pace on a portion of the dataset while another portion is being streamed in the background to reduces local memory consumption. Computing the gradients and aggregating them is performed in a distributed manner, based on the computing abilities of each participating volunteer. Details of the distributed training process are further described in the paper <a href="https://papers.nips.cc/paper/2021/hash/41a60377ba920919939d83326ebee5a1-Abstract.html">Deep Learning in Open Collaborations</a>.
-
-<h2>How to participate in training?</h2>
-
-To join the collaborative training, all you have to do is to keep a notebook running for at <b>least 15 minutes</b>, you're free to close it after that and join again in another time. There are few steps before running the notebook:
-
-<ol class="list-decimal">
-<li>Create an account on <a href="https://huggingface.co">Huggingface</a>.</li>
-<li>Join the <a href="https://huggingface.co/CALM">NCAI-CALM Organization</a> on Huggingface through the invitation link shared with you by email.</li>
-<li>Get your Access Token, it's later required in the notebook
-<ol>
-<li>Go to your <a href="https://huggingface.co">HF account</a></li>
-<li>Go to Settings ⇒ Access Tokensv</li>
-<li>Generate a new Access Token and enter any name for "what's this token for"</li>
-<li>Select <code>read</code> role</li>
-<li>Copy your access token</li>
-<li>Paste it in the execution prompt in the notebook</li>
-</ol>
-</li>
-</ol>
-
-<h2>Start training</h2>
-
-<p>Pick one of the following methods to run the training code.
-<br /><em>NOTE: Kaggle gives you around 40 hrs per week of GPU time, so it's preferred over Colab, unless you have Colab Pro or Colab Pro+.</em></p>
-
-<ul class="list-decimal">
-  <li>
-  <span><a href="https://www.kaggle.com/prmais/volunteer-gpu-notebook">
-  <img style="display:inline;margin:0px" src="https://img.shields.io/badge/kaggle-Open%20in%20Kaggle-blue.svg"/>
-  </a></span> <b> (recommended)</b> <br />
-  </li>
-  <li> 
-  <span><a href="https://colab.research.google.com/github/NCAI-Research/CALM/blob/main/notebooks/volunteer-gpu-notebook.ipynb">
-  <img style="display:inline;margin:0px" src="https://colab.research.google.com/assets/colab-badge.svg"/>
-  </a></span>
-  </li>
-  <li><b>Running locally</b>
-  <br />If you have additional local computing GPUs, please visit our discord channel for instructions to set it.
-  </li>
-</ul>
-
-<h2>Issues or questions?</h2>
-We are there to provide any assistance needed, please make sure to join our <span><a href="https://discord.gg/vRNN9ua2">
-  <img style="display:inline;margin:0px" src="https://badgen.net/badge/icon/discord?icon=discord&label"/>
-  </a></span>
-
-</div>