added multiple ways to download a model

download.ipynb

@@ -0,0 +1,168 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Download models from pipeline module"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import pipeline"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "All PyTorch model weights were used when initializing TFT5ForConditionalGeneration.\n",
+      "\n",
+      "All the weights of TFT5ForConditionalGeneration were initialized from the PyTorch model.\n",
+      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.\n"
+     ]
+    }
+   ],
+   "source": [
+    "summarizer = pipeline(\"summarization\", model=\"google-t5/t5-small\", tokenizer=\"google-t5/t5-small\", truncation=True, framework=\"tf\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
+      "I0000 00:00:1725208010.799880 11454710 service.cc:146] XLA service 0x340929ca0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
+      "I0000 00:00:1725208010.799988 11454710 service.cc:154]   StreamExecutor device (0): Host, Default Version\n",
+      "2024-09-01 21:56:50.805880: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n",
+      "I0000 00:00:1725208010.830035 11454710 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[{'summary_text': 'MLflow is an open-source platform created by Databricks . it is designed to manage the four primary phases of the machine learning lifecycle . this includes experiment tracking, project packaging, model deployment, and lifecycle management . each experiment is documented and replicated in the future .'}]\n"
+     ]
+    }
+   ],
+   "source": [
+    "with open('mlflow.txt', 'r') as file:\n",
+    "    print(summarizer(file.read()))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Download Models using hf_hub_download\n",
+    "- This will store the models in your cache, and you don't have to download the model everytime you use it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import hf_hub_download"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
+      "To disable this warning, you can either:\n",
+      "\t- Avoid using `tokenizers` before the fork if possible\n",
+      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "bd7624af3b3d4a2bba2c889754d04560",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "pytorch_model.bin:   0%|          | 0.00/242M [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'/Users/seemasaharan/.cache/huggingface/hub/models--google-t5--t5-small/snapshots/df1b051c49625cf57a3d0d8d3863ed4d13564fe4/pytorch_model.bin'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "hf_hub_download(repo_id=\"google-t5/t5-small\", filename=\"pytorch_model.bin\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Download Models using Git"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!git lfs install\n",
+    "!git clone <model-URL>"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

mlflow.txt

@@ -0,0 +1,43 @@
+ Introduction to MLflow: A Comprehensive Guide to Managing the Machine Learning Lifecycle
+
+Machine learning (ML) has become an essential tool for data scientists and organizations, offering the ability to predict trends, automate decision-making, and gain insights from vast amounts of data. However, the lifecycle of developing, deploying, and managing machine learning models is complex, involving multiple stages and a variety of tools. MLflow, an open-source platform created by Databricks, addresses these challenges by providing an integrated environment for managing the end-to-end machine learning lifecycle.
+
+ Overview of MLflow
+
+MLflow is designed to manage the four primary phases of the machine learning lifecycle: experiment tracking, project packaging, model deployment, and lifecycle management. These components are modular, meaning that they can be used independently or together, depending on the needs of the project.
+
+1. MLflow Tracking: This component allows users to log and query experiments, storing parameters, metrics, and artifacts (such as models or data) associated with each run. This is crucial for reproducibility, as it ensures that every experiment is documented and can be replicated in the future. MLflow Tracking supports a variety of storage backends, including local files, SQL databases, and cloud storage, making it versatile for different deployment environments.
+
+2. MLflow Projects: A project is a way to package data science code in a reusable and reproducible manner. MLflow Projects use standardized formats, typically a directory containing code and a configuration file (`MLproject`), which defines dependencies and execution commands. This modularity promotes collaboration across teams, as projects can be easily shared and run in various environments with consistent results.
+
+3. MLflow Models: Once a model is trained, it needs to be packaged and deployed for inference. MLflow Models standardize how models are stored and deployed. A model in MLflow can be saved in different formats such as TensorFlow, PyTorch, Scikit-learn, and more, allowing for seamless integration with existing frameworks. Furthermore, MLflow Models facilitate the deployment of models as a REST API or within a cloud environment, making the transition from experimentation to production smoother and more efficient.
+
+4. MLflow Model Registry: The Model Registry component provides a central repository for managing the lifecycle of ML models. It allows users to version models, transition models through different stages (e.g., staging, production), and maintain a detailed history of models, including who trained them and when. This component is essential for organizations that need to manage multiple models in production, ensuring that only the best-performing models are deployed.
+
+ Benefits of Using MLflow
+
+MLflow offers several key benefits that make it an attractive option for managing the machine learning lifecycle:
+
+1. Reproducibility: By tracking experiments and standardizing the packaging of projects and models, MLflow ensures that experiments can be reproduced with consistent results. This is vital for both research and production, where reproducibility guarantees the reliability and validity of models.
+
+2. Scalability: MLflow is designed to be scalable, supporting a wide range of deployment environments from local development setups to large-scale cloud platforms. This flexibility allows teams to scale their ML operations as needed, whether they are working on small projects or large enterprise applications.
+
+3. Interoperability: MLflow supports a wide variety of machine learning frameworks and languages, making it a versatile tool for teams that use different tools and technologies. This interoperability reduces the friction between different stages of the ML lifecycle and allows for seamless integration into existing workflows.
+
+4. Collaboration: With features like the Model Registry and the standardized format of MLflow Projects, teams can collaborate more effectively. Models and experiments can be shared, reviewed, and iterated upon by different members of a team, ensuring that the collective knowledge and efforts are harnessed to produce the best possible outcomes.
+
+5. Model Governance: The ability to version and manage models through the MLflow Model Registry ensures that organizations can maintain strict control over their models in production. This is particularly important for industries with regulatory requirements, where knowing which model was used for a particular decision can be crucial.
+
+ Challenges and Considerations
+
+While MLflow provides a robust platform for managing the machine learning lifecycle, there are challenges and considerations to keep in mind:
+
+1. Learning Curve: For teams unfamiliar with MLflow, there is a learning curve associated with understanding and implementing its components. However, once mastered, the benefits typically outweigh the initial investment of time and effort.
+
+2. Integration with Existing Tools: Although MLflow is designed to be interoperable, integrating it with an organization’s existing infrastructure and tools can require significant effort. Custom integrations or plugins may be necessary for specific use cases.
+
+3. Cost of Infrastructure: Depending on the scale at which MLflow is deployed, the associated infrastructure costs can be significant. Organizations need to carefully consider the cost-benefit ratio when deploying MLflow, particularly if they require high availability and scalability.
+
+ Conclusion
+
+MLflow has emerged as a powerful tool for managing the machine learning lifecycle, offering features that address the challenges of reproducibility, scalability, and collaboration. By providing an integrated environment for tracking experiments, packaging projects, deploying models, and managing model lifecycles, MLflow enables organizations to streamline their ML operations and accelerate their path to production. While there are challenges in adopting and integrating MLflow, the benefits it offers make it a valuable asset for data science teams looking to optimize their machine learning workflows.