pages/Workflow & Model Overview.py

import streamlit as st

# Page configuration
st.set_page_config(
    layout="wide", 
    initial_sidebar_state="auto"
)

# Custom CSS for better styling
st.markdown("""

    <style>

        .main-title {

            font-size: 36px;

            color: #4A90E2;

            font-weight: bold;

            text-align: center;

        }

        .sub-title {

            font-size: 24px;

            color: #4A90E2;

            margin-top: 20px;

        }

        .section {

            background-color: #f9f9f9;

            padding: 15px;

            border-radius: 10px;

            margin-top: 20px;

        }

        .section h2 {

            font-size: 22px;

            color: #4A90E2;

        }

        .section p, .section ul {

            color: #666666;

        }

        .link {

            color: #4A90E2;

            text-decoration: none;

        }

        .benchmark-table {

            width: 100%;

            border-collapse: collapse;

            margin-top: 20px;

        }

        .benchmark-table th, .benchmark-table td {

            border: 1px solid #ddd;

            padding: 8px;

            text-align: left;

        }

        .benchmark-table th {

            background-color: #4A90E2;

            color: white;

        }

        .benchmark-table td {

            background-color: #f2f2f2;

        }

    </style>

""", unsafe_allow_html=True)

# Title
st.markdown('<div class="main-title">Introduction to DeBERTa Annotators in Spark NLP</div>', unsafe_allow_html=True)

# Subtitle
st.markdown("""

<div class="section">

    <p>DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is an advanced language model that builds upon BERT and RoBERTa, incorporating novel techniques such as disentangled attention and enhanced mask decoding. DeBERTa models excel in various NLP tasks, including text classification, token classification, masked language modeling, and question answering. Below, we provide an overview of the DeBERTa annotators for these tasks:</p>

</div>

""", unsafe_allow_html=True)

# Tabs for DeBERTa Annotators
tab1, tab2, tab3, tab4 = st.tabs(["DeBERTa for Token Classification", "DeBERTa for Sequence Classification", "DeBERTa for Zero Shot Classification", "DeBERTa for Question Answering"])

# Tab 1: DeBERTa for Token Classification
with tab1:
    st.markdown("""

    <div class="section">

        <h2>Token Classification with Spark NLP</h2>

        <p>The <strong>Token Classification</strong> task is a core component of Natural Language Processing (NLP), focusing on classifying tokens (words or subwords) in a text into predefined categories. This task is fundamental for various applications, such as Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and more.</p>

        <p>Spark NLP offers a robust suite of tools for token classification, leveraging state-of-the-art models like BERT, RoBERTa, and DeBERTa. These models have been fine-tuned on diverse datasets and are readily available in Spark NLP to cater to a wide range of token classification tasks.</p>

        <p>Token classification with Spark NLP enables:</p>

        <ul>

            <li><strong>Named Entity Recognition (NER):</strong> Recognizing and categorizing entities such as locations (LOC), organizations (ORG), persons (PER), and more.</li>

            <li><strong>Information Extraction:</strong> Extracting structured data from unstructured text for deeper analysis and processing.</li>

            <li><strong>Text Categorization:</strong> Enhancing document retrieval, classification, and organization based on identified entities.</li>

        </ul>

        <p>Using Spark NLP for token classification tasks offers several advantages:</p>

        <ul>

            <li><strong>Scalability:</strong> Spark NLP is designed to scale seamlessly with Apache Spark, making it suitable for processing large volumes of text data efficiently.</li>

            <li><strong>Flexibility:</strong> A wide array of pre-trained models are available, allowing you to select the model that best fits your specific task, whether it’s for NER, POS tagging, or another classification task.</li>

            <li><strong>Ease of Use:</strong> Spark NLP integrates smoothly into your existing Spark pipeline, allowing for quick and easy implementation.</li>

            <li><strong>Customization:</strong> Models can be fine-tuned or adapted to new domains, giving you the flexibility to tailor the solution to your specific needs.</li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

    # General Information about Using Token Classification Models
    st.markdown('<div class="sub-title">How to Use Token Classification Models in Spark NLP</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>To perform token classification in Spark NLP, one powerful model you can use is DeBERTa, which stands for <strong>Decoding-enhanced BERT with Disentangled Attention</strong>. DeBERTa improves upon earlier models like BERT and RoBERTa, offering superior performance in tasks such as Named Entity Recognition (NER). Below is a template for setting up a token classification pipeline in Spark NLP using DeBERTa. This approach is flexible, allowing you to adjust the pipeline and parameters to meet your specific needs while leveraging DeBERTa's advanced capabilities.</p>

    </div>

    """, unsafe_allow_html=True)

    st.code('''

    from sparknlp.base import *

    from sparknlp.annotator import *

    from pyspark.ml import Pipeline

    from pyspark.sql.functions import col, expr



    document_assembler = DocumentAssembler() \\

        .setInputCol("text") \\

        .setOutputCol("document")



    tokenizer = Tokenizer() \\

        .setInputCols(["document"]) \\

        .setOutputCol("token")



    # Example of loading a token classification model (e.g., BERT, RoBERTa, DeBERTa)

    tokenClassifier = DeBertaForTokenClassification \\

        .pretrained("deberta_v3_small_token_classifier_conll03", "en") \\

        .setInputCols(["document", "token"]) \\

        .setOutputCol("ner") \\

        .setCaseSensitive(True) \\

        .setMaxSentenceLength(512)



    ner_converter = NerConverter() \\

        .setInputCols(['document', 'token', 'ner']) \\

        .setOutputCol('entities')



    pipeline = Pipeline(stages=[

        document_assembler,

        tokenizer,

        tokenClassifier,

        ner_converter

    ])



    data = spark.createDataFrame([["Spark NLP is an exceptional library for NLP tasks."]]).toDF("text")

    result = pipeline.fit(data).transform(data)

    

    result.selectExpr("explode(entities) as ner_chunk").select(

        col("ner_chunk.result").alias("chunk"),

        col("ner_chunk.metadata.entity").alias("ner_label")

    ).show(truncate=False)

    ''', language='python')

    # Results Example
    st.text("""

    +--------------------------+---------+

    |chunk                     |ner_label|

    +--------------------------+---------+

    |Spark NLP                 |ORG      |

    +--------------------------+---------+

    """)

    # Model Info Section
    st.markdown('<div class="sub-title">Choosing the Right Model</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>Spark NLP offers a variety of pre-trained models for token classification tasks, including BERT, RoBERTa, DeBERTa, and more. The choice of model can significantly impact the accuracy and performance of your task.</p>

        <p>To explore and choose the most suitable model for your specific needs, visit the <a class="link" href="https://sparknlp.org/models" target="_blank">Spark NLP Models Hub</a>. Here, you can find detailed information about each model, including its size, compatibility, and the specific tasks it excels at.</p>

    </div>

    """, unsafe_allow_html=True)

    st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <ul>

            <li><a class="link" href="https://sparknlp.org/docs/en/transformers#debertafortokenclassification" target="_blank">Transformers in Spark NLP</a></li>

        </ul>

    </div>

    """, unsafe_allow_html=True)
# Tab 2: DeBERTa for Sequence Classification
with tab2:
    st.markdown("""

    <div class="section">

        <h2>Sequence Classification with Spark NLP</h2>

        <p><strong>Sequence Classification</strong> is a critical task in Natural Language Processing (NLP) where entire sequences of text (such as sentences or paragraphs) are classified into predefined categories. This task is essential for applications like sentiment analysis, document classification, and more.</p>

        <p>Spark NLP offers robust tools for sequence classification, utilizing advanced models such as BERT, RoBERTa, and DeBERTa. These models are pre-trained on diverse datasets and are readily available within Spark NLP, enabling you to address a wide range of sequence classification challenges.</p>

        <p>Sequence classification with Spark NLP supports:</p>

        <ul>

            <li><strong>Sentiment Analysis:</strong> Determining the sentiment expressed in a sequence, such as positive, negative, or neutral.</li>

            <li><strong>Document Classification:</strong> Categorizing documents into various classes based on their content.</li>

            <li><strong>Intent Detection:</strong> Identifying the underlying intent behind a sequence, often used in chatbot and virtual assistant applications.</li>

        </ul>

        <p>Leveraging Spark NLP for sequence classification offers several advantages:</p>

        <ul>

            <li><strong>Scalability:</strong> Spark NLP scales effortlessly with Apache Spark, making it well-suited for processing large-scale text data.</li>

            <li><strong>Flexibility:</strong> A broad selection of pre-trained models is available, allowing you to choose the most appropriate model for your specific task.</li>

            <li><strong>Ease of Integration:</strong> Spark NLP integrates smoothly into existing Spark pipelines, facilitating quick and efficient implementation.</li>

            <li><strong>Customizability:</strong> Models can be fine-tuned or adapted to different domains, providing tailored solutions for specific needs.</li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

    # General Information about Using Sequence Classification Models
    st.markdown('<div class="sub-title">How to Use Sequence Classification Models in Spark NLP</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>For sequence classification in Spark NLP, one powerful model you can use is DeBERTa, which stands for <strong>Decoding-enhanced BERT with Disentangled Attention</strong>. DeBERTa offers enhanced performance compared to earlier models like BERT and RoBERTa, especially for tasks such as sentiment analysis. Below is a template for setting up a sequence classification pipeline in Spark NLP using DeBERTa. This approach is adaptable, allowing you to adjust the pipeline and parameters to suit your specific requirements while utilizing DeBERTa's advanced features.</p>

    </div>

    """, unsafe_allow_html=True)

    st.code('''

    from sparknlp.base import *

    from sparknlp.annotator import *

    from pyspark.ml import Pipeline

    from pyspark.sql.functions import col, expr



    document_assembler = DocumentAssembler() \\

        .setInputCol("text") \\

        .setOutputCol("document")



    tokenizer = Tokenizer() \\

        .setInputCols(['document']) \\

        .setOutputCol('token') 



    # Example of loading a sequence classification model using DeBERTa

    sequenceClassifier = DeBertaForSequenceClassification \\

        .pretrained("deberta_v3_base_sequence_classifier_imdb", "en") \\

        .setInputCols(["document", "token"]) \\

        .setOutputCol("class") \\

        .setCaseSensitive(True) \\

        .setMaxSentenceLength(512) 



    pipeline = Pipeline(stages=[

        document_assembler,

        tokenizer,

        sequenceClassifier

    ])



    example = spark.createDataFrame([['I really liked that movie!']]).toDF("text")

    result = pipeline.fit(example).transform(example)

    

    result.select("text", "class.result").show(truncate=False)

    ''', language='python')

    # Results Example
    st.text("""

    +------------------------------+---------+

    |text                          |class    |

    +------------------------------+---------+

    |I really liked that movie!    |positive |

    +------------------------------+---------+

    """)

    # Model Info Section
    st.markdown('<div class="sub-title">Choosing the Right Model</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>Spark NLP provides a diverse range of pre-trained models for sequence classification tasks, including BERT, RoBERTa, DeBERTa, and more. The model you choose can greatly impact the accuracy and performance of your task.</p>

        <p>To explore and select the model that best fits your specific needs, visit the <a class="link" href="https://sparknlp.org/models" target="_blank">Spark NLP Models Hub</a>. This resource offers detailed information about each model, including its size, compatibility, and the tasks it excels at.</p>

    </div>

    """, unsafe_allow_html=True)

    # References Section
    st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <ul>

            <li><a class="link" href="https://sparknlp.org/docs/en/transformers#debertaforsequenceclassification" target="_blank">Spark NLP Annotators Documentation</a></li>

            <li><a class="link" href="https://arxiv.org/abs/2006.03654" target="_blank">DeBERTa: Decoding-enhanced BERT with Disentangled Attention (Research Paper)</a></li>

            <li><a class="link" href="https://www.microsoft.com/en-us/research/project/deberta/" target="_blank">Microsoft Research: DeBERTa</a></li>

            <li><a class="link" href="https://github.com/microsoft/DeBERTa" target="_blank">DeBERTa GitHub Repository</a></li>

            <li><a class="link" href="https://paperswithcode.com/task/sequence-classification" target="_blank">Sequence Classification: Research Papers with Code</a></li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

# Tab 3: DeBERTa for Zero Shot Classification
with tab3:
    st.markdown("""

    <div class="section">

        <h2>Zero-Shot Classification with Spark NLP</h2>

        <p><strong>Zero-Shot Classification</strong> is a technique in Natural Language Processing (NLP) that allows a model to classify text into categories that it has not been explicitly trained on. This approach is particularly useful when you have new, unseen classes or labels that were not part of the training data.</p>

        <p>Spark NLP provides powerful tools for zero-shot classification, leveraging models like DeBERTa. These models are trained to handle a wide range of classification tasks without requiring retraining on specific categories. This enables flexibility and adaptability for various classification needs.</p>

        <p>Zero-shot classification with Spark NLP facilitates:</p>

        <ul>

            <li><strong>Dynamic Categorization:</strong> Classify text into new categories without additional training.</li>

            <li><strong>Adaptability:</strong> Easily adapt to evolving classification needs and emerging topics.</li>

            <li><strong>Cost Efficiency:</strong> Reduce the need for extensive retraining and model updates for new classification tasks.</li>

        </ul>

        <p>Using Spark NLP for zero-shot classification offers several benefits:</p>

        <ul>

            <li><strong>Scalability:</strong> Spark NLP integrates with Apache Spark, making it capable of handling large-scale text data efficiently.</li>

            <li><strong>Flexibility:</strong> The zero-shot classification models can be used for various tasks without the need for task-specific retraining.</li>

            <li><strong>Ease of Implementation:</strong> Seamlessly integrate zero-shot classification into existing Spark pipelines for efficient processing.</li>

            <li><strong>Customizable:</strong> Define custom candidate labels and adapt the model to different classification needs.</li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

    # General Information about Using Zero-Shot Classification Models
    st.markdown('<div class="sub-title">How to Use Zero-Shot Classification Models in Spark NLP</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>For zero-shot classification in Spark NLP, one powerful model you can use is DeBERTa, which stands for <strong>Decoding-enhanced BERT with Disentangled Attention</strong>. DeBERTa's zero-shot classification capabilities enable it to classify text into categories without additional training on those specific categories. Below is a template for setting up a zero-shot classification pipeline in Spark NLP using DeBERTa. This approach is flexible, allowing you to adjust the pipeline and parameters to fit your specific needs while leveraging DeBERTa's advanced features.</p>

    </div>

    """, unsafe_allow_html=True)

    st.code('''

    from sparknlp.base import *

    from sparknlp.annotator import *

    from pyspark.ml import Pipeline

    from pyspark.sql.functions import col, expr



    document_assembler = DocumentAssembler() \\

        .setInputCol('text') \\

        .setOutputCol('document')



    tokenizer = Tokenizer() \\

        .setInputCols(['document']) \\

        .setOutputCol('token')



    # Example of loading a zero-shot classification model using DeBERTa

    zeroShotClassifier = DeBertaForZeroShotClassification \\

        .pretrained('deberta_base_zero_shot_classifier_mnli_anli_v3', 'en') \\

        .setInputCols(['token', 'document']) \\

        .setOutputCol('class') \\

        .setCaseSensitive(True) \\

        .setMaxSentenceLength(512) \\

        .setCandidateLabels(["urgent", "mobile", "travel", "movie", "music", "sport", "weather", "technology"])



    pipeline = Pipeline(stages=[

        document_assembler,

        tokenizer,

        zeroShotClassifier

    ])



    example = spark.createDataFrame([['I have a problem with my iphone that needs to be resolved asap!!']]).toDF("text")

    result = pipeline.fit(example).transform(example)

    

    result.select("text", "class.result").show(truncate=False)

    ''', language='python')

    # Results Example
    st.text("""

    +------------------------------------------------------------+-------------+

    |text                                                       |class        |

    +------------------------------------------------------------+-------------+

    |I have a problem with my iphone that needs to be resolved asap!!|mobile     |

    +------------------------------------------------------------+-------------+

    """)

    # Model Info Section
    st.markdown('<div class="sub-title">Choosing the Right Model</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>Spark NLP offers a variety of pre-trained models for zero-shot classification, including BERT, RoBERTa, and DeBERTa. These models are capable of handling a wide range of classification tasks without requiring additional training on specific categories.</p>

        <p>To explore and select the most suitable model for your needs, visit the <a class="link" href="https://sparknlp.org/models" target="_blank">Spark NLP Models Hub</a>. Here, you can find detailed information about each model, including its size, compatibility, and the specific tasks it excels at.</p>

    </div>

    """, unsafe_allow_html=True)

    # References Section
    st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <ul>

            <li><a class="link" href="https://sparknlp.org/docs/en/transformers#distilbertforzeroshotclassification" target="_blank">Spark NLP DistilBertForZeroShotClassification Annotators Documentation</a></li>

            <li><a class="link" href="https://arxiv.org/abs/2006.11477" target="_blank">Enabling Zero-Shot Classification Using Pre-trained Transformers (Research Paper)</a></li>

            <li><a class="link" href="https://paperswithcode.com/task/zero-shot-classification" target="_blank">Zero-Shot Classification: Research Papers with Code</a></li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

# Tab 4: DeBERTa for Question Answering
with tab4:
    st.markdown("""

    <div class="section">

        <h2>Question Answering with Spark NLP</h2>

        <p><strong>Question Answering (QA)</strong> is a fundamental NLP task that involves building models capable of understanding and responding to questions based on a given context. This task is essential for applications such as chatbots, virtual assistants, and information retrieval systems.</p>

        <p>Spark NLP provides robust tools for question answering, leveraging advanced models like DeBERTa. These models are trained to accurately identify and extract answers from a provided context, enhancing the effectiveness of QA systems.</p>

        <p>Question answering with Spark NLP enables:</p>

        <ul>

            <li><strong>Automated Information Retrieval:</strong> Extracting relevant information from text to answer user queries.</li>

            <li><strong>Interactive Systems:</strong> Enhancing chatbots and virtual assistants to provide accurate responses to user questions.</li>

            <li><strong>Knowledge Extraction:</strong> Improving the ability to understand and leverage contextual information for various applications.</li>

        </ul>

        <p>Using Spark NLP for question answering offers several advantages:</p>

        <ul>

            <li><strong>Scalability:</strong> Spark NLP integrates with Apache Spark, making it suitable for handling large-scale QA tasks efficiently.</li>

            <li><strong>Flexibility:</strong> The pre-trained models can be easily adapted to various QA scenarios and domains.</li>

            <li><strong>Ease of Integration:</strong> Seamlessly integrate QA models into existing pipelines for efficient question answering.</li>

            <li><strong>Customization:</strong> Fine-tune or adapt models to specific contexts or industries to improve performance.</li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

    # General Information about Using Question Answering Models
    st.markdown('<div class="sub-title">How to Use Question Answering Models in Spark NLP</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>For question answering in Spark NLP, you can utilize DeBERTa, which stands for <strong>Decoding-enhanced BERT with Disentangled Attention</strong>. The DeBERTa model for question answering is designed to extract precise answers from a given context in response to user queries. Below is a template for setting up a question answering pipeline in Spark NLP using DeBERTa. This approach allows you to effectively manage and process question-answering tasks.</p>

    </div>

    """, unsafe_allow_html=True)

    st.code('''

    from sparknlp.base import *

    from sparknlp.annotator import *

    from pyspark.ml import Pipeline

    from pyspark.sql.functions import col, expr



    documentAssembler = MultiDocumentAssembler() \\

        .setInputCols(["question", "context"]) \\

        .setOutputCols(["document_question", "document_context"])



    spanClassifier = DebertaForQuestionAnswering \\

        .pretrained("deberta_v3_xsmall_qa_squad2", "en") \\

        .setInputCols(["document_question", "document_context"]) \\

        .setOutputCol("answer") \\

        .setCaseSensitive(True)



    pipeline = Pipeline(stages=[

        documentAssembler,

        spanClassifier

    ])



    data = spark.createDataFrame([["What is my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")

    result = pipeline.fit(data).transform(data)

    

    result.select("question", "context", "answer.result").show(truncate=False)

    ''', language='python')

    # Results Example
    st.text("""

    +------------------------------+--------------------------------------------+------------------+

    |question                     |context                                    |answer            |

    +------------------------------+--------------------------------------------+------------------+

    |What is my name?             |My name is Clara and I live in Berkeley.  |Clara             |

    +------------------------------+--------------------------------------------+------------------+

    """)

    # Model Info Section
    st.markdown('<div class="sub-title">Choosing the Right Model</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <p>Spark NLP offers a range of pre-trained models for question answering tasks, including DeBERTa and other advanced transformers. Selecting the right model can significantly impact the quality of your QA system.</p>

        <p>To explore and select the most appropriate model for your QA needs, visit the <a class="link" href="https://sparknlp.org/models" target="_blank">Spark NLP Models Hub</a>. Here, you can find detailed information about each model, including its capabilities and performance.</p>

    </div>

    """, unsafe_allow_html=True)

    # References Section
    st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
    st.markdown("""

    <div class="section">

        <ul>

            <li><a class="link" href="https://sparknlp.org/docs/en/transformers#debertaforsequenceclassification" target="_blank">Transformers in Spark NLP</a></li>

            <li><a class="link" href="https://arxiv.org/abs/2005.00979" target="_blank">SQuAD2.0: The Stanford Question Answering Dataset (Research Paper)</a></li>

            <li><a class="link" href="https://paperswithcode.com/task/question-answering" target="_blank">Question Answering: Research Papers with Code</a></li>

        </ul>

    </div>

    """, unsafe_allow_html=True)

st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)

# Footer
st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>

        <li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>

        <li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>

        <li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>

    </ul>

</div>

""", unsafe_allow_html=True)

st.markdown('<div class="sub-title">Quick Links</div>', unsafe_allow_html=True)

st.markdown("""

<div class="section">

    <ul>

        <li><a class="link" href="https://sparknlp.org/docs/en/quickstart" target="_blank">Getting Started</a></li>

        <li><a class="link" href="https://nlp.johnsnowlabs.com/models" target="_blank">Pretrained Models</a></li>

        <li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp/tree/master/examples/python/annotation/text/english" target="_blank">Example Notebooks</a></li>

        <li><a class="link" href="https://sparknlp.org/docs/en/install" target="_blank">Installation Guide</a></li>

    </ul>

</div>

""", unsafe_allow_html=True)