class_docstr: MarianTransformer: Fast Neural Machine Translation
  
  Marian is an efficient, free Neural Machine Translation framework written in
  pure C++ with minimal dependencies. It is mainly being developed by the
  Microsoft Translator team. Many academic (most notably the University of
  Edinburgh and in the past the Adam Mickiewicz University in Poznań) and
  commercial contributors help with its development. MarianTransformer uses
  the models trained by MarianNMT.
  
  It is currently the engine behind the Microsoft Translator Neural Machine
  Translation services and being deployed by many companies, organizations and
  research projects.
  
  Note that this model only supports inputs up to 512 tokens. If you are
  working with longer inputs, consider splitting them first. For example, you
  can use the SentenceDetectorDL annotator to split longer texts into
  sentences.
  
  Pretrained models can be loaded with :meth:`.pretrained` of the companion
  object:
  
  >>> marian = MarianTransformer.pretrained() \
  ...     .setInputCols(["sentence"]) \
  ...     .setOutputCol("translation")
  
  The default model is ``"opus_mt_en_fr"``, default language is ``"xx"``
  (meaning multi-lingual), if no values are provided.
  
  For available pretrained models please see the `Models Hub <https://sparknlp.org/models?task=Translation>`__.
  
  For extended examples of usage, see the `Examples <https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/annotation/text/multilingual/Translation_Marian.ipynb>`__.
  
  ====================== ======================
  Input Annotation types Output Annotation type
  ====================== ======================
  ``DOCUMENT``           ``DOCUMENT``
  ====================== ======================
  
  Parameters
  ----------
  batchSize
      Size of every batch, by default 1
  configProtoBytes
      ConfigProto from tensorflow, serialized into byte array.
  langId
      Transformer's task, e.g. "summarize>", by default ""
  maxInputLength
      Controls the maximum length for encoder inputs (source language texts),
      by default 40
  maxOutputLength
      Controls the maximum length for decoder outputs (target language texts),
      by default 40
  
  Notes
  -----
  This is a very computationally expensive module especially on larger
  sequence. The use of an accelerator such as GPU is recommended.
  
  References
  ----------
  `MarianNMT at GitHub <https://marian-nmt.github.io/>`__
  
  `Marian: Fast Neural Machine Translation in C++  <https://www.aclweb.org/anthology/P18-4020/>`__
  
  **Paper Abstract:**
  
  *We present Marian, an efficient and self-contained Neural Machine
  Translation framework with an integrated automatic differentiation
  engine based on dynamic computation graphs. Marian is written entirely in
  C++. We describe the design of the encoder-decoder framework and
  demonstrate that a research-friendly toolkit can achieve high training
  and translation speed.*
  
  Examples
  --------
  >>> import sparknlp
  >>> from sparknlp.base import *
  >>> from sparknlp.annotator import *
  >>> from pyspark.ml import Pipeline
  >>> documentAssembler = DocumentAssembler() \
  ...     .setInputCol("text") \
  ...     .setOutputCol("document")
  >>> sentence = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx") \
  ...     .setInputCols("document") \
  ...     .setOutputCol("sentence")
  >>> marian = MarianTransformer.pretrained() \
  ...     .setInputCols("sentence") \
  ...     .setOutputCol("translation") \
  ...     .setMaxInputLength(30)
  >>> pipeline = Pipeline() \
  ...     .setStages([
  ...       documentAssembler,
  ...       sentence,
  ...       marian
  ...     ])
  >>> data = spark.createDataFrame([["What is the capital of France? We should know this in french."]]).toDF("text")
  >>> result = pipeline.fit(data).transform(data)
  >>> result.selectExpr("explode(translation.result) as result").show(truncate=False)
  +-------------------------------------+
  |result                               |
  +-------------------------------------+
  |Quelle est la capitale de la France ?|
  |On devrait le savoir en français.    |
  +-------------------------------------+
class_docstr_tok: ["MarianTransformer",":","Fast","Neural","Machine","Translation","Marian","is","an","efficient",",","free","Neural","Machine","Translation","framework","written","in","pure","C++","with","minimal","dependencies",".","It","is","mainly","being","developed","by","the","Microsoft","Translator","team",".","Many","academic","(","most","notably","the","University","of","Edinburgh","and","in","the","past","the","Adam","Mickiewicz","University","in","Pozna\u0144",")","and","commercial","contributors","help","with","its","development",".","MarianTransformer","uses","the","models","trained","by","MarianNMT",".","It","is","currently","the","engine","behind","the","Microsoft","Translator","Neural","Machine","Translation","services","and","being","deployed","by","many","companies",",","organizations","and","research","projects",".","Note","that","this","model","only","supports","inputs","up","to","512","tokens",".","If","you","are","working","with","longer","inputs",",","consider","splitting","them","first",".","For","example",",","you","can","use","the","SentenceDetectorDL","annotator","to","split","longer","texts","into","sentences",".","Pretrained","models","can","be","loaded","with",":","meth",":","`",".pretrained","`","of","the","companion","object",":",">",">",">","marian","=","MarianTransformer.pretrained","(",")","\\","...",".setInputCols","(","[","``","sentence","''","]",")","\\","...",".setOutputCol","(","``","translation","''",")","The","default","model","is","``","``","opus_mt_en_fr","''","``",",","default","language","is","``","``","xx","''","``","(","meaning","multi-lingual",")",",","if","no","values","are","provided",".","For","available","pretrained","models","please","see","the","`","Models","Hub","<","https",":","\/\/sparknlp.org\/models","?","task=Translation",">","`","__",".","For","extended","examples","of","usage",",","see","the","`","Examples","<","https",":","\/\/github.com\/JohnSnowLabs\/spark-nlp\/blob\/master\/examples\/python\/annotation\/text\/multilingual\/Translation_Marian.ipynb",">","`","__",".","======================","======================","Input","Annotation","types","Output","Annotation","type","======================","======================","``","DOCUMENT","``","``","DOCUMENT","``","======================","======================","Parameters","--","--","--","--","--","batchSize","Size","of","every","batch",",","by","default","1","configProtoBytes","ConfigProto","from","tensorflow",",","serialized","into","byte","array",".","langId","Transformer","'s","task",",","e.g",".","``","summarize",">","''",",","by","default","``","''","maxInputLength","Controls","the","maximum","length","for","encoder","inputs","(","source","language","texts",")",",","by","default","40","maxOutputLength","Controls","the","maximum","length","for","decoder","outputs","(","target","language","texts",")",",","by","default","40","Notes","--","--","-","This","is","a","very","computationally","expensive","module","especially","on","larger","sequence",".","The","use","of","an","accelerator","such","as","GPU","is","recommended",".","References","--","--","--","--","--","`","MarianNMT","at","GitHub","<","https",":","\/\/marian-nmt.github.io\/",">","`","__","`","Marian",":","Fast","Neural","Machine","Translation","in","C++","<","https",":","\/\/www.aclweb.org\/anthology\/P18-4020\/",">","`","__","*","*","Paper","Abstract",":","*","*","*","We","present","Marian",",","an","efficient","and","self-contained","Neural","Machine","Translation","framework","with","an","integrated","automatic","differentiation","engine","based","on","dynamic","computation","graphs",".","Marian","is","written","entirely","in","C++",".","We","describe","the","design","of","the","encoder-decoder","framework","and","demonstrate","that","a","research-friendly","toolkit","can","achieve","high","training","and","translation","speed",".","*","Examples","--","--","--","--",">",">",">","import","sparknlp",">",">",">","from","sparknlp.base","import","*",">",">",">","from","sparknlp.annotator","import","*",">",">",">","from","pyspark.ml","import","Pipeline",">",">",">","documentAssembler","=","DocumentAssembler","(",")","\\","...",".setInputCol","(","``","text","''",")","\\","...",".setOutputCol","(","``","document","''",")",">",">",">","sentence","=","SentenceDetectorDLModel.pretrained","(","``","sentence_detector_dl","''",",","``","xx","''",")","\\","...",".setInputCols","(","``","document","''",")","\\","...",".setOutputCol","(","``","sentence","''",")",">",">",">","marian","=","MarianTransformer.pretrained","(",")","\\","...",".setInputCols","(","``","sentence","''",")","\\","...",".setOutputCol","(","``","translation","''",")","\\","...",".setMaxInputLength","(","30",")",">",">",">","pipeline","=","Pipeline","(",")","\\","...",".setStages","(","[","...","documentAssembler",",","...","sentence",",","...","marian","...","]",")",">",">",">","data","=","spark.createDataFrame","(","[","[","``","What","is","the","capital","of","France","?","We","should","know","this","in","french",".","``","]","]",")",".toDF","(","``","text","''",")",">",">",">","result","=","pipeline.fit","(","data",")",".transform","(","data",")",">",">",">","result.selectExpr","(","``","explode","(","translation.result",")","as","result","''",")",".show","(","truncate=False",")","+","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","-+","|result","|","+","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","-+","|Quelle","est","la","capitale","de","la","France","?","|","|On","devrait","le","savoir","en","fran\u00e7ais",".","|","+","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","--","-+"]
docstr: Sets whether or not to use sampling, use greedy decoding otherwise.
  
  Parameters
  ----------
  value : bool
      Whether or not to use sampling; use greedy decoding otherwise
docstr_tok: ["Sets","whether","or","not","to","use","sampling",",","use","greedy","decoding","otherwise",".","Parameters","--","--","--","--","--","value",":","bool","Whether","or","not","to","use","sampling",";","use","greedy","decoding","otherwise"]
code: def setDoSample(self, value):
      """Sets whether or not to use sampling, use greedy decoding otherwise.
  
      Parameters
      ----------
      value : bool
          Whether or not to use sampling; use greedy decoding otherwise
      """
      return self._set(doSample=value)
  
file_path: spark-nlp/python/sparknlp/annotator/seq2seq/marian_transformer.py
imports: from sparknlp.common import *
label_desc: Use image node_id 6 for calling the MarianTransformer obj's underlying member method code with example usage: obj.setDoSample(value) and returns: self