docstr: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
      Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
      `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
      are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
  
  Returns:
  
  Examples:
  
  ```python
  >>> from transformers import ImageGPTFeatureExtractor, ImageGPTForCausalImageModeling
  >>> import torch
  >>> import matplotlib.pyplot as plt
  >>> import numpy as np
  
  >>> feature_extractor = ImageGPTFeatureExtractor.from_pretrained("openai/imagegpt-small")
  >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small")
  >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  >>> model.to(device)
  
  >>> # unconditional generation of 8 images
  >>> batch_size = 8
  >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1)  # initialize with SOS token
  >>> context = torch.tensor(context).to(device)
  >>> output = model.generate(
  ...     input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40
  ... )
  
  >>> clusters = feature_extractor.clusters
  >>> n_px = feature_extractor.size
  
  >>> samples = output[:, 1:].cpu().detach().numpy()
  >>> samples_img = [
  ...     np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [n_px, n_px, 3]).astype(np.uint8) for s in samples
  ... ]  # convert color cluster tokens back to pixels
  >>> f, axes = plt.subplots(1, batch_size, dpi=300)
  
  >>> for img, ax in zip(samples_img, axes):
  ...     ax.axis("off")
  ...     ax.imshow(img)
  ```
code: def forward(
      self,
      input_ids: Optional[torch.Tensor] = None,
      past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
      attention_mask: Optional[torch.Tensor] = None,
      token_type_ids: Optional[torch.Tensor] = None,
      position_ids: Optional[torch.Tensor] = None,
      head_mask: Optional[torch.Tensor] = None,
      inputs_embeds: Optional[torch.Tensor] = None,
      encoder_hidden_states: Optional[torch.Tensor] = None,
      encoder_attention_mask: Optional[torch.Tensor] = None,
      labels: Optional[torch.Tensor] = None,
      use_cache: Optional[bool] = None,
      output_attentions: Optional[bool] = None,
      output_hidden_states: Optional[bool] = None,
      return_dict: Optional[bool] = None,
      **kwargs: Any,
  ) -> Union[Tuple, CausalLMOutputWithCrossAttentions]:
      r"""
      labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
          Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
          `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
          are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
  
      Returns:
  
      Examples:
  
      ```python
      >>> from transformers import ImageGPTFeatureExtractor, ImageGPTForCausalImageModeling
      >>> import torch
      >>> import matplotlib.pyplot as plt
      >>> import numpy as np
  
      >>> feature_extractor = ImageGPTFeatureExtractor.from_pretrained("openai/imagegpt-small")
      >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small")
      >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
      >>> model.to(device)
  
      >>> # unconditional generation of 8 images
      >>> batch_size = 8
      >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1)  # initialize with SOS token
      >>> context = torch.tensor(context).to(device)
      >>> output = model.generate(
      ...     input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40
      ... )
  
      >>> clusters = feature_extractor.clusters
      >>> n_px = feature_extractor.size
  
      >>> samples = output[:, 1:].cpu().detach().numpy()
      >>> samples_img = [
      ...     np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [n_px, n_px, 3]).astype(np.uint8) for s in samples
      ... ]  # convert color cluster tokens back to pixels
      >>> f, axes = plt.subplots(1, batch_size, dpi=300)
  
      >>> for img, ax in zip(samples_img, axes):
      ...     ax.axis("off")
      ...     ax.imshow(img)
      ```"""
  
      if "pixel_values" in kwargs:
          warnings.warn(
              "The `pixel_values` argument is deprecated and will be removed in a future version, use `input_ids`"
              " instead.",
              FutureWarning,
          )
  
          if input_ids is not None:
              raise ValueError(
                  "You cannot pass both `pixel_values` and `input_ids`. Please make sure to only pass `input_ids`."
              )
  
          input_ids = kwargs.pop("pixel_values")
  
      return_dict = (
          return_dict
          if return_dict is not None
          else self.config.use_return_dict
      )
  
      transformer_outputs = self.transformer(
          input_ids,
          past_key_values=past_key_values,
          attention_mask=attention_mask,
          token_type_ids=token_type_ids,
          position_ids=position_ids,
          head_mask=head_mask,
          inputs_embeds=inputs_embeds,
          encoder_hidden_states=encoder_hidden_states,
          encoder_attention_mask=encoder_attention_mask,
          use_cache=use_cache,
          output_attentions=output_attentions,
          output_hidden_states=output_hidden_states,
          return_dict=return_dict,
      )
      hidden_states = transformer_outputs[0]
  
      lm_logits = self.lm_head(hidden_states)
  
      loss = None
      if labels is not None:
          # Shift so that tokens < n predict n
          shift_logits = lm_logits[..., :-1, :].contiguous()
          shift_labels = labels[..., 1:].contiguous()
          # Flatten the tokens
          loss_fct = CrossEntropyLoss()
          loss = loss_fct(
              shift_logits.view(-1, shift_logits.size(-1)),
              shift_labels.view(-1),
          )
  
      if not return_dict:
          output = (lm_logits,) + transformer_outputs[1:]
          return ((loss,) + output) if loss is not None else output
  
      return CausalLMOutputWithCrossAttentions(
          loss=loss,
          logits=lm_logits,
          past_key_values=transformer_outputs.past_key_values,
          hidden_states=transformer_outputs.hidden_states,
          attentions=transformer_outputs.attentions,
          cross_attentions=transformer_outputs.cross_attentions,
      )
  
imports: import math
  import os
  import warnings
  from typing import Any, Optional, Tuple, Union
  import torch
  import torch.utils.checkpoint
  from torch import nn
  from torch.cuda.amp import autocast
  from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
  from ...activations import ACT2FN
  from ...modeling_outputs import BaseModelOutputWithPastAndCrossAttentions, CausalLMOutputWithCrossAttentions, SequenceClassifierOutputWithPast
  from ...modeling_utils import PreTrainedModel
  from ...pytorch_utils import Conv1D, find_pruneable_heads_and_indices, prune_conv1d_layer
  from ...utils import add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings
  from .configuration_imagegpt import ImageGPTConfig
label_desc: Use image node_id 5 for calling the ImageGPTForCausalImageModeling obj's underlying member method code with example usage: obj.forward(input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, labels, use_cache, output_attentions, output_hidden_states, return_dict) and returns: CausalLMOutputWithCrossAttentions, unknown