We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
tfa.seq2seq.sequence_loss can't average over one dimension (batch or timesteps) while summing over the other one
**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
- TensorFlow installed from (source or binary): binary
- TensorFlow version (use command below): 2.0.0=beta1
- TensorFlow Addons installed from (source, PyPi): PyPi
- TensorFlow Addons version: 0.4.0
- Python version and type (eg. Anaconda Python, Stock Python as in Mac, or homebrew installed Python etc): Google Colab Python
- Is GPU used? (yes/no): yes
- GPU model (if used): T4

**Describe the bug**

`tfa.seq2seq.sequence_loss` can't average over one dimension (`batch` or `timesteps`) while summing over the other one. It will arbitrarily only execute the averaging and ignore the sum right now.

**Describe the expected behavior**

I think the weights should be associated with the summing operation, and then the averaging should happen irrespective of that.
Concretely, when passing, say `average_across_batch=True` and `sum_over_timesteps=True` (of course, making sure `average_across_timesteps=False` is set), you should expect either of these things:

1. An error stating that this is not implemented (might be the wisest).
2. Return a scalar tensor obtained by either of these two following orders:
  a) first computing the *weighted sum* of xents over timesteps (yielding a batchsize-sized tensor of xent-sums), then simply averaging this vector, i.e., summing and dividing by the batchsize. The result, however, is just the both-averaged version times the batchsize, divided by the sum of all weights.
  b) first computing the *weighted average* over the batchsize, then summing these averages over all timesteps. The result here is different from 1a and the double-averaged (of course, there is some correlation...)!

I think 1a is the desired behavior (as the loglikelihood of a sequence really is the sum of the individual loglikelihoods and batches do correspond to sequence-length agnostic averages) and I'd be happy to establish it as the standard for this. Either way, doing something other than failing with an error will require an explicit notice in the docs. An error (or warning for backwards-compatibility?) might just be the simplest and safest option.

**Code to reproduce the issue**

```python
tfa.seq2seq.sequence_loss(
  logits=tf.random.normal([3, 5, 7]),
  targets=tf.zeros([3, 5], dtype=tf.int32),
  weights=tf.sequence_mask(lengths=[3, 5, 1], maxlen=5, dtype=tf.float32),
  average_across_batch=True,
  average_across_timesteps=False,
  sum_over_batch=False,
  sum_over_timesteps=True,
)
```
...should return a scalar but returns only the batch-averaged tensor.

**Some more code to play with to test the claims above**

```python
import tensorflow.compat.v2 as tf
import tensorflow_addons as tfa
import numpy as np
import random

case1b = []
dblavg = []

for _ in range(100):
  dtype = tf.float32
  batchsize = random.randint(2, 10)
  maxlen = random.randint(2, 10)
  logits = tf.random.normal([batchsize, maxlen, 3])
  labels = tf.zeros([batchsize, maxlen], dtype=tf.int32)
  lengths = tf.squeeze(tf.random.categorical(tf.zeros([1, maxlen - 1]), batchsize)) + 1
  weights = tf.sequence_mask(lengths=lengths, maxlen=maxlen, dtype=tf.float32)

  def sl(ab, sb, at, st):
    return tfa.seq2seq.sequence_loss(
        logits,
        labels,
        weights,
        average_across_batch=ab,
        average_across_timesteps=at,
        sum_over_batch=sb,
        sum_over_timesteps=st,
    )

  all_b_all_t = sl(ab=False, sb=False, at=False, st=False)
  avg_b_avg_t = sl(ab=True,  sb=False, at=True,  st=False)
  sum_b_all_t = sl(ab=False, sb=True,  at=False, st=False)

  tf.assert_equal(sum_b_all_t, tf.math.divide_no_nan(tf.reduce_sum(all_b_all_t, axis=0), tf.reduce_sum(weights, axis=0)))

  weighted = all_b_all_t * weights

  first_sum_timesteps = tf.reduce_sum(weighted, axis=1)
  then_average_batch = tf.reduce_sum(first_sum_timesteps) / batchsize

  first_average_batch = tf.math.divide_no_nan(tf.reduce_sum(weighted, axis=0), tf.reduce_sum(weights, axis=0))
  then_sum_timesteps = tf.reduce_sum(first_average_batch)

  # Case 1a and 1b are different.
  assert not np.isclose(then_average_batch, then_sum_timesteps)
  # Case 1a is just the double-averaging up to a constant.
  assert     np.allclose(then_average_batch * batchsize / tf.reduce_sum(weights), avg_b_avg_t)
  # Case 1b is not just the averaging.
  assert not np.allclose(then_sum_timesteps / maxlen, avg_b_avg_t)
  # They only kind of correlate:
  case1b.append(then_sum_timesteps / maxlen)
  dblavg.append(avg_b_avg_t)
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `tensorflow_addons/seq2seq/loss.py`
Content:
```
1 # Copyright 2016 The TensorFlow Authors. All Rights Reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 # ==============================================================================
15 """Seq2seq loss operations for use in sequence models."""
16 
17 from __future__ import absolute_import
18 from __future__ import division
19 from __future__ import print_function
20 
21 import tensorflow as tf
22 
23 
24 def sequence_loss(logits,
25                   targets,
26                   weights,
27                   average_across_timesteps=True,
28                   average_across_batch=True,
29                   sum_over_timesteps=False,
30                   sum_over_batch=False,
31                   softmax_loss_function=None,
32                   name=None):
33     """Weighted cross-entropy loss for a sequence of logits.
34 
35     Depending on the values of `average_across_timesteps` /
36     `sum_over_timesteps` and `average_across_batch` / `sum_over_batch`, the
37     return Tensor will have rank 0, 1, or 2 as these arguments reduce the
38     cross-entropy at each target, which has shape
39     `[batch_size, sequence_length]`, over their respective dimensions. For
40     example, if `average_across_timesteps` is `True` and `average_across_batch`
41     is `False`, then the return Tensor will have shape `[batch_size]`.
42 
43     Note that `average_across_timesteps` and `sum_over_timesteps` cannot be
44     True at same time. Same for `average_across_batch` and `sum_over_batch`.
45 
46     The recommended loss reduction in tf 2.0 has been changed to sum_over,
47     instead of weighted average. User are recommend to use `sum_over_timesteps`
48     and `sum_over_batch` for reduction.
49 
50     Args:
51       logits: A Tensor of shape
52         `[batch_size, sequence_length, num_decoder_symbols]` and dtype float.
53         The logits correspond to the prediction across all classes at each
54         timestep.
55       targets: A Tensor of shape `[batch_size, sequence_length]` and dtype
56         int. The target represents the true class at each timestep.
57       weights: A Tensor of shape `[batch_size, sequence_length]` and dtype
58         float. `weights` constitutes the weighting of each prediction in the
59         sequence. When using `weights` as masking, set all valid timesteps to 1
60         and all padded timesteps to 0, e.g. a mask returned by
61         `tf.sequence_mask`.
62       average_across_timesteps: If set, sum the cost across the sequence
63         dimension and divide the cost by the total label weight across
64         timesteps.
65       average_across_batch: If set, sum the cost across the batch dimension and
66         divide the returned cost by the batch size.
67       sum_over_timesteps: If set, sum the cost across the sequence dimension
68         and divide the size of the sequence. Note that any element with 0
69         weights will be excluded from size calculation.
70       sum_over_batch: if set, sum the cost across the batch dimension and
71         divide the total cost by the batch size. Not that any element with 0
72         weights will be excluded from size calculation.
73       softmax_loss_function: Function (labels, logits) -> loss-batch
74         to be used instead of the standard softmax (the default if this is
75         None). **Note that to avoid confusion, it is required for the function
76         to accept named arguments.**
77       name: Optional name for this operation, defaults to "sequence_loss".
78 
79     Returns:
80       A float Tensor of rank 0, 1, or 2 depending on the
81       `average_across_timesteps` and `average_across_batch` arguments. By
82       default, it has rank 0 (scalar) and is the weighted average cross-entropy
83       (log-perplexity) per symbol.
84 
85     Raises:
86       ValueError: logits does not have 3 dimensions or targets does not have 2
87                   dimensions or weights does not have 2 dimensions.
88     """
89     if len(logits.get_shape()) != 3:
90         raise ValueError("Logits must be a "
91                          "[batch_size x sequence_length x logits] tensor")
92     if len(targets.get_shape()) != 2:
93         raise ValueError(
94             "Targets must be a [batch_size x sequence_length] tensor")
95     if len(weights.get_shape()) != 2:
96         raise ValueError(
97             "Weights must be a [batch_size x sequence_length] tensor")
98     if average_across_timesteps and sum_over_timesteps:
99         raise ValueError(
100             "average_across_timesteps and sum_over_timesteps cannot "
101             "be set to True at same time.")
102     if average_across_batch and sum_over_batch:
103         raise ValueError(
104             "average_across_batch and sum_over_batch cannot be set "
105             "to True at same time.")
106     with tf.name_scope(name or "sequence_loss"):
107         num_classes = tf.shape(input=logits)[2]
108         logits_flat = tf.reshape(logits, [-1, num_classes])
109         targets = tf.reshape(targets, [-1])
110         if softmax_loss_function is None:
111             crossent = tf.nn.sparse_softmax_cross_entropy_with_logits(
112                 labels=targets, logits=logits_flat)
113         else:
114             crossent = softmax_loss_function(
115                 labels=targets, logits=logits_flat)
116         crossent *= tf.reshape(weights, [-1])
117         if average_across_timesteps and average_across_batch:
118             crossent = tf.reduce_sum(input_tensor=crossent)
119             total_size = tf.reduce_sum(input_tensor=weights)
120             crossent = tf.math.divide_no_nan(crossent, total_size)
121         elif sum_over_timesteps and sum_over_batch:
122             crossent = tf.reduce_sum(input_tensor=crossent)
123             total_count = tf.cast(
124                 tf.math.count_nonzero(weights), crossent.dtype)
125             crossent = tf.math.divide_no_nan(crossent, total_count)
126         else:
127             crossent = tf.reshape(crossent, tf.shape(input=logits)[0:2])
128             if average_across_timesteps or average_across_batch:
129                 reduce_axis = [0] if average_across_batch else [1]
130                 crossent = tf.reduce_sum(
131                     input_tensor=crossent, axis=reduce_axis)
132                 total_size = tf.reduce_sum(
133                     input_tensor=weights, axis=reduce_axis)
134                 crossent = tf.math.divide_no_nan(crossent, total_size)
135             elif sum_over_timesteps or sum_over_batch:
136                 reduce_axis = [0] if sum_over_batch else [1]
137                 crossent = tf.reduce_sum(
138                     input_tensor=crossent, axis=reduce_axis)
139                 total_count = tf.cast(
140                     tf.math.count_nonzero(weights, axis=reduce_axis),
141                     dtype=crossent.dtype)
142                 crossent = tf.math.divide_no_nan(crossent, total_count)
143         return crossent
144 
145 
146 class SequenceLoss(tf.keras.losses.Loss):
147     """Weighted cross-entropy loss for a sequence of logits."""
148 
149     def __init__(self,
150                  average_across_timesteps=False,
151                  average_across_batch=False,
152                  sum_over_timesteps=True,
153                  sum_over_batch=True,
154                  softmax_loss_function=None,
155                  name=None):
156         super(SequenceLoss, self).__init__(name=name)
157         self.average_across_timesteps = average_across_timesteps
158         self.average_across_batch = average_across_batch
159         self.sum_over_timesteps = sum_over_timesteps
160         self.sum_over_batch = sum_over_batch
161         self.softmax_loss_function = softmax_loss_function
162 
163     def __call__(self, y_true, y_pred, sample_weight=None):
164         """Override the parent __call__ to have a customized reduce
165         behavior."""
166         return sequence_loss(
167             y_pred,
168             y_true,
169             sample_weight,
170             average_across_timesteps=self.average_across_timesteps,
171             average_across_batch=self.average_across_batch,
172             sum_over_timesteps=self.sum_over_timesteps,
173             sum_over_batch=self.sum_over_batch,
174             softmax_loss_function=self.softmax_loss_function,
175             name=self.name)
176 
177     def call(self, y_true, y_pred):
178         # Skip this method since the __call__ contains real implementation.
179         pass
180
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/tensorflow_addons/seq2seq/loss.py b/tensorflow_addons/seq2seq/loss.py
--- a/tensorflow_addons/seq2seq/loss.py
+++ b/tensorflow_addons/seq2seq/loss.py
@@ -103,6 +103,14 @@
         raise ValueError(
             "average_across_batch and sum_over_batch cannot be set "
             "to True at same time.")
+    if average_across_batch and sum_over_timesteps:
+        raise ValueError(
+            "average_across_batch and sum_over_timesteps cannot be set "
+            "to True at same time because of ambiguous order.")
+    if sum_over_batch and average_across_timesteps:
+        raise ValueError(
+            "sum_over_batch and average_across_timesteps cannot be set "
+            "to True at same time because of ambiguous order.")
     with tf.name_scope(name or "sequence_loss"):
         num_classes = tf.shape(input=logits)[2]
         logits_flat = tf.reshape(logits, [-1, num_classes])

{"golden_diff": "diff --git a/tensorflow_addons/seq2seq/loss.py b/tensorflow_addons/seq2seq/loss.py\n--- a/tensorflow_addons/seq2seq/loss.py\n+++ b/tensorflow_addons/seq2seq/loss.py\n@@ -103,6 +103,14 @@\n         raise ValueError(\n             \"average_across_batch and sum_over_batch cannot be set \"\n             \"to True at same time.\")\n+    if average_across_batch and sum_over_timesteps:\n+        raise ValueError(\n+            \"average_across_batch and sum_over_timesteps cannot be set \"\n+            \"to True at same time because of ambiguous order.\")\n+    if sum_over_batch and average_across_timesteps:\n+        raise ValueError(\n+            \"sum_over_batch and average_across_timesteps cannot be set \"\n+            \"to True at same time because of ambiguous order.\")\n     with tf.name_scope(name or \"sequence_loss\"):\n         num_classes = tf.shape(input=logits)[2]\n         logits_flat = tf.reshape(logits, [-1, num_classes])\n", "issue": "tfa.seq2seq.sequence_loss can't average over one dimension (batch or timesteps) while summing over the other one\n**System information**\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab\r\n- TensorFlow installed from (source or binary): binary\r\n- TensorFlow version (use command below): 2.0.0=beta1\r\n- TensorFlow Addons installed from (source, PyPi): PyPi\r\n- TensorFlow Addons version: 0.4.0\r\n- Python version and type (eg. Anaconda Python, Stock Python as in Mac, or homebrew installed Python etc): Google Colab Python\r\n- Is GPU used? (yes/no): yes\r\n- GPU model (if used): T4\r\n\r\n**Describe the bug**\r\n\r\n`tfa.seq2seq.sequence_loss` can't average over one dimension (`batch` or `timesteps`) while summing over the other one. It will arbitrarily only execute the averaging and ignore the sum right now.\r\n\r\n**Describe the expected behavior**\r\n\r\nI think the weights should be associated with the summing operation, and then the averaging should happen irrespective of that.\r\nConcretely, when passing, say `average_across_batch=True` and `sum_over_timesteps=True` (of course, making sure `average_across_timesteps=False` is set), you should expect either of these things:\r\n\r\n1. An error stating that this is not implemented (might be the wisest).\r\n2. Return a scalar tensor obtained by either of these two following orders:\r\n  a) first computing the *weighted sum* of xents over timesteps (yielding a batchsize-sized tensor of xent-sums), then simply averaging this vector, i.e., summing and dividing by the batchsize. The result, however, is just the both-averaged version times the batchsize, divided by the sum of all weights.\r\n  b) first computing the *weighted average* over the batchsize, then summing these averages over all timesteps. The result here is different from 1a and the double-averaged (of course, there is some correlation...)!\r\n\r\nI think 1a is the desired behavior (as the loglikelihood of a sequence really is the sum of the individual loglikelihoods and batches do correspond to sequence-length agnostic averages) and I'd be happy to establish it as the standard for this. Either way, doing something other than failing with an error will require an explicit notice in the docs. An error (or warning for backwards-compatibility?) might just be the simplest and safest option.\r\n\r\n**Code to reproduce the issue**\r\n\r\n```python\r\ntfa.seq2seq.sequence_loss(\r\n  logits=tf.random.normal([3, 5, 7]),\r\n  targets=tf.zeros([3, 5], dtype=tf.int32),\r\n  weights=tf.sequence_mask(lengths=[3, 5, 1], maxlen=5, dtype=tf.float32),\r\n  average_across_batch=True,\r\n  average_across_timesteps=False,\r\n  sum_over_batch=False,\r\n  sum_over_timesteps=True,\r\n)\r\n```\r\n...should return a scalar but returns only the batch-averaged tensor.\r\n\r\n**Some more code to play with to test the claims above**\r\n\r\n```python\r\nimport tensorflow.compat.v2 as tf\r\nimport tensorflow_addons as tfa\r\nimport numpy as np\r\nimport random\r\n\r\ncase1b = []\r\ndblavg = []\r\n\r\nfor _ in range(100):\r\n  dtype = tf.float32\r\n  batchsize = random.randint(2, 10)\r\n  maxlen = random.randint(2, 10)\r\n  logits = tf.random.normal([batchsize, maxlen, 3])\r\n  labels = tf.zeros([batchsize, maxlen], dtype=tf.int32)\r\n  lengths = tf.squeeze(tf.random.categorical(tf.zeros([1, maxlen - 1]), batchsize)) + 1\r\n  weights = tf.sequence_mask(lengths=lengths, maxlen=maxlen, dtype=tf.float32)\r\n\r\n  def sl(ab, sb, at, st):\r\n    return tfa.seq2seq.sequence_loss(\r\n        logits,\r\n        labels,\r\n        weights,\r\n        average_across_batch=ab,\r\n        average_across_timesteps=at,\r\n        sum_over_batch=sb,\r\n        sum_over_timesteps=st,\r\n    )\r\n\r\n  all_b_all_t = sl(ab=False, sb=False, at=False, st=False)\r\n  avg_b_avg_t = sl(ab=True,  sb=False, at=True,  st=False)\r\n  sum_b_all_t = sl(ab=False, sb=True,  at=False, st=False)\r\n\r\n  tf.assert_equal(sum_b_all_t, tf.math.divide_no_nan(tf.reduce_sum(all_b_all_t, axis=0), tf.reduce_sum(weights, axis=0)))\r\n\r\n  weighted = all_b_all_t * weights\r\n\r\n  first_sum_timesteps = tf.reduce_sum(weighted, axis=1)\r\n  then_average_batch = tf.reduce_sum(first_sum_timesteps) / batchsize\r\n\r\n  first_average_batch = tf.math.divide_no_nan(tf.reduce_sum(weighted, axis=0), tf.reduce_sum(weights, axis=0))\r\n  then_sum_timesteps = tf.reduce_sum(first_average_batch)\r\n\r\n  # Case 1a and 1b are different.\r\n  assert not np.isclose(then_average_batch, then_sum_timesteps)\r\n  # Case 1a is just the double-averaging up to a constant.\r\n  assert     np.allclose(then_average_batch * batchsize / tf.reduce_sum(weights), avg_b_avg_t)\r\n  # Case 1b is not just the averaging.\r\n  assert not np.allclose(then_sum_timesteps / maxlen, avg_b_avg_t)\r\n  # They only kind of correlate:\r\n  case1b.append(then_sum_timesteps / maxlen)\r\n  dblavg.append(avg_b_avg_t)\r\n```\n", "before_files": [{"content": "# Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"Seq2seq loss operations for use in sequence models.\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport tensorflow as tf\n\n\ndef sequence_loss(logits,\n                  targets,\n                  weights,\n                  average_across_timesteps=True,\n                  average_across_batch=True,\n                  sum_over_timesteps=False,\n                  sum_over_batch=False,\n                  softmax_loss_function=None,\n                  name=None):\n    \"\"\"Weighted cross-entropy loss for a sequence of logits.\n\n    Depending on the values of `average_across_timesteps` /\n    `sum_over_timesteps` and `average_across_batch` / `sum_over_batch`, the\n    return Tensor will have rank 0, 1, or 2 as these arguments reduce the\n    cross-entropy at each target, which has shape\n    `[batch_size, sequence_length]`, over their respective dimensions. For\n    example, if `average_across_timesteps` is `True` and `average_across_batch`\n    is `False`, then the return Tensor will have shape `[batch_size]`.\n\n    Note that `average_across_timesteps` and `sum_over_timesteps` cannot be\n    True at same time. Same for `average_across_batch` and `sum_over_batch`.\n\n    The recommended loss reduction in tf 2.0 has been changed to sum_over,\n    instead of weighted average. User are recommend to use `sum_over_timesteps`\n    and `sum_over_batch` for reduction.\n\n    Args:\n      logits: A Tensor of shape\n        `[batch_size, sequence_length, num_decoder_symbols]` and dtype float.\n        The logits correspond to the prediction across all classes at each\n        timestep.\n      targets: A Tensor of shape `[batch_size, sequence_length]` and dtype\n        int. The target represents the true class at each timestep.\n      weights: A Tensor of shape `[batch_size, sequence_length]` and dtype\n        float. `weights` constitutes the weighting of each prediction in the\n        sequence. When using `weights` as masking, set all valid timesteps to 1\n        and all padded timesteps to 0, e.g. a mask returned by\n        `tf.sequence_mask`.\n      average_across_timesteps: If set, sum the cost across the sequence\n        dimension and divide the cost by the total label weight across\n        timesteps.\n      average_across_batch: If set, sum the cost across the batch dimension and\n        divide the returned cost by the batch size.\n      sum_over_timesteps: If set, sum the cost across the sequence dimension\n        and divide the size of the sequence. Note that any element with 0\n        weights will be excluded from size calculation.\n      sum_over_batch: if set, sum the cost across the batch dimension and\n        divide the total cost by the batch size. Not that any element with 0\n        weights will be excluded from size calculation.\n      softmax_loss_function: Function (labels, logits) -> loss-batch\n        to be used instead of the standard softmax (the default if this is\n        None). **Note that to avoid confusion, it is required for the function\n        to accept named arguments.**\n      name: Optional name for this operation, defaults to \"sequence_loss\".\n\n    Returns:\n      A float Tensor of rank 0, 1, or 2 depending on the\n      `average_across_timesteps` and `average_across_batch` arguments. By\n      default, it has rank 0 (scalar) and is the weighted average cross-entropy\n      (log-perplexity) per symbol.\n\n    Raises:\n      ValueError: logits does not have 3 dimensions or targets does not have 2\n                  dimensions or weights does not have 2 dimensions.\n    \"\"\"\n    if len(logits.get_shape()) != 3:\n        raise ValueError(\"Logits must be a \"\n                         \"[batch_size x sequence_length x logits] tensor\")\n    if len(targets.get_shape()) != 2:\n        raise ValueError(\n            \"Targets must be a [batch_size x sequence_length] tensor\")\n    if len(weights.get_shape()) != 2:\n        raise ValueError(\n            \"Weights must be a [batch_size x sequence_length] tensor\")\n    if average_across_timesteps and sum_over_timesteps:\n        raise ValueError(\n            \"average_across_timesteps and sum_over_timesteps cannot \"\n            \"be set to True at same time.\")\n    if average_across_batch and sum_over_batch:\n        raise ValueError(\n            \"average_across_batch and sum_over_batch cannot be set \"\n            \"to True at same time.\")\n    with tf.name_scope(name or \"sequence_loss\"):\n        num_classes = tf.shape(input=logits)[2]\n        logits_flat = tf.reshape(logits, [-1, num_classes])\n        targets = tf.reshape(targets, [-1])\n        if softmax_loss_function is None:\n            crossent = tf.nn.sparse_softmax_cross_entropy_with_logits(\n                labels=targets, logits=logits_flat)\n        else:\n            crossent = softmax_loss_function(\n                labels=targets, logits=logits_flat)\n        crossent *= tf.reshape(weights, [-1])\n        if average_across_timesteps and average_across_batch:\n            crossent = tf.reduce_sum(input_tensor=crossent)\n            total_size = tf.reduce_sum(input_tensor=weights)\n            crossent = tf.math.divide_no_nan(crossent, total_size)\n        elif sum_over_timesteps and sum_over_batch:\n            crossent = tf.reduce_sum(input_tensor=crossent)\n            total_count = tf.cast(\n                tf.math.count_nonzero(weights), crossent.dtype)\n            crossent = tf.math.divide_no_nan(crossent, total_count)\n        else:\n            crossent = tf.reshape(crossent, tf.shape(input=logits)[0:2])\n            if average_across_timesteps or average_across_batch:\n                reduce_axis = [0] if average_across_batch else [1]\n                crossent = tf.reduce_sum(\n                    input_tensor=crossent, axis=reduce_axis)\n                total_size = tf.reduce_sum(\n                    input_tensor=weights, axis=reduce_axis)\n                crossent = tf.math.divide_no_nan(crossent, total_size)\n            elif sum_over_timesteps or sum_over_batch:\n                reduce_axis = [0] if sum_over_batch else [1]\n                crossent = tf.reduce_sum(\n                    input_tensor=crossent, axis=reduce_axis)\n                total_count = tf.cast(\n                    tf.math.count_nonzero(weights, axis=reduce_axis),\n                    dtype=crossent.dtype)\n                crossent = tf.math.divide_no_nan(crossent, total_count)\n        return crossent\n\n\nclass SequenceLoss(tf.keras.losses.Loss):\n    \"\"\"Weighted cross-entropy loss for a sequence of logits.\"\"\"\n\n    def __init__(self,\n                 average_across_timesteps=False,\n                 average_across_batch=False,\n                 sum_over_timesteps=True,\n                 sum_over_batch=True,\n                 softmax_loss_function=None,\n                 name=None):\n        super(SequenceLoss, self).__init__(name=name)\n        self.average_across_timesteps = average_across_timesteps\n        self.average_across_batch = average_across_batch\n        self.sum_over_timesteps = sum_over_timesteps\n        self.sum_over_batch = sum_over_batch\n        self.softmax_loss_function = softmax_loss_function\n\n    def __call__(self, y_true, y_pred, sample_weight=None):\n        \"\"\"Override the parent __call__ to have a customized reduce\n        behavior.\"\"\"\n        return sequence_loss(\n            y_pred,\n            y_true,\n            sample_weight,\n            average_across_timesteps=self.average_across_timesteps,\n            average_across_batch=self.average_across_batch,\n            sum_over_timesteps=self.sum_over_timesteps,\n            sum_over_batch=self.sum_over_batch,\n            softmax_loss_function=self.softmax_loss_function,\n            name=self.name)\n\n    def call(self, y_true, y_pred):\n        # Skip this method since the __call__ contains real implementation.\n        pass\n", "path": "tensorflow_addons/seq2seq/loss.py"}], "after_files": [{"content": "# Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"Seq2seq loss operations for use in sequence models.\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport tensorflow as tf\n\n\ndef sequence_loss(logits,\n                  targets,\n                  weights,\n                  average_across_timesteps=True,\n                  average_across_batch=True,\n                  sum_over_timesteps=False,\n                  sum_over_batch=False,\n                  softmax_loss_function=None,\n                  name=None):\n    \"\"\"Weighted cross-entropy loss for a sequence of logits.\n\n    Depending on the values of `average_across_timesteps` /\n    `sum_over_timesteps` and `average_across_batch` / `sum_over_batch`, the\n    return Tensor will have rank 0, 1, or 2 as these arguments reduce the\n    cross-entropy at each target, which has shape\n    `[batch_size, sequence_length]`, over their respective dimensions. For\n    example, if `average_across_timesteps` is `True` and `average_across_batch`\n    is `False`, then the return Tensor will have shape `[batch_size]`.\n\n    Note that `average_across_timesteps` and `sum_over_timesteps` cannot be\n    True at same time. Same for `average_across_batch` and `sum_over_batch`.\n\n    The recommended loss reduction in tf 2.0 has been changed to sum_over,\n    instead of weighted average. User are recommend to use `sum_over_timesteps`\n    and `sum_over_batch` for reduction.\n\n    Args:\n      logits: A Tensor of shape\n        `[batch_size, sequence_length, num_decoder_symbols]` and dtype float.\n        The logits correspond to the prediction across all classes at each\n        timestep.\n      targets: A Tensor of shape `[batch_size, sequence_length]` and dtype\n        int. The target represents the true class at each timestep.\n      weights: A Tensor of shape `[batch_size, sequence_length]` and dtype\n        float. `weights` constitutes the weighting of each prediction in the\n        sequence. When using `weights` as masking, set all valid timesteps to 1\n        and all padded timesteps to 0, e.g. a mask returned by\n        `tf.sequence_mask`.\n      average_across_timesteps: If set, sum the cost across the sequence\n        dimension and divide the cost by the total label weight across\n        timesteps.\n      average_across_batch: If set, sum the cost across the batch dimension and\n        divide the returned cost by the batch size.\n      sum_over_timesteps: If set, sum the cost across the sequence dimension\n        and divide the size of the sequence. Note that any element with 0\n        weights will be excluded from size calculation.\n      sum_over_batch: if set, sum the cost across the batch dimension and\n        divide the total cost by the batch size. Not that any element with 0\n        weights will be excluded from size calculation.\n      softmax_loss_function: Function (labels, logits) -> loss-batch\n        to be used instead of the standard softmax (the default if this is\n        None). **Note that to avoid confusion, it is required for the function\n        to accept named arguments.**\n      name: Optional name for this operation, defaults to \"sequence_loss\".\n\n    Returns:\n      A float Tensor of rank 0, 1, or 2 depending on the\n      `average_across_timesteps` and `average_across_batch` arguments. By\n      default, it has rank 0 (scalar) and is the weighted average cross-entropy\n      (log-perplexity) per symbol.\n\n    Raises:\n      ValueError: logits does not have 3 dimensions or targets does not have 2\n                  dimensions or weights does not have 2 dimensions.\n    \"\"\"\n    if len(logits.get_shape()) != 3:\n        raise ValueError(\"Logits must be a \"\n                         \"[batch_size x sequence_length x logits] tensor\")\n    if len(targets.get_shape()) != 2:\n        raise ValueError(\n            \"Targets must be a [batch_size x sequence_length] tensor\")\n    if len(weights.get_shape()) != 2:\n        raise ValueError(\n            \"Weights must be a [batch_size x sequence_length] tensor\")\n    if average_across_timesteps and sum_over_timesteps:\n        raise ValueError(\n            \"average_across_timesteps and sum_over_timesteps cannot \"\n            \"be set to True at same time.\")\n    if average_across_batch and sum_over_batch:\n        raise ValueError(\n            \"average_across_batch and sum_over_batch cannot be set \"\n            \"to True at same time.\")\n    if average_across_batch and sum_over_timesteps:\n        raise ValueError(\n            \"average_across_batch and sum_over_timesteps cannot be set \"\n            \"to True at same time because of ambiguous order.\")\n    if sum_over_batch and average_across_timesteps:\n        raise ValueError(\n            \"sum_over_batch and average_across_timesteps cannot be set \"\n            \"to True at same time because of ambiguous order.\")\n    with tf.name_scope(name or \"sequence_loss\"):\n        num_classes = tf.shape(input=logits)[2]\n        logits_flat = tf.reshape(logits, [-1, num_classes])\n        targets = tf.reshape(targets, [-1])\n        if softmax_loss_function is None:\n            crossent = tf.nn.sparse_softmax_cross_entropy_with_logits(\n                labels=targets, logits=logits_flat)\n        else:\n            crossent = softmax_loss_function(\n                labels=targets, logits=logits_flat)\n        crossent *= tf.reshape(weights, [-1])\n        if average_across_timesteps and average_across_batch:\n            crossent = tf.reduce_sum(input_tensor=crossent)\n            total_size = tf.reduce_sum(input_tensor=weights)\n            crossent = tf.math.divide_no_nan(crossent, total_size)\n        elif sum_over_timesteps and sum_over_batch:\n            crossent = tf.reduce_sum(input_tensor=crossent)\n            total_count = tf.cast(\n                tf.math.count_nonzero(weights), crossent.dtype)\n            crossent = tf.math.divide_no_nan(crossent, total_count)\n        else:\n            crossent = tf.reshape(crossent, tf.shape(input=logits)[0:2])\n            if average_across_timesteps or average_across_batch:\n                reduce_axis = [0] if average_across_batch else [1]\n                crossent = tf.reduce_sum(\n                    input_tensor=crossent, axis=reduce_axis)\n                total_size = tf.reduce_sum(\n                    input_tensor=weights, axis=reduce_axis)\n                crossent = tf.math.divide_no_nan(crossent, total_size)\n            elif sum_over_timesteps or sum_over_batch:\n                reduce_axis = [0] if sum_over_batch else [1]\n                crossent = tf.reduce_sum(\n                    input_tensor=crossent, axis=reduce_axis)\n                total_count = tf.cast(\n                    tf.math.count_nonzero(weights, axis=reduce_axis),\n                    dtype=crossent.dtype)\n                crossent = tf.math.divide_no_nan(crossent, total_count)\n        return crossent\n\n\nclass SequenceLoss(tf.keras.losses.Loss):\n    \"\"\"Weighted cross-entropy loss for a sequence of logits.\"\"\"\n\n    def __init__(self,\n                 average_across_timesteps=False,\n                 average_across_batch=False,\n                 sum_over_timesteps=True,\n                 sum_over_batch=True,\n                 softmax_loss_function=None,\n                 name=None):\n        super(SequenceLoss, self).__init__(name=name)\n        self.average_across_timesteps = average_across_timesteps\n        self.average_across_batch = average_across_batch\n        self.sum_over_timesteps = sum_over_timesteps\n        self.sum_over_batch = sum_over_batch\n        self.softmax_loss_function = softmax_loss_function\n\n    def __call__(self, y_true, y_pred, sample_weight=None):\n        \"\"\"Override the parent __call__ to have a customized reduce\n        behavior.\"\"\"\n        return sequence_loss(\n            y_pred,\n            y_true,\n            sample_weight,\n            average_across_timesteps=self.average_across_timesteps,\n            average_across_batch=self.average_across_batch,\n            sum_over_timesteps=self.sum_over_timesteps,\n            sum_over_batch=self.sum_over_batch,\n            softmax_loss_function=self.softmax_loss_function,\n            name=self.name)\n\n    def call(self, y_true, y_pred):\n        # Skip this method since the __call__ contains real implementation.\n        pass\n", "path": "tensorflow_addons/seq2seq/loss.py"}]}