<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS, Linux</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): Anaconda binary</li>
<li>TensorFlow version (use command below): 1.14</li>
<li>Python version: 3.7.3</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: N/A</li>
<li>GPU model and memory: N/A</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture<br>
<a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with: 1. TF 1.0: <code class="notranslate">python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</code> 2. TF 2.0: <code class="notranslate">python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"</code></p>
<p dir="auto">$ python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"<br>
unknown 1.14.0</p>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">cloudpickle cannot unpickle <code class="notranslate">tf.keras</code> because the <code class="notranslate">deprecation_wrapper</code> introduced in 1.14.</p>
<p dir="auto">In one python session:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import cloudpickle
import tensorflow.keras as K
with open(&quot;/tmp/K.pkl&quot;, &quot;wb&quot;) as f:
  cloudpickle.dump(K, f)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">cloudpickle</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">keras</span> <span class="pl-k">as</span> <span class="pl-v">K</span>
<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">"/tmp/K.pkl"</span>, <span class="pl-s">"wb"</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
  <span class="pl-s1">cloudpickle</span>.<span class="pl-en">dump</span>(<span class="pl-v">K</span>, <span class="pl-s1">f</span>)</pre></div>
<p dir="auto">Then start another python session:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import cloudpickle
with open(&quot;/tmp/K.pkl&quot;, &quot;rb&quot;) as f:
  cloudpickle.load(f)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">cloudpickle</span>
<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">"/tmp/K.pkl"</span>, <span class="pl-s">"rb"</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
  <span class="pl-s1">cloudpickle</span>.<span class="pl-en">load</span>(<span class="pl-s1">f</span>)</pre></div>
<p dir="auto">Error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 2, in &lt;module&gt;
  File &quot;/Users/meng/conda/envs/tf-1.14/lib/python3.7/site-packages/tensorflow/python/util/deprecation_wrapper.py&quot;, line 148, in __setstate__
    sys.modules[d]._dw_wrapped_module,
KeyError: 'tensorflow.keras'"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 2, in &lt;module&gt;
  File "/Users/meng/conda/envs/tf-1.14/lib/python3.7/site-packages/tensorflow/python/util/deprecation_wrapper.py", line 148, in __setstate__
    sys.modules[d]._dw_wrapped_module,
KeyError: 'tensorflow.keras'
</code></pre></div>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">The same code worked in TensorFlow 1.13. In TensorFlow 1.13, <code class="notranslate">K.__module__</code> does not exist, while in TensorFlow 1.14, <code class="notranslate">K.__module__</code> is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="'tensorflow.python.util.deprecation_wrapper'"><pre class="notranslate"><code class="notranslate">'tensorflow.python.util.deprecation_wrapper'
</code></pre></div>
<p dir="auto">It makes the <code class="notranslate">tensorflow.keras</code> module not loaded during unpickling.</p>
<p dir="auto">There are multiple applications use pickle to ship TensorFlow code to a remote machine to unpickle and execute, e.g., Spark, horovod.spark. So the behavior change would fail the jobs.</p>
<p dir="auto">Btw, one workaround is to do <code class="notranslate">from tensorflow import keras as K</code> instead of <code class="notranslate">import tensorflow.keras as K</code>. The diff is that <code class="notranslate">K.__name__</code> is <code class="notranslate">'tensorflow.python.keras.api._v1.keras'</code> in the first case and <code class="notranslate">tensorflow.keras</code> in the second. And pickle uses <code class="notranslate">__name__</code> to get global names: <a href="https://github.com/python/cpython/blob/3.7/Lib/pickle.py#L952">https://github.com/python/cpython/blob/3.7/Lib/pickle.py#L952</a></p>
<p dir="auto">cc: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/alsrgv/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/alsrgv">@alsrgv</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/hanyucui/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hanyucui">@hanyucui</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/annarev/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/annarev">@annarev</a></p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Provide a reproducible test case that is the bare minimum necessary to generate the problem.</p>
<p dir="auto">See above.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes (code included below in the issue)</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.14.3</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): pip</li>
<li>TensorFlow version (use command below): v2.0.0-beta1-5101-gc75bb66a99 2.0.0-rc0</li>
<li>Python version: Python 3.6.7 :: Anaconda, Inc.</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: N/A</li>
<li>GPU model and memory: N/A</li>
</ul>
<p dir="auto">Using cloudpickle to serialize a Python function that uses <code class="notranslate">tf.keras.Sequential</code> fails with a recursion error.</p>
<p dir="auto"><strong>Note</strong> that this works with <code class="notranslate">tensorflow==1.14.0</code>.</p>
<p dir="auto">I imagine it also fails with other things, not just <code class="notranslate">tf.keras.Sequential</code>.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import cloudpickle  # cloudpickle.__version__ == '1.2.1'
import tensorflow as tf  # tf.__version__ == '2.0.0-rc0'

def f():
    tf.keras.Sequential

cloudpickle.loads(cloudpickle.dumps(f))  # This fails."><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">cloudpickle</span>  <span class="pl-c"># cloudpickle.__version__ == '1.2.1'</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>  <span class="pl-c"># tf.__version__ == '2.0.0-rc0'</span>

<span class="pl-k">def</span> <span class="pl-en">f</span>():
    <span class="pl-s1">tf</span>.<span class="pl-s1">keras</span>.<span class="pl-v">Sequential</span>

<span class="pl-s1">cloudpickle</span>.<span class="pl-en">loads</span>(<span class="pl-s1">cloudpickle</span>.<span class="pl-en">dumps</span>(<span class="pl-s1">f</span>))  <span class="pl-c"># This fails.</span></pre></div>
<p dir="auto">The last line fails with</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
&lt;ipython-input-23-25cc307e6227&gt; in &lt;module&gt;
----&gt; 1 cloudpickle.loads(cloudpickle.dumps(f))

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48 
     49   def __getattr__(self, item):
---&gt; 50     module = self._load()
     51     return getattr(module, item)
     52 

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in _load(self)
     42   def _load(self):
     43     &quot;&quot;&quot;Import the target module and insert it into the parent's namespace.&quot;&quot;&quot;
---&gt; 44     module = _importlib.import_module(self.__name__)
     45     self._parent_module_globals[self._local_name] = module
     46     self.__dict__.update(module.__dict__)

... last 2 frames repeated, from the frame below ...

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48 
     49   def __getattr__(self, item):
---&gt; 50     module = self._load()
     51     return getattr(module, item)
     52 

RecursionError: maximum recursion depth exceeded while calling a Python object"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
&lt;ipython-input-23-25cc307e6227&gt; in &lt;module&gt;
----&gt; 1 cloudpickle.loads(cloudpickle.dumps(f))

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48 
     49   def __getattr__(self, item):
---&gt; 50     module = self._load()
     51     return getattr(module, item)
     52 

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in _load(self)
     42   def _load(self):
     43     """Import the target module and insert it into the parent's namespace."""
---&gt; 44     module = _importlib.import_module(self.__name__)
     45     self._parent_module_globals[self._local_name] = module
     46     self.__dict__.update(module.__dict__)

... last 2 frames repeated, from the frame below ...

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48 
     49   def __getattr__(self, item):
---&gt; 50     module = self._load()
     51     return getattr(module, item)
     52 

RecursionError: maximum recursion depth exceeded while calling a Python object
</code></pre></div>
<p dir="auto">See <a href="https://stackoverflow.com/questions/57750920/ray-tensorflow-gpu-2-0-recursionerror/57761034#57761034" rel="nofollow">https://stackoverflow.com/questions/57750920/ray-tensorflow-gpu-2-0-recursionerror/57761034#57761034</a></p>