<p dir="auto">When using the "deep" functionality in the Wide and Deep Learning tutorial on Python 3.4, I get the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError: Duplicate feature column key found for column: education_embedding. This usually means that the column is almost identical to another column, and one must be discarded."><pre class="notranslate"><code class="notranslate">ValueError: Duplicate feature column key found for column: education_embedding. This usually means that the column is almost identical to another column, and one must be discarded.
</code></pre></div>
<p dir="auto">Looks like a bug in tensorflow/contrib/layers/python/layers/feature_column.py in the _EmbeddingColumn class. The key(self) property is plagued by this bug: <a href="https://bugs.python.org/issue24931" rel="nofollow">https://bugs.python.org/issue24931</a></p>
<p dir="auto">So instead of coming out with a nice unique key, we get the following key for all _EmbeddingColumn instances: '_EmbeddingColumn()'</p>
<p dir="auto">This causes the feature_column_ops.py's check_feature_columns() function to determine that the second _EmbeddingColumn instance is a duplicate since they keys of all of them are the same.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?</h3>
<p dir="auto">I found this StackOverflow thread <a href="http://stackoverflow.com/questions/39249704/tensorflow-valueerror-duplicate-feature-column-key-found-for-column/" rel="nofollow">http://stackoverflow.com/questions/39249704/tensorflow-valueerror-duplicate-feature-column-key-found-for-column/</a> where someone had the same problem, with no answers.  So I did the debugging and answered the question.</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Fedora Core 23</p>
<p dir="auto">Installed version of CUDA and cuDNN: 8 and 5.1<br>
(please attach the output of <code class="notranslate">ls -l /path/to/cuda/lib/libcud*</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[jpangburn@localhost examples]$ ls /usr/local/cuda/lib64/libcud*
/usr/local/cuda/lib64/libcudadevrt.a
/usr/local/cuda/lib64/libcudart.so
/usr/local/cuda/lib64/libcudart.so.8.0
/usr/local/cuda/lib64/libcudart.so.8.0.27
/usr/local/cuda/lib64/libcudart_static.a
/usr/local/cuda/lib64/libcudnn.so
/usr/local/cuda/lib64/libcudnn.so.5
/usr/local/cuda/lib64/libcudnn.so.5.1.5
/usr/local/cuda/lib64/libcudnn_static.a
"><pre class="notranslate"><code class="notranslate">[jpangburn@localhost examples]$ ls /usr/local/cuda/lib64/libcud*
/usr/local/cuda/lib64/libcudadevrt.a
/usr/local/cuda/lib64/libcudart.so
/usr/local/cuda/lib64/libcudart.so.8.0
/usr/local/cuda/lib64/libcudart.so.8.0.27
/usr/local/cuda/lib64/libcudart_static.a
/usr/local/cuda/lib64/libcudnn.so
/usr/local/cuda/lib64/libcudnn.so.5
/usr/local/cuda/lib64/libcudnn.so.5.1.5
/usr/local/cuda/lib64/libcudnn_static.a

</code></pre></div>
<p dir="auto">If installed from source, provide</p>
<ol dir="auto">
<li>The commit hash (<code class="notranslate">git rev-parse HEAD</code>)</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[jpangburn@localhost examples]$ pwd
/home/jpangburn/open_source/tensorflow/tensorflow/examples
[jpangburn@localhost examples]$ git rev-parse HEAD
17d0e46e6cc31af0bcf6e80ff4c5670d233b4940
"><pre class="notranslate"><code class="notranslate">[jpangburn@localhost examples]$ pwd
/home/jpangburn/open_source/tensorflow/tensorflow/examples
[jpangburn@localhost examples]$ git rev-parse HEAD
17d0e46e6cc31af0bcf6e80ff4c5670d233b4940

</code></pre></div>
<ol dir="auto">
<li>The output of <code class="notranslate">bazel version</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[jpangburn@localhost examples]$ bazel version
.
Build label: 0.3.1
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jul 29 09:09:52 2016 (1469783392)
Build timestamp: 1469783392
Build timestamp as int: 1469783392
"><pre class="notranslate"><code class="notranslate">[jpangburn@localhost examples]$ bazel version
.
Build label: 0.3.1
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jul 29 09:09:52 2016 (1469783392)
Build timestamp: 1469783392
Build timestamp as int: 1469783392

</code></pre></div>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">Be in the "tensorflow/tensorflow/examples/learn" directory.  Patch the wide_n_deep_tutorial.py to work in Python 3 by replacing "urllib" instances with "urllib.request".  Then run it "python3 wide_n_deep_tutorial_py3.py --train_data=train_data --test_data=test_data --model_type=wide_n_deep", obviously replacing the train_data and test_data files with your own copies, or omitting those switches altogether to download the files fresh.</p>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<p dir="auto">I worked around the problem by creating a subclass of _EmbeddedColumn and using that, but not sure my key(self) property implementation is working because the accuracy result with wide_n_deep is worse than with just "wide".  You can see this in my answer to the StackOverflow question at <a href="http://stackoverflow.com/questions/39249704/tensorflow-valueerror-duplicate-feature-column-key-found-for-column/39268045#39268045" rel="nofollow">http://stackoverflow.com/questions/39249704/tensorflow-valueerror-duplicate-feature-column-key-found-for-column/39268045#39268045</a></p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;wide_n_deep_tutorial_py3.py&quot;, line 220, in &lt;module&gt;
    tf.app.run()
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/python/platform/app.py&quot;, line 30, in run
    sys.exit(main(sys.argv))
  File &quot;wide_n_deep_tutorial_py3.py&quot;, line 216, in main
    train_and_eval()
  File &quot;wide_n_deep_tutorial_py3.py&quot;, line 204, in train_and_eval
    m.fit(input_fn=lambda: input_fn(df_train), steps=FLAGS.train_steps)
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py&quot;, line 219, in fit
    max_steps=max_steps)
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py&quot;, line 479, in _train_model
    train_op, loss_op = self._get_train_ops(features, targets)
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py&quot;, line 166, in _get_train_ops
    logits = self._logits(features, is_training=True)
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py&quot;, line 244, in _logits
    dnn_feature_columns = self._get_dnn_feature_columns()
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py&quot;, line 208, in _get_dnn_feature_columns
    feature_column_ops.check_feature_columns(self._dnn_feature_columns)
  File &quot;/usr/lib/python3.4/site-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py&quot;, line 318, in check_feature_columns
    f.name))
ValueError: Duplicate feature column key found for column: education_embedding. This usually means that the column is almost identical to another column, and one must be discarded.
"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "wide_n_deep_tutorial_py3.py", line 220, in &lt;module&gt;
    tf.app.run()
  File "/usr/lib/python3.4/site-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "wide_n_deep_tutorial_py3.py", line 216, in main
    train_and_eval()
  File "wide_n_deep_tutorial_py3.py", line 204, in train_and_eval
    m.fit(input_fn=lambda: input_fn(df_train), steps=FLAGS.train_steps)
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 219, in fit
    max_steps=max_steps)
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 479, in _train_model
    train_op, loss_op = self._get_train_ops(features, targets)
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 166, in _get_train_ops
    logits = self._logits(features, is_training=True)
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 244, in _logits
    dnn_feature_columns = self._get_dnn_feature_columns()
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 208, in _get_dnn_feature_columns
    feature_column_ops.check_feature_columns(self._dnn_feature_columns)
  File "/usr/lib/python3.4/site-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py", line 318, in check_feature_columns
    f.name))
ValueError: Duplicate feature column key found for column: education_embedding. This usually means that the column is almost identical to another column, and one must be discarded.

</code></pre></div>
<p dir="auto">I build TF on a biweekly basis, the last successful build was done 3 days ago (29th Oct), but the build job is failing today, while there has been no changes in the installation script.</p>
<p dir="auto">The build happens on an amazon linux machine with cuda 9.0 and cudnn 7.3, using the following command,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bazel build --config=opt --config=mk1 --config=cuda --copt=-mavx --copt=-msse4.2 --copt=-msse4.1 //tensorflow/tools/pip_package:build_pip_package"><pre class="notranslate"><code class="notranslate">bazel build --config=opt --config=mk1 --config=cuda --copt=-mavx --copt=-msse4.2 --copt=-msse4.1 //tensorflow/tools/pip_package:build_pip_package
</code></pre></div>
<p dir="auto">Some of the ralevent flags are set as follows,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="export TF_NEED_CUDA=1
export TF_CUDA_VERSION=&quot;$(nvcc --version | sed -n 's/^.*release \(.*\),.*/\1/p')&quot;
    export TF_CUDNN_VERSION=&quot;$(sed -n 's/^#define CUDNN_MAJOR\s*\(.*\).*/\1/p' $CUDNN_INSTALL_PATH/include/cudnn.h)&quot;"><pre class="notranslate"><code class="notranslate">export TF_NEED_CUDA=1
export TF_CUDA_VERSION="$(nvcc --version | sed -n 's/^.*release \(.*\),.*/\1/p')"
    export TF_CUDNN_VERSION="$(sed -n 's/^#define CUDNN_MAJOR\s*\(.*\).*/\1/p' $CUDNN_INSTALL_PATH/include/cudnn.h)"
</code></pre></div>
<p dir="auto">The job failes with the following error message,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+ bazel build --config=opt --config=cuda --copt=-mavx --copt=-msse4.2 --copt=-msse4.1 //tensorflow/tools/pip_package:build_pip_package
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/home/ec2-user/workspace/build-dl-package/src/deep-learning/tensorflow/tensorflow/tools/bazel.rc
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/ec2-user/workspace/build-dl-package/src/deep-learning/tensorflow/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu//bin/python --action_env PYTHON_LIB_PATH=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu/lib/python2.7/site-packages --python_path=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu//bin/python --define with_jemalloc=true --define with_hdfs_support=true --define with_aws_support=true --define with_kafka_support=true --action_env TF_NEED_OPENCL_SYCL=0 --action_env TF_NEED_CUDA=1 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda --action_env TF_CUDA_VERSION=9.0 --action_env CUDNN_INSTALL_PATH=/usr/local/cuda-9.0 --action_env TF_CUDNN_VERSION=7 --action_env NCCL_INSTALL_PATH=/usr/local/nccl --action_env TF_NCCL_VERSION=2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.0,3.5,5.2 --action_env LD_LIBRARY_PATH=/usr/java/jdk1.7.0_67/jre/lib/amd64/server:/usr/java/jdk1.7.0_67/jre/lib/amd64:/usr/java/jdk1.7.0_67/jre/../lib/amd64:/usr/lib/hadoop2.6/lib/native/:/usr/local/cuda-8.0/lib64:/usr/java/jdk1.8.0_121//jre/lib/amd64/server:/usr/lib/torch/install/lib:/usr/local/mpi/lib: --action_env TF_CUDA_CLANG=0 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/gcc --config=cuda --define with_mpi_support=true --define grpc_no_ares=true
ERROR: Config value cuda is not defined in any .rc file"><pre class="notranslate"><code class="notranslate">+ bazel build --config=opt --config=cuda --copt=-mavx --copt=-msse4.2 --copt=-msse4.1 //tensorflow/tools/pip_package:build_pip_package
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/home/ec2-user/workspace/build-dl-package/src/deep-learning/tensorflow/tensorflow/tools/bazel.rc
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/ec2-user/workspace/build-dl-package/src/deep-learning/tensorflow/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu//bin/python --action_env PYTHON_LIB_PATH=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu/lib/python2.7/site-packages --python_path=/usr/lib/a-4.2.9-py-2.7.13-dl-gpu//bin/python --define with_jemalloc=true --define with_hdfs_support=true --define with_aws_support=true --define with_kafka_support=true --action_env TF_NEED_OPENCL_SYCL=0 --action_env TF_NEED_CUDA=1 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda --action_env TF_CUDA_VERSION=9.0 --action_env CUDNN_INSTALL_PATH=/usr/local/cuda-9.0 --action_env TF_CUDNN_VERSION=7 --action_env NCCL_INSTALL_PATH=/usr/local/nccl --action_env TF_NCCL_VERSION=2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.0,3.5,5.2 --action_env LD_LIBRARY_PATH=/usr/java/jdk1.7.0_67/jre/lib/amd64/server:/usr/java/jdk1.7.0_67/jre/lib/amd64:/usr/java/jdk1.7.0_67/jre/../lib/amd64:/usr/lib/hadoop2.6/lib/native/:/usr/local/cuda-8.0/lib64:/usr/java/jdk1.8.0_121//jre/lib/amd64/server:/usr/lib/torch/install/lib:/usr/local/mpi/lib: --action_env TF_CUDA_CLANG=0 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/gcc --config=cuda --define with_mpi_support=true --define grpc_no_ares=true
ERROR: Config value cuda is not defined in any .rc file
</code></pre></div>
<p dir="auto">The build is run against <code class="notranslate">r1.11</code> along with commit <code class="notranslate">89979f42e827d9eb5c349259a5aa2ec32d38c86a</code> cherry-picked.</p>
<p dir="auto">Should I change something?<br>
Thank you.</p>