<p dir="auto">All I'm really trying to do is read a large array from a tfrecords file. I can do it so long that it's below a 8million ints but above that I get a unicode error, which doesn't seem to make much sense to me. Also, it worth noting that this script works using r0.9. I have about 64GB of ram on my computer so I doubt it's a memory issue.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?</h3>
<p dir="auto">None</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:<br>
Linux Mint 18 Sarah (GNU/Linux 4.4.0-47-generic x86_64)</p>
<p dir="auto">Installed version of CUDA and cuDNN:<br>
not installed</p>
<p dir="auto">If installed from binary pip package, provide:</p>
<ol dir="auto">
<li>
<p dir="auto">A link to the pip package you installed:<br>
<a href="https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0-cp27-none-linux_x86_64.whl</a></p>
</li>
<li>
<p dir="auto">The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>.<br>
python -c "import tensorflow; print(tensorflow.<strong>version</strong>)"</p>
</li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np

dim_size = 9000000

# Create the array
array = np.zeros([dim_size], dtype=int)

example = tf.train.Example(
    features = tf.train.Features(
        feature = {
            'array': tf.train.Feature(
                bytes_list= tf.train.BytesList(value=[array.tostring()])
            )
        }
    )
)

# Write the example to disk
writer = tf.python_io.TFRecordWriter('temp.tfrecords')
writer.write(example.SerializeToString())
writer.close()

# Define input pipeline
queue = tf.train.string_input_producer(['temp.tfrecords'], shuffle=False, num_epochs=1)
reader = tf.TFRecordReader()
_, se = reader.read(queue)

#Define the parsed dict
parsed = tf.parse_single_example(
    serialized=se,
    features={
        &quot;array&quot;:tf.FixedLenFeature([], tf.string),
    }
)

# Decode the example
tens = tf.decode_raw(parsed['array'], tf.int64)

# Define the init op.
init_op = tf.group(
    tf.initialize_all_variables(),
    tf.initialize_local_variables()
)
with tf.Session() as sess:
    sess.run(init_op)
    # Define coordinator and threads
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord, sess=sess)

    # Run the reading operation
    print sess.run([tens])
    coord.request_stop()
    coord.join(threads)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import numpy as np

dim_size = 9000000

# Create the array
array = np.zeros([dim_size], dtype=int)

example = tf.train.Example(
    features = tf.train.Features(
        feature = {
            'array': tf.train.Feature(
                bytes_list= tf.train.BytesList(value=[array.tostring()])
            )
        }
    )
)

# Write the example to disk
writer = tf.python_io.TFRecordWriter('temp.tfrecords')
writer.write(example.SerializeToString())
writer.close()

# Define input pipeline
queue = tf.train.string_input_producer(['temp.tfrecords'], shuffle=False, num_epochs=1)
reader = tf.TFRecordReader()
_, se = reader.read(queue)

#Define the parsed dict
parsed = tf.parse_single_example(
    serialized=se,
    features={
        "array":tf.FixedLenFeature([], tf.string),
    }
)

# Decode the example
tens = tf.decode_raw(parsed['array'], tf.int64)

# Define the init op.
init_op = tf.group(
    tf.initialize_all_variables(),
    tf.initialize_local_variables()
)
with tf.Session() as sess:
    sess.run(init_op)
    # Define coordinator and threads
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord, sess=sess)

    # Run the reading operation
    print sess.run([tens])
    coord.request_stop()
    coord.join(threads)
</code></pre></div>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<p dir="auto">This works using r0.9 and it also works if you lower the dim_size below 800000. So it seems to be an issue with the size and r0.11. I also get the same results if I use a float array as opposed to an int.</p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: Could not parse example input, value: '
?Ī&quot;
?Ī&quot;
array?Ī&quot;
?Ī&quot;
?Ī&quot;
Traceback (most recent call last):
  File &quot;temp.py&quot;, line 52, in &lt;module&gt;
    print sess.run([tens])
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 717, in run
    run_metadata_ptr)
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 915, in _run
    feed_dict_string, options, run_metadata)
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 965, in _do_run
    target_list, options, run_metadata)
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 972, in _do_call
    return fn(*args)
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 954, in _run_fn
    status, run_metadata)
  File &quot;/usr/lib/python2.7/contextlib.py&quot;, line 24, in __exit__
    self.gen.next()
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors.py&quot;, line 462, in raise_exception_on_not_ok_status
    compat.as_text(pywrap_tensorflow.TF_Message(status)),
  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/compat.py&quot;, line 82, in as_text
    return bytes_or_text.decode('utf-8')
  File &quot;/usr/lib/python2.7/encodings/utf_8.py&quot;, line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9b in position 40: invalid start byte"><pre class="notranslate"><code class="notranslate">W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: Could not parse example input, value: '
?Ī"
?Ī"
array?Ī"
?Ī"
?Ī"
Traceback (most recent call last):
  File "temp.py", line 52, in &lt;module&gt;
    print sess.run([tens])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 972, in _do_call
    return fn(*args)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 954, in _run_fn
    status, run_metadata)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors.py", line 462, in raise_exception_on_not_ok_status
    compat.as_text(pywrap_tensorflow.TF_Message(status)),
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/compat.py", line 82, in as_text
    return bytes_or_text.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9b in position 40: invalid start byte
</code></pre></div>
<p dir="auto">I have already read this thread: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="205768541" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/7311" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/7311/hovercard" href="https://github.com/tensorflow/tensorflow/issues/7311">#7311</a> but I am still encountering this issue with version 1.3.0. Was wondering if anyone knows why?</p>
<p dir="auto">Tested on 2 systems:<br>
OS: Windows 10, Ubuntu 16.04<br>
Tensorflow installed from source,<br>
Tensorflow version: 1.3.0<br>
One system running tensorflow non gpu and another system running CUDA 8 and cuDNN v6<br>
GPU model: none and GTX Titan Z</p>
<p dir="auto">The error I get is:</p>
<p dir="auto">`  File "C:\Users\benja\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status<br>
pywrap_tensorflow.TF_GetCode(status))<br>
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not parse example input, value: '<br>
�ם9<br>
ѻ�<br>
�csv�Ȼ�<br>
Ļ�<br>
���<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cq/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cq">@cq</a>��.�������M�B</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "D:\write_to_tfrecords.py", line 581, in <br>
main()<br>
File "D:\write_to_tfrecords.py", line 577, in main<br>
read_dataset_from_tfrecords()<br>
File "D:\write_to_tfrecords.py", line 554, in read_dataset_from_tfrecords<br>
image_dataset, csv_dataset = sess.run([image_out_reshaped, csv_out_reshaped])<br>
File "C:\Users\benja\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 895, in run<br>
run_metadata_ptr)<br>
File "C:\Users\benja\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1124, in _run<br>
feed_dict_tensor, options, run_metadata)<br>
File "C:\Users\benja\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run<br>
options, run_metadata)<br>
File "C:\Users\benja\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call<br>
raise type(e)(node_def, op, message)<br>
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not parse example input, value: '<br>
�ם9<br>
ѻ�<br>
�csv�Ȼ�<br>
Ļ�<br>
���<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cq/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cq">@cq</a>��.�������M�B</p>
<blockquote>
<blockquote>
<blockquote>
</blockquote>
</blockquote>
</blockquote>
<p dir="auto">`</p>
<p dir="auto">When I try to read .tfrecords files that are less than 64mb this error does not occur.</p>
<p dir="auto">My code is here:<br>
`        with tf.Session() as sess:<br>
try:<br>
feature = {'images': tf.FixedLenFeature([], tf.string),<br>
'csv': tf.FixedLenFeature([], tf.string)<br>
}</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="            filename_queue = tf.train.string_input_producer([data_path], num_epochs=1)

            reader = tf.TFRecordReader()

            _, serialized_example = reader.read(filename_queue)

            features = tf.parse_single_example(serialized_example, features=feature)

            image_out = tf.decode_raw(features['images'], tf.uint8)
            csv_out = tf.decode_raw(features['csv'], tf.float32)

            image_out_reshaped = tf.reshape(image_out, [1000, 200, 200, 3])
            csv_out_reshaped = tf.reshape(csv_out, [1000, 6])

            sess.run(tf.global_variables_initializer())
            sess.run(tf.local_variables_initializer())

            # Create a coordinator and run all QueueRunner objects
            coord = tf.train.Coordinator()
            threads = tf.train.start_queue_runners(coord=coord)

            image_dataset, csv_dataset = sess.run([image_out_reshaped, csv_out_reshaped])

            coord.request_stop()
            coord.join(threads)"><pre class="notranslate"><code class="notranslate">            filename_queue = tf.train.string_input_producer([data_path], num_epochs=1)

            reader = tf.TFRecordReader()

            _, serialized_example = reader.read(filename_queue)

            features = tf.parse_single_example(serialized_example, features=feature)

            image_out = tf.decode_raw(features['images'], tf.uint8)
            csv_out = tf.decode_raw(features['csv'], tf.float32)

            image_out_reshaped = tf.reshape(image_out, [1000, 200, 200, 3])
            csv_out_reshaped = tf.reshape(csv_out, [1000, 6])

            sess.run(tf.global_variables_initializer())
            sess.run(tf.local_variables_initializer())

            # Create a coordinator and run all QueueRunner objects
            coord = tf.train.Coordinator()
            threads = tf.train.start_queue_runners(coord=coord)

            image_dataset, csv_dataset = sess.run([image_out_reshaped, csv_out_reshaped])

            coord.request_stop()
            coord.join(threads)
</code></pre></div>
<p dir="auto">`</p>