<p dir="auto">In a series simple tensorflow programs I obtain memory leaks (unbounded growth of CPU memory).<br>
On original program on a computer with 64GB of RAM this leak is about 640 megabytes per hour (1% of total memory).</p>
<h3 dir="auto">Plots of computer's memory over time:</h3>
<h4 dir="auto">Long time scale picture:</h4>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/1411079/24841025/1064f380-1d2f-11e7-8738-943be211bb1b.png"><img src="https://cloud.githubusercontent.com/assets/1411079/24841025/1064f380-1d2f-11e7-8738-943be211bb1b.png" alt="unknown-1" style="max-width: 100%;"></a></p>
<h4 dir="auto">short time scale picture:</h4>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/1411079/24841026/12d78984-1d2f-11e7-8efd-62bd8e46b46c.png"><img src="https://cloud.githubusercontent.com/assets/1411079/24841026/12d78984-1d2f-11e7-8efd-62bd8e46b46c.png" alt="unknown" style="max-width: 100%;"></a></p>
<h2 dir="auto">Problem description</h2>
<p dir="auto">The original program was more advanced and included RNNs/Saving/Loading etc.. but I "narrowed it down" to a simple for loop with no gradient descent where memory grows over time without bound.<br>
Tested on Fedora 25 and Mac OSX 10.11.5. Issue occurs when running on single GPU (Titan X Pascal) or on CPU. Varying the sizes of the variables in the graph only changes the degree of growth, but does not prevent the effect from occurring. This issue occurs on tensorflow 0.12 and on current tensorflow 1.0.1. No custom code was used. Tensorflow was installed using pip in both cases (pre-compiled binary. Each time this was <code class="notranslate">pip3 install tensorflow-gpu</code>). Using CUDA 8.0,  CuDNN v5 [though this should not impact the use-case, since no cudnn kernels are being used]. GPU is a Titan X Pascal 12GB of VRAM (not Titan Xp).</p>
<h2 dir="auto">To reproduce:</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import argparse
import psutil

from os import getpid
import tensorflow as tf
import numpy as np

def fc(inputs, output_size):
    with tf.variable_scope(&quot;FC&quot;):
        input_size = inputs.get_shape()[-1].value
        W = tf.get_variable(&quot;W&quot;, shape=[input_size, output_size])
        b = tf.get_variable(&quot;b&quot;, shape=[output_size], initializer=tf.constant_initializer(0))
        out = tf.nn.xw_plus_b(inputs, W, b)
    return out

def create_model(input_size, output_size):
    # model placeholders:
    with tf.variable_scope(&quot;Inputs&quot;):
        input_placeholder = tf.placeholder(
            tf.float32, [None, input_size], name=&quot;input_placeholder&quot;
        )
    # meaningless function of inputs
    op = tf.reduce_mean(tf.reduce_sum(fc(input_placeholder, output_size), 1))
    return input_placeholder, op

def parse_args(args=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('--max_epochs', type=int, default=1000)
    parser.add_argument('--batch_size', type=int, default=7000)
    parser.add_argument('--input_size', type=int, default=100)
    parser.add_argument('--output_size', type=int, default=100)
    parser.add_argument('--device', type=str, default=&quot;gpu:0&quot;)
    return parser.parse_args(args=args)

def create_batches(inputs, input_size, batch_size, n):
    batches = []
    for i in range(n):
        X = np.random.uniform(-1.0, 1.0, size=(batch_size, input_size))
        batches.append({inputs: X})
    return batches

def main():
    args = parse_args()
    session_conf = tf.ConfigProto(allow_soft_placement=True)
    np.random.seed(1234)
    process = psutil.Process(getpid())

    with tf.Session(config=session_conf) as session, tf.device(args.device):
        inputs, op = create_model(args.input_size, args.output_size)
        session.run(tf.global_variables_initializer())
        batches = create_batches(inputs, args.input_size, args.batch_size, 20)

        for epoch in range(args.max_epochs):
            before = process.memory_percent()
            for feed_dict in batches:
                session.run(op, feed_dict)
            after = process.memory_percent()
            print(&quot;MEMORY CHANGE %.4f -&gt; %.4f&quot; % (before, after))

if __name__ == &quot;__main__&quot;:
    main()"><pre class="notranslate"><code class="notranslate">import argparse
import psutil

from os import getpid
import tensorflow as tf
import numpy as np

def fc(inputs, output_size):
    with tf.variable_scope("FC"):
        input_size = inputs.get_shape()[-1].value
        W = tf.get_variable("W", shape=[input_size, output_size])
        b = tf.get_variable("b", shape=[output_size], initializer=tf.constant_initializer(0))
        out = tf.nn.xw_plus_b(inputs, W, b)
    return out

def create_model(input_size, output_size):
    # model placeholders:
    with tf.variable_scope("Inputs"):
        input_placeholder = tf.placeholder(
            tf.float32, [None, input_size], name="input_placeholder"
        )
    # meaningless function of inputs
    op = tf.reduce_mean(tf.reduce_sum(fc(input_placeholder, output_size), 1))
    return input_placeholder, op

def parse_args(args=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('--max_epochs', type=int, default=1000)
    parser.add_argument('--batch_size', type=int, default=7000)
    parser.add_argument('--input_size', type=int, default=100)
    parser.add_argument('--output_size', type=int, default=100)
    parser.add_argument('--device', type=str, default="gpu:0")
    return parser.parse_args(args=args)

def create_batches(inputs, input_size, batch_size, n):
    batches = []
    for i in range(n):
        X = np.random.uniform(-1.0, 1.0, size=(batch_size, input_size))
        batches.append({inputs: X})
    return batches

def main():
    args = parse_args()
    session_conf = tf.ConfigProto(allow_soft_placement=True)
    np.random.seed(1234)
    process = psutil.Process(getpid())

    with tf.Session(config=session_conf) as session, tf.device(args.device):
        inputs, op = create_model(args.input_size, args.output_size)
        session.run(tf.global_variables_initializer())
        batches = create_batches(inputs, args.input_size, args.batch_size, 20)

        for epoch in range(args.max_epochs):
            before = process.memory_percent()
            for feed_dict in batches:
                session.run(op, feed_dict)
            after = process.memory_percent()
            print("MEMORY CHANGE %.4f -&gt; %.4f" % (before, after))

if __name__ == "__main__":
    main()
</code></pre></div>
<p dir="auto">Output will be (exact numbers are percentages of computer's ram, so should change based on hardware, but main point is that memory continues to grow when the program has no variation between graph runs, batches are all the same size, no randomness is left in the program, etc.):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="MEMORY CHANGE 1.2427 -&gt; 1.3101
MEMORY CHANGE 1.3101 -&gt; 1.3103
MEMORY CHANGE 1.3103 -&gt; 1.3104
MEMORY CHANGE 1.3104 -&gt; 1.3106
MEMORY CHANGE 1.3106 -&gt; 1.3108
MEMORY CHANGE 1.3108 -&gt; 1.3108
MEMORY CHANGE 1.3108 -&gt; 1.3108
...
MEMORY CHANGE 1.3108 -&gt; 1.3109
...
MEMORY CHANGE 1.3109 -&gt; 1.3110
..."><pre class="notranslate"><code class="notranslate">MEMORY CHANGE 1.2427 -&gt; 1.3101
MEMORY CHANGE 1.3101 -&gt; 1.3103
MEMORY CHANGE 1.3103 -&gt; 1.3104
MEMORY CHANGE 1.3104 -&gt; 1.3106
MEMORY CHANGE 1.3106 -&gt; 1.3108
MEMORY CHANGE 1.3108 -&gt; 1.3108
MEMORY CHANGE 1.3108 -&gt; 1.3108
...
MEMORY CHANGE 1.3108 -&gt; 1.3109
...
MEMORY CHANGE 1.3109 -&gt; 1.3110
...
</code></pre></div>
<p dir="auto">How can I fix this? I currently suspect a CPU memory pool issue inside tensorflow since the problem is fairly generic, and does not depend on the ops inside the graph (much). From what I've gathered most likely candidate is the <code class="notranslate">tf.asarray</code>/copying of numpy arrays in <code class="notranslate">feed_dict</code>, leading to memory fragmentation etc. Supposing this were the case, I've heard that <code class="notranslate">tcmalloc</code> should alleviate this, but no dice (note: I've also checked that <code class="notranslate">objgraph</code> shows no growth in program over time).</p>
<p dir="auto"><a href="https://www.tensorflow.org/tutorials/keras/basic_regression#split_features_from_labels" rel="nofollow">https://www.tensorflow.org/tutorials/keras/basic_regression#split_features_from_labels</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def plot_history(history):
  plt.figure()
  plt.xlabel('Epoch')
  plt.ylabel('Mean Abs Error [MPG]')
  plt.plot(hist['epoch'], hist['mean_absolute_error'],
           label='Train Error')"><pre class="notranslate"><code class="notranslate">def plot_history(history):
  plt.figure()
  plt.xlabel('Epoch')
  plt.ylabel('Mean Abs Error [MPG]')
  plt.plot(hist['epoch'], hist['mean_absolute_error'],
           label='Train Error')
</code></pre></div>
<p dir="auto">This function references <code class="notranslate">hist</code> instead of <code class="notranslate">history</code> so it doesn't parse.</p>