I have a huge training CSV file (709M) and a large testing CSV file (125M) that I want to send into a `DNNClassifier` in the context of using the high-level Tensorflow API.

It appears that the `input_fn` param accepted by `fit` and `evaluate` must hold all feature and label data in memory, but I currently would like to run this on my local machine, and thus expect it to run out of memory rather quickly if I read these files into memory and then process them.

I skimmed the doc on [streamed-reading of data](https://www.tensorflow.org/programmers_guide/reading_data), but the sample code for reading CSVs appears to be for the low-level Tensorflow API. 

And - if you'll forgive a bit of whining - it seems overly-complex for the trivial use case of sending well-prepared files of training and test data into an `Estimator` ... although, perhaps that level of complexity is actually required for training and testing large volumes of data in Tensorflow?

In any case, I'd really appreciate an example of using that approach with the high-level API, if it's even possible, which I'm beginning to doubt.

After poking around, I did manage to find [`DNNClassifier#partial_fit`](https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/learn/DNNClassifier#partial_fit), and will attempt to use it for training. 

Examples of how to use this method would save me some time, though hopefully I'll stumble into the correct usage in the next few hours.

However, there doesn't seem to be a corresponding `DNNClassifier#partial_evaluate` ... though I suspect that I could break-up the testing data into smaller pieces and run `DNNClassifier#evaluate` successively on each batch, which might actually be a great way to do it since I could segment the testing data into cohorts, and thereby obtain per-cohort accuracy.

==== Update ====

Short version: 

1. DomJack's recommendation should be the accepted answer.
2. However, my Mac's 16GB of RAM enough for it to hold the entire 709Mb training data set in memory without crashing. So, while I will use the DataSets feature when I eventually deploy the app, I'm not using it yet for local dev work.

Longer version:

I started by using the `partial_fit` API as described above, but upon every use it emitted a warning. 

So, I went to look at the source for the method [here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/estimator.py), and discovered that its complete implementation looks like this:

```
logging.warning('The current implementation of partial_fit is not optimized'
                ' for use in a loop. Consider using fit() instead.')
return self.fit(x=x, y=y, input_fn=input_fn, steps=steps,
                batch_size=batch_size, monitors=monitors)

```

... which reminds me of this scene from Hitchhiker's Guide: 

Arthur Dent: What happens if I press this button?

Ford Prefect: I wouldn't-

Arthur Dent: Oh.

Ford Prefect: What happened?

Arthur Dent: A sign lit up, saying 'Please do not press this button again'.

Which is to say: `partial_fit` seems to exist for the sole purpose of telling you not to use it.

Furthermore, the model generated by using `partial_fit` iteratively on training file chunks was much smaller than the one generated by using `fit` on the whole training file, which strongly suggests that only the last `partial_fit` training chunk actually "took".