title: Using IPython / Jupyter Notebooks Under Version Control
body: <p>What is a good strategy for keeping <a href="http://ipython.org" rel="noreferrer">IPython</a> notebooks under version control?</p>
  
  <p>The notebook format is quite amenable for version control: if one wants to version control the notebook and the outputs then this works quite well. The annoyance comes when one wants only to version control the input, excluding the cell outputs (aka. "build products") which can be large binary blobs, especially for movies and plots.  In particular, I am trying to find a good workflow that:</p>
  
  <ul>
  <li>allows me to choose between including or excluding output,</li>
  <li>prevents me from accidentally committing output if I do not want it,</li>
  <li>allows me to keep output in my local version,</li>
  <li>allows me to see when I have changes in the inputs using my version control system (i.e. if I only version control the inputs but my local file has outputs, then I would like to be able to see if the inputs have changed (requiring a commit).  Using the version control status command will always register a difference since the local file has outputs.)</li>
  <li>allows me to update my working notebook (which contains the output) from an updated clean notebook. <strong>(update)</strong></li>
  </ul>
  
  <p>As mentioned, if I chose to include the outputs (which is desirable when using <a href="http://nbviewer.ipython.org" rel="noreferrer">nbviewer</a> for example), then everything is fine.  The problem is when I <em>do not</em> want to version control the output.  There are some tools and scripts for stripping the output of the notebook, but frequently I encounter the following issues:</p>
  
  <ol>
  <li>I accidentally commit a version with the the output, thereby polluting my repository.</li>
  <li>I clear output to use version control, but would really rather keep the output in my local copy (sometimes it takes a while to reproduce for example).</li>
  <li>Some of the scripts that strip output change the format slightly compared to the <code>Cell/All Output/Clear</code> menu option, thereby creating unwanted noise in the diffs. This is resolved by some of the answers.</li>
  <li>When pulling changes to a clean version of the file, I need to find some way of incorporating those changes in my working notebook without having to rerun everything.
  <strong>(update)</strong></li>
  </ol>
  
  <p>I have considered several options that I shall discuss below, but have yet to find a good comprehensive solution.  A full solution might require some changes to IPython, or may rely on some simple external scripts.  I currently use <a href="https://www.mercurial-scm.org" rel="noreferrer">mercurial</a>, but would like a solution that also works with <a href="https://github.com" rel="noreferrer">git</a>: an ideal solution would be version-control agnostic.</p>
  
  <p>This issue has been discussed many times, but there is no definitive or clear solution from the user's perspective. The answer to this question should provide the definitive strategy.  It is fine if it requires a recent (even development) version of <a href="http://ipython.org" rel="noreferrer">IPython</a> or an easily installed extension.</p>
  
  <p><strong>Update:</strong> I have been playing with <a href="https://github.com/mforbes/ipython" rel="noreferrer">my modified notebook</a> version which optionally saves a <code>.clean</code> version with every save using <a href="https://stackoverflow.com/a/19761645/1115187" title="Navigate to an answer">Gregory Crosswhite's suggestions</a>.  This satisfies most of my constraints but leaves the following unresolved:</p>
  
  <ol>
  <li>This is not yet a standard solution (requires a modification of the ipython source.  Is there a way of achieving this behaviour with a simple extension?  Needs some sort of on-save hook.</li>
  <li>A problem I have with the current workflow is pulling changes.  These will come in to the <code>.clean</code> file, and then need to be integrated somehow into my working version.  (Of course, I can always re-execute the notebook, but this can be a pain, especially if some of the results depend on long calculations, parallel computations, etc.)  I do not have a good idea about how to resolve this yet.  Perhaps a workflow involving an extension like <a href="https://github.com/rossant/ipycache" rel="noreferrer">ipycache</a> might work, but that seems a little too complicated.</li>
  </ol>
  
  <h1>Notes</h1>
  
  <h2>Removing (stripping) Output</h2>
  
  <ul>
  <li>When the notebook is running, one can use the <code>Cell/All Output/Clear</code> menu option for removing the output.</li>
  <li>There are some scripts for removing output, such as the script <a href="https://gist.github.com/minrk/6176788" rel="noreferrer">nbstripout.py</a> which remove the output, but does not produce the same output as using the notebook interface.  This was eventually included in the <a href="https://github.com/ipython/nbconvert" rel="noreferrer" title="NBConvert on GitHub">ipython/nbconvert</a> repo, but this has been closed stating that the changes are now included in <a href="https://github.com/ipython/ipython" rel="noreferrer" title="IPython on GitHub">ipython/ipython</a>,but the corresponding functionality seems not to have been included yet.  <strong>(update)</strong> That being said, <a href="https://stackoverflow.com/a/19761645/1115187" title="Navigate to an answer">Gregory Crosswhite's solution</a> shows that this is pretty easy to do, even without invoking <a href="https://github.com/ipython/nbconvert" rel="noreferrer" title="NBConvert on GitHub">ipython/nbconvert</a>, so this approach is probably workable if it can be properly hooked in.  (Attaching it to each version control system, however, does not seem like a good idea — this should somehow hook in to the notebook mechanism.)</li>
  </ul>
  
  <h2>Newsgroups</h2>
  
  <ul>
  <li><a href="http://mail.scipy.org/pipermail/ipython-dev/2011-November/008351.html" rel="noreferrer">Thoughts on the notebook format for version control</a>.</li>
  </ul>
  
  <h2>Issues</h2>
  
  <ul>
  <li><a href="https://github.com/ipython/ipython/issues/977#issuecomment-5558823" rel="noreferrer">977: Notebook feature requests (Open)</a>.</li>
  <li><a href="https://github.com/ipython/ipython/issues/1280" rel="noreferrer">1280: Clear-all on save option (Open)</a>. (Follows from <a href="https://plus.google.com/103190342755104432973/posts/HsinJRLDj6o" rel="noreferrer">this discussion</a>.)</li>
  <li><a href="https://github.com/ipython/ipython/issues/3295" rel="noreferrer">3295: autoexported notebooks: only export explicitly marked cells (Closed)</a>.  Resolved by extension <a href="https://github.com/minrk/ipython_extensions/pull/11" rel="noreferrer">11 Add writeandexecute magic (Merged)</a>.</li>
  </ul>
  
  <h2>Pull Requests</h2>
  
  <ul>
  <li><a href="https://github.com/ipython/ipython/pull/1621" rel="noreferrer">1621: clear In[] prompt numbers on "Clear All Output" (Merged)</a>. (See also <a href="https://github.com/ipython/ipython/pull/2519" rel="noreferrer">2519 (Merged)</a>.)</li>
  <li><a href="https://github.com/ipython/ipython/pull/1563" rel="noreferrer">1563: clear_output improvements (Merged)</a>.</li>
  <li><a href="https://github.com/ipython/ipython/issues/3065" rel="noreferrer">3065: diff-ability of notebooks (Closed)</a>.</li>
  <li><a href="https://github.com/ipython/ipython/pull/3291" rel="noreferrer">3291: Add the option to skip output cells when saving. (Closed)</a>.  This seems extremely relevant, however was closed with the suggestion to use a "clean/smudge" filter.  A relevant question <a href="https://github.com/ipython/ipython/pull/3291#issuecomment-17712281" rel="noreferrer">what can you use if you want to strip off output before running git diff?</a> seems not to have been answered.</li>
  <li><a href="https://github.com/ipython/ipython/pull/3312" rel="noreferrer">3312: WIP: Notebook save hooks (Closed)</a>.</li>
  <li><a href="https://github.com/ipython/ipython/pull/3747" rel="noreferrer">3747: ipynb -> ipynb transformer (Closed)</a>. This is rebased in <a href="https://github.com/ipython/ipython/pull/4175" rel="noreferrer">4175</a>.</li>
  <li><a href="https://github.com/ipython/ipython/pull/4175" rel="noreferrer">4175: nbconvert: Jinjaless exporter base (Merged)</a>.</li>
  <li><a href="https://github.com/ipython/nbconvert/pull/142" rel="noreferrer">142: Use STDIN in nbstripout if no input is given (Open)</a>.</li>
  </ul>