<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [2]: df = pd.DataFrame([[1,2,5,6], [3,4,7,8]], index=['a', 'a'], columns=pd.MultiIndex.from_product([[1,2], ['A', 'B']]))

In [3]: df.loc['c'] = -1

In [4]: df.loc['c', (1, 'A')] = 3

In [5]: df.loc['d', (1, 'A')] = 3
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-5-34c3deb1645d&gt; in &lt;module&gt;()
----&gt; 1 df.loc['d', (1, 'A')] = 3

/home/pietro/nobackup/repo/pandas/pandas/core/indexing.py in __setitem__(self, key, value)
    176             key = com._apply_if_callable(key, self.obj)
    177         indexer = self._get_setitem_indexer(key)
--&gt; 178         self._setitem_with_indexer(indexer, value)
    179 
    180     def _has_valid_type(self, k, axis):

/home/pietro/nobackup/repo/pandas/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
    348                     index = self.obj._get_axis(i)
    349                     labels = index.insert(len(index), key)
--&gt; 350                     self.obj._data = self.obj.reindex_axis(labels, i)._data
    351                     self.obj._maybe_update_cacher(clear=True)
    352                     self.obj.is_copy = None

/home/pietro/nobackup/repo/pandas/pandas/core/frame.py in reindex_axis(self, labels, axis, method, level, copy, limit, fill_value)
   2844                      self).reindex_axis(labels=labels, axis=axis,
   2845                                         method=method, level=level, copy=copy,
-&gt; 2846                                         limit=limit, fill_value=fill_value)
   2847 
   2848     @Appender(_shared_docs['rename'] % _shared_doc_kwargs)

/home/pietro/nobackup/repo/pandas/pandas/core/generic.py in reindex_axis(self, labels, axis, method, level, copy, limit, fill_value)
   2492                                                  limit=limit)
   2493         return self._reindex_with_indexers({axis: [new_index, indexer]},
-&gt; 2494                                            fill_value=fill_value, copy=copy)
   2495 
   2496     def _reindex_with_indexers(self, reindexers, fill_value=np.nan, copy=False,

/home/pietro/nobackup/repo/pandas/pandas/core/generic.py in _reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups)
   2515                                                 fill_value=fill_value,
   2516                                                 allow_dups=allow_dups,
-&gt; 2517                                                 copy=copy)
   2518 
   2519         if copy and new_data is self._data:

/home/pietro/nobackup/repo/pandas/pandas/core/internals.py in reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy)
   3879         # some axes don't allow reindexing with dups
   3880         if not allow_dups:
-&gt; 3881             self.axes[axis]._can_reindex(indexer)
   3882 
   3883         if axis &gt;= self.ndim:

/home/pietro/nobackup/repo/pandas/pandas/indexes/base.py in _can_reindex(self, indexer)
   2733         # trying to reindex on an axis with duplicates
   2734         if not self.is_unique and len(indexer):
-&gt; 2735             raise ValueError(&quot;cannot reindex from a duplicate axis&quot;)
   2736 
   2737     def reindex(self, target, method=None, level=None, limit=None,

ValueError: cannot reindex from a duplicate axis

"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>([[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">5</span>,<span class="pl-c1">6</span>], [<span class="pl-c1">3</span>,<span class="pl-c1">4</span>,<span class="pl-c1">7</span>,<span class="pl-c1">8</span>]], <span class="pl-s1">index</span><span class="pl-c1">=</span>[<span class="pl-s">'a'</span>, <span class="pl-s">'a'</span>], <span class="pl-s1">columns</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-v">MultiIndex</span>.<span class="pl-en">from_product</span>([[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>], [<span class="pl-s">'A'</span>, <span class="pl-s">'B'</span>]]))

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s">'c'</span>] <span class="pl-c1">=</span> <span class="pl-c1">-</span><span class="pl-c1">1</span>

<span class="pl-v">In</span> [<span class="pl-c1">4</span>]: <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s">'c'</span>, (<span class="pl-c1">1</span>, <span class="pl-s">'A'</span>)] <span class="pl-c1">=</span> <span class="pl-c1">3</span>

<span class="pl-v">In</span> [<span class="pl-c1">5</span>]: <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s">'d'</span>, (<span class="pl-c1">1</span>, <span class="pl-s">'A'</span>)] <span class="pl-c1">=</span> <span class="pl-c1">3</span>
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span>
<span class="pl-v">ValueError</span>                                <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">5</span><span class="pl-c1">-</span><span class="pl-c1">34</span><span class="pl-s1">c3deb1645d</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>()
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">1</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s">'d'</span>, (<span class="pl-c1">1</span>, <span class="pl-s">'A'</span>)] <span class="pl-c1">=</span> <span class="pl-c1">3</span>

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">indexing</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">__setitem__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">key</span>, <span class="pl-s1">value</span>)
    <span class="pl-c1">176</span>             <span class="pl-s1">key</span> <span class="pl-c1">=</span> <span class="pl-s1">com</span>.<span class="pl-en">_apply_if_callable</span>(<span class="pl-s1">key</span>, <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>)
    <span class="pl-c1">177</span>         <span class="pl-s1">indexer</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_get_setitem_indexer</span>(<span class="pl-s1">key</span>)
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">178</span>         <span class="pl-s1">self</span>.<span class="pl-en">_setitem_with_indexer</span>(<span class="pl-s1">indexer</span>, <span class="pl-s1">value</span>)
    <span class="pl-c1">179</span> 
    <span class="pl-c1">180</span>     <span class="pl-k">def</span> <span class="pl-en">_has_valid_type</span>(<span class="pl-s1">self</span>, <span class="pl-s1">k</span>, <span class="pl-s1">axis</span>):

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">indexing</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_setitem_with_indexer</span>(<span class="pl-s1">self</span>, <span class="pl-s1">indexer</span>, <span class="pl-s1">value</span>)
    <span class="pl-c1">348</span>                     <span class="pl-s1">index</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>.<span class="pl-en">_get_axis</span>(<span class="pl-s1">i</span>)
    <span class="pl-c1">349</span>                     <span class="pl-s1">labels</span> <span class="pl-c1">=</span> <span class="pl-s1">index</span>.<span class="pl-en">insert</span>(<span class="pl-en">len</span>(<span class="pl-s1">index</span>), <span class="pl-s1">key</span>)
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">350</span>                     <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>.<span class="pl-s1">_data</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>.<span class="pl-en">reindex_axis</span>(<span class="pl-s1">labels</span>, <span class="pl-s1">i</span>).<span class="pl-s1">_data</span>
    <span class="pl-c1">351</span>                     <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>.<span class="pl-en">_maybe_update_cacher</span>(<span class="pl-s1">clear</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-c1">352</span>                     <span class="pl-s1">self</span>.<span class="pl-s1">obj</span>.<span class="pl-s1">is_copy</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">frame</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">reindex_axis</span>(<span class="pl-s1">self</span>, <span class="pl-s1">labels</span>, <span class="pl-s1">axis</span>, <span class="pl-s1">method</span>, <span class="pl-s1">level</span>, <span class="pl-s1">copy</span>, <span class="pl-s1">limit</span>, <span class="pl-s1">fill_value</span>)
   <span class="pl-c1">2844</span>                      <span class="pl-s1">self</span>).<span class="pl-en">reindex_axis</span>(<span class="pl-s1">labels</span><span class="pl-c1">=</span><span class="pl-s1">labels</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-s1">axis</span>,
   <span class="pl-c1">2845</span>                                         <span class="pl-s1">method</span><span class="pl-c1">=</span><span class="pl-s1">method</span>, <span class="pl-s1">level</span><span class="pl-c1">=</span><span class="pl-s1">level</span>, <span class="pl-s1">copy</span><span class="pl-c1">=</span><span class="pl-s1">copy</span>,
<span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">2846</span>                                         <span class="pl-s1">limit</span><span class="pl-c1">=</span><span class="pl-s1">limit</span>, <span class="pl-s1">fill_value</span><span class="pl-c1">=</span><span class="pl-s1">fill_value</span>)
   <span class="pl-c1">2847</span> 
   <span class="pl-c1">2848</span>     @<span class="pl-v">Appender</span>(<span class="pl-s1">_shared_docs</span>[<span class="pl-s">'rename'</span>] <span class="pl-c1">%</span> <span class="pl-s1">_shared_doc_kwargs</span>)

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">generic</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">reindex_axis</span>(<span class="pl-s1">self</span>, <span class="pl-s1">labels</span>, <span class="pl-s1">axis</span>, <span class="pl-s1">method</span>, <span class="pl-s1">level</span>, <span class="pl-s1">copy</span>, <span class="pl-s1">limit</span>, <span class="pl-s1">fill_value</span>)
   <span class="pl-c1">2492</span>                                                  <span class="pl-s1">limit</span><span class="pl-c1">=</span><span class="pl-s1">limit</span>)
   <span class="pl-c1">2493</span>         <span class="pl-s1">return</span> <span class="pl-s1">self</span>.<span class="pl-en">_reindex_with_indexers</span>({<span class="pl-s1">axis</span>: [<span class="pl-s1">new_index</span>, <span class="pl-s1">indexer</span>]},
<span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">2494</span>                                            <span class="pl-s1">fill_value</span><span class="pl-c1">=</span><span class="pl-s1">fill_value</span>, <span class="pl-s1">copy</span><span class="pl-c1">=</span><span class="pl-s1">copy</span>)
   <span class="pl-c1">2495</span> 
   <span class="pl-c1">2496</span>     <span class="pl-k">def</span> <span class="pl-s1">_reindex_with_indexers</span>(<span class="pl-s1">self</span>, <span class="pl-s1">reindexers</span>, <span class="pl-s1">fill_value</span><span class="pl-c1">=</span><span class="pl-s1">np</span>.<span class="pl-s1">nan</span>, <span class="pl-s1">copy</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">generic</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-s1">_reindex_with_indexers</span>(<span class="pl-s1">self</span>, <span class="pl-s1">reindexers</span>, <span class="pl-s1">fill_value</span>, <span class="pl-s1">copy</span>, <span class="pl-s1">allow_dups</span>)
   <span class="pl-c1">2515</span>                                                 <span class="pl-s1">fill_value</span><span class="pl-c1">=</span><span class="pl-s1">fill_value</span>,
   <span class="pl-c1">2516</span>                                                 <span class="pl-s1">allow_dups</span><span class="pl-c1">=</span><span class="pl-s1">allow_dups</span>,
<span class="pl-c1">-&gt;</span> <span class="pl-c1">2517</span>                                                 <span class="pl-s1">copy</span><span class="pl-c1">=</span><span class="pl-s1">copy</span>)
   <span class="pl-c1">2518</span> 
   <span class="pl-c1">2519</span>         <span class="pl-k">if</span> <span class="pl-s1">copy</span> <span class="pl-c1">and</span> <span class="pl-s1">new_data</span> <span class="pl-c1">is</span> <span class="pl-s1">self</span>.<span class="pl-s1">_data</span>:

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">core</span><span class="pl-c1">/</span><span class="pl-s1">internals</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">reindex_indexer</span>(<span class="pl-s1">self</span>, <span class="pl-s1">new_axis</span>, <span class="pl-s1">indexer</span>, <span class="pl-s1">axis</span>, <span class="pl-s1">fill_value</span>, <span class="pl-s1">allow_dups</span>, <span class="pl-s1">copy</span>)
   <span class="pl-c1">3879</span>         <span class="pl-c"># some axes don't allow reindexing with dups</span>
   <span class="pl-c1">3880</span>         <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">allow_dups</span>:
<span class="pl-c1">-&gt;</span> <span class="pl-c1">3881</span>             <span class="pl-s1">self</span>.<span class="pl-s1">axes</span>[<span class="pl-s1">axis</span>].<span class="pl-en">_can_reindex</span>(<span class="pl-s1">indexer</span>)
   <span class="pl-c1">3882</span> 
   <span class="pl-c1">3883</span>         <span class="pl-k">if</span> <span class="pl-s1">axis</span> <span class="pl-c1">&gt;=</span> <span class="pl-s1">self</span>.<span class="pl-s1">ndim</span>:

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pietro</span><span class="pl-c1">/</span><span class="pl-s1">nobackup</span><span class="pl-c1">/</span><span class="pl-s1">repo</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">pandas</span><span class="pl-c1">/</span><span class="pl-s1">indexes</span><span class="pl-c1">/</span><span class="pl-s1">base</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_can_reindex</span>(<span class="pl-s1">self</span>, <span class="pl-s1">indexer</span>)
   <span class="pl-c1">2733</span>         <span class="pl-c"># trying to reindex on an axis with duplicates</span>
   <span class="pl-c1">2734</span>         <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">self</span>.<span class="pl-s1">is_unique</span> <span class="pl-c1">and</span> <span class="pl-en">len</span>(<span class="pl-s1">indexer</span>):
<span class="pl-c1">-&gt;</span> <span class="pl-c1">2735</span>             <span class="pl-k">raise</span> <span class="pl-v">ValueError</span>(<span class="pl-s">"cannot reindex from a duplicate axis"</span>)
   <span class="pl-c1">2736</span> 
   <span class="pl-c1">2737</span>     <span class="pl-k">def</span> <span class="pl-s1">reindex</span>(<span class="pl-s1">self</span>, <span class="pl-s1">target</span>, <span class="pl-s1">method</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">level</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">limit</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,

<span class="pl-v">ValueError</span>: <span class="pl-s1">cannot</span> <span class="pl-s1">reindex</span> <span class="pl-k">from</span> <span class="pl-s1">a</span> <span class="pl-s1">duplicate</span> <span class="pl-s1">axis</span>
</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I think the reindexing code path shouldn't be taken at all (in this case - I'm not referring to the general design choice): anyway, the operation should work, coherently with the case where the index is unique and with the case where the row exists (<code class="notranslate">In [4]</code>).</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Like <code class="notranslate">In [4]</code>.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.5.3.final.0<br>
python-bits: 64<br>
OS: Linux<br>
OS-release: 4.7.0-1-amd64<br>
machine: x86_64<br>
processor:<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: it_IT.utf8<br>
LOCALE: it_IT.UTF-8</p>
<p dir="auto">pandas: 0.19.0+783.gcd35d22a0<br>
pytest: 3.0.6<br>
pip: 9.0.1<br>
setuptools: 33.1.1<br>
Cython: 0.25.2<br>
numpy: 1.12.0<br>
scipy: 0.18.1<br>
xarray: 0.9.1<br>
IPython: 5.1.0.dev<br>
sphinx: 1.4.9<br>
patsy: 0.3.0-dev<br>
dateutil: 2.5.3<br>
pytz: 2016.7<br>
blosc: None<br>
bottleneck: 1.2.0<br>
tables: 3.3.0<br>
numexpr: 2.6.1<br>
feather: 0.3.1<br>
matplotlib: 2.0.0<br>
openpyxl: 2.3.0<br>
xlrd: 1.0.0<br>
xlwt: 1.1.2<br>
xlsxwriter: 0.9.6<br>
lxml: 3.7.1<br>
bs4: 4.5.3<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.0.15<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.8<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: 0.2.1</p>
</details>
<p dir="auto">I'm not sure if this functionality exists but it is boiler plate which I need to use quite often.</p>
<p dir="auto">Typically I need to work with very large datasets, e.g. 10mil rows or more.<br>
On these datasets performing a calculations such as taking a date in a specific string format can be sped up incredibly by assessing what needs to be done first.</p>
<p dir="auto">As such I find myself writing the following boiler plate code for a lot of functions.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def map_strptime(series, format=None):
    mapping = {x: datetime.strptime(x, format) for x in set(series)}
    return series.map(mapping)"><pre class="notranslate"><code class="notranslate">def map_strptime(series, format=None):
    mapping = {x: datetime.strptime(x, format) for x in set(series)}
    return series.map(mapping)
</code></pre></div>
<p dir="auto">Now comparing this implementation v.s a more naive implementation just using the standard apply function, e.g.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def nonmap_strptime(x, format=None):
    return datetime.strptime(x, format)"><pre class="notranslate"><code class="notranslate">def nonmap_strptime(x, format=None):
    return datetime.strptime(x, format)
</code></pre></div>
<p dir="auto">Usage:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df[&quot;Date&quot;] = map_strptime(df[&quot;Trading Date&quot;], format=&quot;%Y-%m-%d&quot;)

# vs.

df[&quot;Date&quot;] = df[&quot;Trading Date&quot;].apply(nonmap_strptime, format=&quot;%Y-%m-%d&quot;)"><pre class="notranslate"><code class="notranslate">df["Date"] = map_strptime(df["Trading Date"], format="%Y-%m-%d")

# vs.

df["Date"] = df["Trading Date"].apply(nonmap_strptime, format="%Y-%m-%d")
</code></pre></div>
<p dir="auto">Assessing these two implementations for performance I get the following.<br>
Note, my Dataset has 10 million rows across 365 days, e.g. a lot of repeated calculations.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Mapped version
&gt;&gt;&gt;&gt; %timeit  df[&quot;Date&quot;] = map_strptime(df[&quot;Trading Date&quot;], format=&quot;%Y-%m-%d&quot;)
&gt;&gt;&gt;&gt; 900 ms

# Nonmapped version
&gt;&gt;&gt;&gt; %timeit df[&quot;Date&quot;] = df[&quot;Trading Date&quot;].apply(nonmap_strptime, format=&quot;%Y-%m-%d&quot;)
&gt;&gt;&gt;&gt; (Still hasn't finished in the time it has taken me to write this issue, 10min plus)"><pre class="notranslate"><code class="notranslate"># Mapped version
&gt;&gt;&gt;&gt; %timeit  df["Date"] = map_strptime(df["Trading Date"], format="%Y-%m-%d")
&gt;&gt;&gt;&gt; 900 ms

# Nonmapped version
&gt;&gt;&gt;&gt; %timeit df["Date"] = df["Trading Date"].apply(nonmap_strptime, format="%Y-%m-%d")
&gt;&gt;&gt;&gt; (Still hasn't finished in the time it has taken me to write this issue, 10min plus)
</code></pre></div>
<p dir="auto">As you can see for some use cases, e.g. when a calculation is being repeated often a simple set - map type wrapper around the operation could improve the performance substantially.</p>
<p dir="auto">However, if a large number of non-duplicates exists then this would obviously slow the operation down quite substantially as first the calculation would be performed, then the mapping in two steps rather than one.</p>
<p dir="auto">To get around this the number of unique items and non-unique items could be computed. If a large difference between them exists then a shift to the mapping implementation could impart large performance gains.</p>
<p dir="auto">I'm not sure as to the best implementation, but something along the lines of:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df.memoapply(*args, **kargs):

if len(set(df[cols]) &gt;&gt; len(df[cols]):
    # use memo version

else:
    # Use standard versions."><pre class="notranslate"><code class="notranslate">df.memoapply(*args, **kargs):

if len(set(df[cols]) &gt;&gt; len(df[cols]):
    # use memo version

else:
    # Use standard versions.
</code></pre></div>
<p dir="auto">This works best for one to one type operations, I'm unsure how well it would work for one to many or for many to one type applications.</p>
<p dir="auto">Could possible be extended by using the inbuilt merge operations of Pandas and using a multi indexed series for the result?</p>