<h4 dir="auto">Description</h4>

<p dir="auto">I've been unable to perform a grid-search on a pipeline that changes sample numbers. The pipeline itself works fine, since I use a FunctionTransformer() that feeds into the classifier with the fewer number of samples.</p>
<p dir="auto">The idea here is that I need to train a topic model embedding of a large corpus, and since this is unsupervised, I use the entire corpus I have to learn these topics (or alternatively use w2v). Then, only a subset of these are <em>labeled</em>, so I extract the labeled docs and train a  LinearSVC to predict some label based on the topic vectors. I cobbled this pipe together using Textacy (based on SpaCy) and SKLearn, but it does work, taking a corpus object as input and getting a classification as output.</p>
<p dir="auto">I would like to grid search the topic number <em>using the cross-validation score of the classifier</em>. This seems perfectly reasonable to me, since the parameters I'm picking are in the large-sample space and the CV score I get is in the small-sample space, but neither affect the ordering of the other, in principal.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>

<p dir="auto">Here's <a href="https://gist.github.com/tbsexton/88c5a6a3cebd98311244c065378892e0">a notebook gist</a>, but note that there are a few extra classes I've defined externally to process the text corpus and return topic vectors. Still, the pipe works just <em>not in the grid-search</em>.</p>
<h4 dir="auto">Expected Results</h4>

<p dir="auto">CV score of a classifier fed topic vectors (trained from more samples than the classifier has labels for) for varying numbers of topics/model types.</p>
<h4 dir="auto">Actual Results</h4>

<p dir="auto">See gist.</p>
<h4 dir="auto">Versions</h4>

<p dir="auto">Windows-7-6.1.7601-SP1<br>
('Python', '2.7.13 |Anaconda custom (64-bit)| (default, Dec 19 2016, 13:29:36) [MSC v.1500 64 bit (AMD64)]')<br>
('NumPy', '1.11.3')<br>
('SciPy', '0.18.1')<br>
('Scikit-Learn', '0.18.1')</p>
<p dir="auto">Some data transformations -- including over/under-sampling (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="9116662" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/1454" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/1454/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/1454">#1454</a>), outlier removal, instance reduction, and other forms of dataset compression, like that used in BIRCH (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46758226" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3802" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3802/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3802">#3802</a>) -- entail altering a dataset at training time, but leaving it unaltered at prediction time. (In some cases, such as outlier removal, it makes sense to reapply a fitted model to new data, while in others model reuse after fitting seems less applicable. )</p>
<p dir="auto">As noted <a href="https://github.com/scikit-learn/scikit-learn/pull/1454#issuecomment-11313359" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/1454/hovercard">elsewhere</a>, transformers that change the number of samples are not currently supported, certainly in the context of <code class="notranslate">Pipeline</code>s where a transformation is applied both at <code class="notranslate">fit</code> and <code class="notranslate">predict</code> time (although a hack might abuse <code class="notranslate">fit_transform</code> to make this not so). <code class="notranslate">Pipeline</code>s of <code class="notranslate">Transformer</code>s also would not cope with changes in the sample size at fit time for supervised problems because <code class="notranslate">Transformer</code>s do not return a modified <code class="notranslate">y</code>, only <code class="notranslate">X</code>.</p>
<p dir="auto">To handle this class of problems, I propose introducing a new category of estimator, called a <code class="notranslate">Resampler</code>. It must define at least a <code class="notranslate">fit_resample</code> method, which <code class="notranslate">Pipeline</code> will call at <code class="notranslate">fit</code> time, passing the data unchanged at other times. (For this reason, a <code class="notranslate">Resampler</code> cannot also be a <code class="notranslate">Transformer</code>, or else we need to define their precedence.)</p>
<p dir="auto">For many models, <code class="notranslate">fit_resample</code> needs only return <code class="notranslate">sample_weight</code>. For sample compression approaches (e.g. that in BIRCH), this is not sufficient as the representative centroids are modified from the input samples. Hence I think <code class="notranslate">fit_resample</code> should return altered data directly, in the form of a dict with keys <code class="notranslate">X</code>, <code class="notranslate">y</code>, <code class="notranslate">sample_weight</code> as required. (It still might be appropriate for many <code class="notranslate">Resampler</code>s to only modify <code class="notranslate">sample_weight</code>; if necessary, another <code class="notranslate">Resampler</code> can be chained that realises the weights as replicated or deleted entries in <code class="notranslate">X</code> and <code class="notranslate">y</code>.)</p>