Apply a filter according to some function
Say I have a function which approximates the data I'm trying to filter
  from my signal. Is there a technique -- perhaps some sort of
  optimisation --


Let us say that you have a model of your data, in the form of a polynomial fit of certain order. In this case you will have co-efficients that produce a polynomial that best fits your data in the Least Squares Sense. Thus, you would have already solved the LSE problem via optimization routines, which would yield the co-efficients $p_i$. for a $m$ degree polynomial.

$$
y[n] = p_0x^0[n] + p_1x[n] + p_2x^2[n] ... p_mx^m[n]
$$


  that can take this estimation as a "template" for a filter design,
  rather than specifying strict filter parameters, as when one designs a
  Butterworth filter, for example.


Technically yes, you can do this. Once you have your polynomial co-efficients, you can generate a polynomial at sample spacing commensurate with your sampling rate, and perform a DFT on it and inspect the absolute magnitude of your result, in order to get a frequency domain representation of your 'template'. This is certainly do-able. 

In fact, this is precisely the theory behind matched filters. Matched filters work optimally (in AWGN) because they take into account frequency bins where your signal resides, and ignore other bins with noise only. This is how they maximize SNR when performing active detection of (obviously known) signals. (This is also closely related to the technique of Empirical filtering in the frequency domain). 

I say the above is doable, however the wisdom of doing it would depend heavily on your application, which your question does not really delve into. 

EDIT: Based on the new information you have provided in your comments: 

First off, note that the Hilbert Spectrum (instantaneous frequency) is not defined for signals with more than one complex exponential. 

Secondly, the way you have described it, you may be over-thinking this. You want to perform a narrow-band filtering operation on a frequency-varying signal over time. One basic solution for this is to simply use an Adaptive Filter, whereby the weights of your (adaptive) filter are constantly updated so as to minimize the error between your estimate and the empirical data.

One implementation of an LSE Adaptive filter will seek to minimize the cost function, defined by:

$$
C[n] = \Big[y[n] -  \mathbf{h}^T[n]\mathbf{x}[n]\Big]^2
$$

Where $\mathbf{h}^T[n]$ is the $M$ length adaptive weight vector at instant $n$, $\mathbf{x}[n]$ is the $M$ length data vector taken as column vectors of the Toeplitz data matrix $\mathbf{X}$, and $y[n]$ is the desired signal at instant $n$. A filter structure such as this will 'track' your signal as it changes frequencies.