Correlated signals separation with reference
I followed the link from you more current version.  I think your problem is better stated here.

1) I don't think you can solve it as stated.  You will have a one unknown parameter family of solutions.

2) If you can use a portion of Y, and the coefficients remain the same, you can easily solve it.

$$ \vec S = a \vec X + b \vec Y $$

Dot this equation with $ \vec X $ and $ \vec Y $:

$$ ( \vec S \cdot \vec X ) = a ( \vec X \cdot \vec X ) + b ( \vec Y \cdot \vec X ) $$
$$ ( \vec S \cdot \vec Y ) = a ( \vec X \cdot \vec Y ) + b ( \vec Y \cdot \vec Y ) $$

This is a solvable system of two equations, two unknowns.

3) Rescaling will not affect the values of $a$ and $b$.

Hope this helps.

Ced



Followup

If you expect that $a$ and $b$ may be changing, then I think this problem is basically intractable.  At least this approach would be.  

These vector equations will apply to any subset of your signal.  You will get the best values for $a$ in sections where you guess that $\vec Y$ is small, and the best values for $b \vec Y$ where $\vec X$ is small (as you already stated).  Also, if you can determine sections where $ \vec X \cdot \vec Y = 0 $ (completely uncorrelated), you will also get good readings on $a$ and $b \vec Y$.  In sections where they are highly correlated, you will get unreliable values for $a$ and $b \vec Y$.

If you have a rough idea of what $a$ is, you can find uncorrelated sections by looking for intervals where $ ( \vec S - a \vec X ) \cdot \vec X \approx 0 $, but this won't get you a better value for $a$ because that's what you started with.

There is no way to separate $b$ from $ b \vec Y $ unless you know something about $ \vec Y $.

Any kind of linear transform is going to look like this:

$$ T( \vec S ) = a T( \vec X ) + b T( \vec Y ) $$

I'm not sure if that could be helpful, I doubt it.