limiter algorithm wanted: mixing two channels without clipping
You can keep the mixing itself linear but add a mapping to the mixing variable $x$ so that you can control its dynamics. The closest this is to reality is the use of logarithmic potentiometers.

So, given $C_1, C_2$ being your mixing channels and $x$ being your mixing variable, the mixed track $M$ at some time instance $n$ is:

$$ M[n] = C_1[n] \cdot x + C_2[n] \cdot (1-x)$$

At $x=0.5$, both tracks are contributing equally, provided that they are mixed to similar levels. If the tracks are not mixed to similar levels, then this inserts a bias on $x$ on either side.

Anyway, so now another "problem" arises which is how to control $x$ and even how to limit it.

To achieve this, we setup a mapping via a function of the form $y = f(x)$. Ideally, $x,y \in \left[0 \ldots 1 \right]$ So now our mixing process looks like this:

$$ M[n] = C_1[n] \cdot f(x) + C_2[n] \cdot (1-f(x))$$

(The right part could also have been $f(1-x)$)

If you want $f$ to limit your values in a subset of $x$, then you could use a sigmoid function. The sigmoid function has a linear part (where the mapping between $y,x$ is almost identical) and two "non-linear" parts called the shoulder (at the top) and the knee (at the bottom). These are the points where the function starts to plateau and no matter what the value of $x$ is, the value of $y$ is pinned down to a specific limit.

Now, obviously, for some sigmoid function $f$, its output $y \in \left[-1 \ldots 1\right]$. But that is not much of a problem because you can simply $y = \frac{f(x)+1}{2}$ and now your $y$ goes from $0$ to $1$. If you don't want it to hit $1$, then you can now multiply the output of your $f$ with another scaling factor. So $y = q \cdot \frac{f(x)+1}{2}$, where $q$ could be $0.8$ in your case.

Another thing you can do with this scheme is control the "dynamics" of the mixing. So, the user might be moving the slider linearly but their effect taking place non-linearly or with some other rate. This could account for the fact that perception is not linear (for example).

For this purpose, you can set up a logarithmic function $f$ that goes from 0 to 1. The logarithmic also has a "limiting" region towards its maximum value.

The general idea is to keep the mixing linear but not necessarily the way that the mixing variable varies.

The other thing you might want to consider if your tracks are "difficult" might be the use of a compressor before the mixing, so that the levels of both audio channels are kept within reasonable limits before they get mixed down. But, inserting a compressor introduces more controls (more variables) that someone would have to fiddle with depending on the tracks. Maybe you don't want this (?)

Hope this helps.