significance test and sample size estimation for classifiers
I think there are two primary possibilities for answers, so I'll try to address both of them:
(Recommended) If you have multiple independent trials per classifier:
This is a lot simpler than if you have only one trial per classifer (and much better). If you have multiple $F_1$ scores per classifier, then you can simply conduct a paired t-test or a Wilcoxon signed rank test. You'd probably go with Wilcoxon if you don't have many trials to approximate a normal distribution via the central limit theorem or if for whatever reason your $F_1$ scores are not normally distributed.
If you only have one trial per classifier:
In this situation, we are going to find a way to express $F_1$ in terms of $F_1\pm \delta_{F_1}$ to see if the two scores overlap given some confidence level.
Step 1: Decomposition
We can first decompose the metric we are trying to calculate:
$$F_1\equiv2\frac{precision\cdot recall}{precision+recall}\equiv2\frac{PPV\cdot TPR}{PPV+TPR}$$
We are looking to find the uncertainty in the $F_1$ score, so we first need to find the uncertainties in the values composing it. To do this, we can consider the precision and recall metrics in terms of Bernoulli trials by partitioning the instances into $P$ and $N$, positives and negatives. Let's also assume $P+N\gg1$.

Edit to answer the question about determining sample sizes:
First, choose a maximum width $w$ for your confidence interval in terms of percentage points (i.e. choose a margin of error MoE and double it to account for both sides of the mean). Then, using our notion of Bernoulli trials, we can determine your minimum sample size based on the z-score for your confidence interval as well as the width (MoE):
$$n\ge \frac{4z^2(p(1-p))}{w^2} \because z\sqrt{\frac{p(1-p)}{n}}\le MoE$$
The safest estimate would be to choose $p=0.5$ which would maximize $n$ for a given $z$ and $MoE$. Realistically though, this would be an overestimate for the required $n$ because you probably don't have a random classifier.
As an example, if you wanted an MoE of 10% for a CI of 95%, you'd need:
$n\ge 97$ assuming worst case $p=0.5$.

When we consider Bernoulli trials, we can talk in terms of success/failure. So the number of successful $P$, which would be $TP$, can be considered a percentage of the total number of $P$ instances, which is simply the true positive rate $TPR$. The same goes for $TN$ and $N$, being $TNR$. In other words:
$$TPR=\frac{TP}{P}=\frac{TP}{TP+FN}\enspace \enspace, \enspace\enspace TNR=\frac{TN}{N}=\frac{TN}{TN+FP}$$
Step 2: Binomial Proportion Confidence Intervals
Now, we can develop binomial proportion confidence intervals per $P$ and $N$. There are many formulas to do this, but I am going to stick with probably the most basic one. That way, we make sure that our result assumes the form $TPR\pm \delta_{TPR}$.
Now let's pick a confidence level:

If you want a $99\%$, then $z_{0.01/2}\approx2.576$
If you want $95\%$, then $z_{0.05/2}\approx 1.96$ confidence level

We can calculate the individual confidence intervals:
$$TPR\pm z\sqrt{\frac{TPR(1-TPR)}{P}}\enspace \enspace, \enspace\enspace TNR\pm z\sqrt{\frac{TNR(1-TNR)}{N}}$$
To get $PPV$, we can first define $TP=P(TPR)$ and $FP=N(1-TNR)$. Then:
$$ PPV=\frac{TP}{TP+FP}=\frac{P(TPR)}{P(TPR)+N(1-TNR)} $$
Step 3: Propagate PPV Uncertainties
We want to find $PPV\pm \delta_{PPV}$, so we need to propagate the uncertainties from $TPR$ and $TNR$. Because $P$ and $N$ are determined by the dataset rather than your model (and they are exact &quot;measurements&quot;), they are without uncertainties:
$$ PPV\pm \frac{P(TPR\pm \delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}=\frac{X}{Y} $$
Let's first find the individual components $X$ and $Y$, keeping in mind that when uncertain values add, their uncertainties add as well:
$$ X=P(TPR)\pm P(\delta_{TPR}) $$
$$ Y=(N+P(TPR)+N(TNR))\pm (P(\delta_{TPR})+N(\delta_{TNR})) $$
$$ = \frac{P(TPR)\pm P(\delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}$$
Now we can divide $X$ by $Y$. To do so, we need to keep in mind that the resulting uncertainty is the sum of their fractional uncertainties, like so:
$$ PPV=\frac{X}{Y}\enspace \enspace , \enspace \enspace \frac{\delta_{PPV}}{|PPV|}=\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\Rightarrow \delta{PPV}=\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right) $$
For simplicity, I'll write $PPV$ because we know that $$PPV\pm \delta_{PPV}\equiv \frac{X}{Y}\pm\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right)$$
(Finally) Step 4: Propagate TPR and PPV Uncertainties
Just as we did with $PPV$, we need to propagate the uncertainties for the $F_1$ score calculation. Fortunately, it's the same process as the uncertainties for multiplication combine in the same manner as division:
$$ F_1 = 2\frac{PPV\cdot TPR}{PPV+TPR} = 2\frac{A}{B}$$
$$ A=(PPV\cdot TPR)\enspace \enspace, \enspace \enspace B=(PPV+TPR) $$
$$ \frac{\delta_A}{|A|}=\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\Rightarrow \delta_A=|A|\left(\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\right) $$
$$ \delta_B = \delta_{PPV} + \delta_{TPR} $$
Then we also have to combine the uncertainties for the division of $A$ by $B$:
$$ \delta_{\frac{A}{B}} = \left|\frac{A}{B}\right|\left(\frac{\delta_A}{|A|}+\frac{\delta_B}{|B|}\right) $$
This yields our final answer because we know that:
$$ F_1 \pm \delta_{F_1} \equiv 2\left(\frac{A}{B} \pm \delta_\frac{A}{B}\right) $$
Now you can see whether your $F_1$ scores overlap given some confidence interval! You could also probably see how much they overlap in terms of a ratio, but it'd probably be more understandable if you just stuck with a binary yes/no overlap.