This post originates from this discussion: [Two one-sided hypothesis tests instead of a two-sided test?](https://stats.stackexchange.com/questions/452296/two-one-sided-hypothesis-tests-instead-of-a-two-sided-test/452461#452461) Say we repeatedly draw a sample of size $n$ and conduct a two tailed t-test against a hypothesized mean $\mu\_0$ at a certain significance level $\alpha$. Let us further assume the samples are from one (normal) distribution (it may or may not be the one we hypothesize) that does not change over time and is unknown to us. After a few runs, we may have some hits in both rejection regions of our hypothesized distribution and may reject the null. My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of $H\_0$ and thus the relevance of $\alpha$?. After all, an actual error only occurs with a chance of $$P(reject\:H\_0\cap H\_0\:true)=P(H\_0\:true)\cdot P\_{H\_0\:true}(reject\:H\_0)=P(H\_0\:true)\cdot \alpha$$ Doesn't it just become unreasonable to believe in the possibility of $H\_0\:true$ and the realistic chance of a type I error if results repeatedly suggest otherwise?
Moreover, would it be reasonable to reject (or accept) $H\_0$ after one test (as we often see being done in practice)? I mean, since frequentist probabilities refer to an "infinite" number of runs, isn't one result rather meaningless? On the other hand, if we conduct many runs, hypothesis testing also seems to become quite pointless as I pointed out before...