How do we know that the reduction is correct?
Your first question is easy to answer: you are confusing "we know" with "it is the case". It can happen that something is the case but we do not know it. In particular, it is the case that $M\#x$ halts or it does not halt, but we do not know which. It will rain a year from now or not, but we do not know which.

To answer the second question, let us see what goes wrong if we try to reduce the Halting Problem to a decidable problem. Consider the problem:


  Given a single-tape Turing machine $M$ and input $x$, decide whether $M\#x$ halts in fewer than $\mathrm{length}(x)$ steps.


Now we need to come up with a reduction, i.e., given any machine $M$ and input $x$, we must produce a machine $M'$ and input $x'$, such that
$$\text{$M$ halts on $x$} \iff \text{$M'\#x'$ halts in fewer than $\mathrm{length}(x')$ steps}$$
Here is an idea: let $x'$ be the pair $(x, 0^k)$ where $0^k$ is a a string of $k$ zeroes, where we are going to determine $k$ later (this trick is know as "padding the input"). The machine $M'$ takes this input $(x, 0^k)$, ignores $0^k$ and just does whatever $M\#x$ would do. Now, for this reduction to work, we just need $k$ such that
$$\text{$M\#x$ halts} \iff \text{$M'\#(x,0^k)$ takes fewer than $\mathrm{length}(x,0^k)$ steps of execution}$$
Such a $k$ exists because $M\#x$ halts if, and only if, $M'\#(x,0^k)$ halts. So there are two cases:


If $M\#x$ halts then so does $M'\#(x,0^k)$ in some number of steps, say $m$, and we can take $k = m + 1$.
If $M\#x$ does not halt then we can take any $k$.


There is a small technical issue: the number $m$ should not depend on $k$. We can arrange this in several ways. One possibility is to allow $M'$ to have an extra symbol which it uses as "ignore input from here onwards". In any case, this is not essential for the discussion at hand.

Can you spot the problem? A reduction needs to be computable, that is, we need a computable procedure which generates $M'$ and $x'$ from $M$ and $x$. We have not given a procedure for computing $k$, even though such a $k$ exists. Perhaps if we think harder we can come up with the required procedure for computing $k$. Or perhaps it does not exist, and our attempt is not worth anything. We just do not know.

So, this is one way in which an attempt at a reduction may fail. It is in fact typical: if we are allowed to use non-computable reductions, then we can reduce any problem to any other non-trivial problem. I leave it as an exercise. (Here "non-trivial" means that the problem has an instance where the answer is "yes" and an instance where the answer is "no".)