How to properly seal a leaky pump coupling on a hot tub/spa?
Your coupling or pipe might have a crack, too difficult for you to see or notice. 

You are correct they are supposed to be hand tightened and that should be enough .

The connection on your coupling needs to be straight against your pipe this is where your seal should be. The coupling will not have a seal on the back side and should not - you state no-o-rings and that is correct.

Normally at the pipe to pipe connection is where your seal will be (You seem to indicate a silicone rubber seal). If the pipes are offset a bit - you can end up with a poor seal and the coupling will not perform. To exaggerate a bit think about a  shim stick is wider at one end then the other - your connection of pipes might sit like that. In this case hand tighten will get so far and then will not function. You would need to wiggle your pipes into better position as you tighten to get rid of any offset. Given this coupling is by a pump and the way it is connected - the vibrations of the pump can cause some issues. 

Your coupling might be worn at the threads and the fitting (or possibly a hair line crack).. might be better to simply replace it with a new one - I would replace it rather than play around with a $5 part - just faster, also I could ensure that my pipes were straight together perfect - so I have a nice fit.

Your liquid thread tape - if the pipes are fine threads use it and it might be ok, you will need to let it set up follow the directions. 

I personally do not use it, I don't like it at all.
I use the thread tape - I stretch the tape as I wrap and I roll backwards off of the tape core - wrapping my pipe. I make sure I stretched a little so it fits nicely in the threads of the pipe - in a coarse thread like that coupling I would have several wraps - the tape is cheap enough and I want a good fit and good seal - the coupling should go on a little snug to turn but not so difficult that it could break.