French-press coffee has a powdery taste
You didn't mention the size of your grind. You definitely want a course to medium grind. A fine/espresso grind will leave a lot of sediment and produce a bitter cup of coffee from over extraction.

Two things you can do to minimize sediment:


After letting it brew, before plunging, use a couple of spoons to scoop out as much of the coffee grounds as you can. This video shows a demonstration of this.
Decant the coffee. Pour the coffee from the press into another jug and let it settle a moment before pouring into your cups.


Another tip: Place a saucer over the top of the press for the four minutes to minimise heat loss.