New derailleur - shifts up smoothly, or down - but not both
You mentioned replacing the RD and cassette but not the chain. Unless somehow the chain is nearly new already, you should replace it along with the new cassette. Unless it's very worn, it's not likely that's what's causing this though.

This is almost always a problem with either the hanger alignment or the cable.

Cleaning and lubricating the cable does not eliminate it as a potential factor. If the housings are too short, if they're contaminated with something you couldn't clean out, or if the cable has any kinks within them, they could be causing your issue. Having too short of a rear loop in particular is a common problem.

Hanger alignemt doesn't have to be obviously bent to still be out enough to cause problems, especially if you're not used to looking at it. For better or for worse, getting it aligned with a proper tool is pretty much required to eliminate it as a factor.