What is the best measurement for how good an action of a reinforcement learning agent really is?
Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?


To understand the short-term effects of an action, just take each of the available actions from the current state and observe the reward for each of them. The action that gives you the highest immediate reward is the best action. However, note that the reward function may change or could be stochastic. In those cases, you may need to estimate the best action e.g. by executing it multiple times.

If you want to know the action that gives you the highest amount of reward in the long run (i.e. that gives you the highest return), then you can use one of the available RL algorithms, which were invented exactly to solve this problem. Basically, you're asking us what is the best RL algorithm. It depends on the problem, as usual.

If you want to know the effects of an action in e.g. $n$ steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the discount factor so that the next $n$ rewards are more valuable (or are the only ones considered) than the rewards after $n$ steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.

Note that, in this answer, I am just trying to give you the idea and intuition behind a possible answer to your question (also because your question isn't really suited to provide more detailed or rigorous answers).