Is reinforcement learning only about determining the value function?
There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function.

The important part of the image is when Model-Free RL splits into  Policy Optimization (which includes policy gradients) and Q-Learning. Later you can see the two sections coming back together in algorithms that are a mix of both techniques. Even the bottom three methods in policy optimization involve some form of learning a value function. The best and most advanced algorithms use value function learning and policy optimization. The value function is only for training. Then when the agent is tested, it only uses the policy.
The most likely reason you have only heard of value function methods is because policy gradients are more complicated. There are many algorithms more advanced than ones that only use value functions and policy gradients can learn to operate in continuous actions spaces (an action can be between -1 and 1, like when moving a robot arm) while value functions can only operate with discrete action spaces (move 1 right or 1 left).
Summary: Yes, there are other methods that learn the optimal policy without a value function. The best algorithms use both types of reinforcement learning.
The SpinningUp website has a lot of information about reinforcement learning algorithms and implementations. You can learn more about direct policy optimization there. That is also where I got the image from.
This answer is specific to the most common types of Model-Free RL. There are other algorithms related to the RL problem that do not learn value functions, like inverse reinforcement learning and imitation reinforcement learning.