My first React custom hook
I don't see any issue with the code.

It's technically a hook, because it's using other hooks, so it must
start with use keyword. That feels weird to somebody coming outside of
the React world like me. Furthermore, it doesn't return any title...

The use prefix is just a React hook naming convention, and React hooks aren't required to return anything.

How would you name it?

If you are merely asking about what to name it, well, this is rather opinionated, but I suggest:
useDocumentTitleUpdater

Alone the functionality is to update the document title, something like updateDocumentTitle or documentTitleUpdater. The former, to me, implies you are handing the function the value you want it to update something to, while the latter is more of a delegation, telling the function to update the title. Since this is a React hook and there's a hook naming convention, useDocumentTitleUpdater explains the purpose of the hook. It could probably be shortened to useDocTitleUpdater if the previous was too verbose for your taste.

Is it appropriate to keep this functionality in a hook like I did or
is there some other convention in React for stuff that depends on
hooks?

This doesn't appear to be an inappropriate use of a React hook. The alternative would be a utility function that consumes the various arguments from redux, runs the two middle lines to compute the strings, and updates the title. This function would be called from a useEffect hook with the redux state values as dependencies instead.
Your hook has instead encapsulated this into a single line any functional component can use and they won't be responsible for getting the dependencies correct.