What is necessary to work with inset face mode?
In your image I can see that you are using Vertex Select. In this case, activate Face Select.



Inset Faces is a Face Tool. It needs you to select all the vertices/edges that make up a face.

You can use it via Vertex Select (or Edge Select) as well, however, you need to select the central vertex to select the tris (triangles or faces with 3 vertices).