Do I need a normal matrix if I calculate lighting in an objects space? As a quick reminder, here is what normal matrix is and why do we need it.

Now the question - if I take the light position and transform it into an object space than I don't need the normal matrix at all and actually it will save me a good amount of calculations (don't need to multiply a normal by the matrix). By I can't draw a mental picture of how this could work for a scaled object, what does it mean to transform a point (light position) into a scaled object space? And what about a directional light where I have no position at all?
By lighting in object space, sure, you could avoid transforming the normals into world space. However, you'd have to transform the light positions/vectors into the object's space. Also, if you want any view-dependent shading such as specular, you would have to transform the camera position into object space as well, to calculate view vectors.

Now, all of this can be done straightforwardly using the inverse of the object's world matrix. The inverse world matrix will include the inverse of the object's scale (as well as correctly accounting for the inverse of its rotation and translation), so it will transform the light &amp; camera positions to the appropriate points in the scaled coordinate system. Directional lights don't have a position, but you can transform their direction vector by the inverse world matrix to account for rotation and scale. All of this is fine so far.

However, you will then run into trouble when doing lighting calculations like N·L or N·V. For exactly the same reason that normals don't transform correctly under non-uniform scaling without using a special normal matrix, these dot products will not come out with the same result when done in a non-uniformly scaled object space as compared to world space. Namely, non-uniform scaling distorts angles. So, all your lighting calculations and BRDFs that depend on angles are going to be messed up.

While there are ways one could correct this problem by using additional matrices when calculating dot products... it's generally easier and faster to do lighting in world space (or view space). In any case, transforming the normals to world space is a pretty trivial amount of math by modern GPU standards, especially as it only needs to be done per vertex.