Rotate a model or a view with mouse input to produce intuitive results
Here's a more complete answer from my comment. The end product we want is:

Projection * ViewTranslate * Model * ViewRotate


All are 4x4 matrices. Where model is your models transformations. To implement you can just separate out the view's translate and rotate during your draw.

If you're using an abstraction that requires you to use Proj * View * Model. Or you just really don't want to ruin your own abstraction that follows that pattern. You'll need to modify model to cancel out views rotational part and put it on the right side:

Model = ModelTranslate * View^-1 * ViewTranslate * ModelRotate * ViewRotate 
      = ModelTranslate * ViewRotate^-1 * ModelRotate * ViewRotate


Here's some of my code that does the above. It does not matter if model or view is translated, objects will still rotate around their local origins:

glm::vec2 diff = (prevMousePos - mousePos) * rotSpeed;
glm::mat4 rotX = MathHelp::matrixRotateY(diff.x);
glm::mat4 rotY = MathHelp::matrixRotateX(diff.y);

// Decompose the translate and rotate part of view
glm::mat4 viewT, viewR;
Decompose(cam-&gt;view, viewT, viewR);

// Decompose the translate and rotate part of model
glm::mat4 modelT, modelR;
Decompose(exampleObj-&gt;model, modelT, modelR);

exampleObj-&gt;model = modelT * glm::inverse(viewR) * rotX * rotY * viewR * modelR;


This preserves the existing rotation and translation in exampleObj->model as the function gets called continually applying small rotations. And assumes you'll be using Proj * View * Model.

Lastly decompose just separates out the 3x3 rotation/scaling from the translational part:

void Decompose(glm::mat4 m, glm::mat4&amp; t, glm::mat4&amp; r)
{
    glm::mat4 I4 = glm::mat4(1.0f); // Identity
    for (unsigned int i = 0; i &lt; 4; i++)
    {
        for (unsigned int j = 0; j &lt; 4; j++)
        {
            if (i == 3 || j == 3)
            {
                r[i][j] = I4[i][j];
                t[i][j] = m[i][j];
            }
            else
            {
                r[i][j] = m[i][j];
                t[i][j] = I4[i][j];
            }
        }
    }
}