Translate an arcball camera upwards or left-right This problem is giving me a really hard time.

I'm always looking at the origin and have a camera that moves around it in an arcball mode. 

That part works beautifully:



I'm calculating the camera position as follows:

camera.pos.x = origin.x + distance (arrow) * cos(angleX)
camera.pos.z = origin.z + distance * sin(angleX)
camera.pos.y : Float = origin.y + distance * sin(angleY)


But now I also want to be able to shift the position of the camera laterally, i.e. on (or parallel to) the tangent plane to the sphere between the origin and the camera position. (That tangent plane being the screen plane.)

I.e. I want to shift the view matrix to the left/right and upwards/downwards, so that the camera is shifted laterally. (I need this for features such as moving the view on a touchscreen [two-finger gesture] or texture projection).

How should I go about this?
Here's a basic camera with panning (some parts removed for brevity). It's not arcball but should get the idea across. Basically it just transforms some directions of a plane in view space to one that is in world space.

rotate and pan are called on mouse move with the change in mouse position.

// Updates the view matrix based on parameters
void updateCam()
{
    // Calc eye position on sphere
    glm::vec3 eyePos = glm::vec3(
        rho * sin(phi) * cos(theta),
        rho * cos(phi),
        rho * sin(phi) * sin(theta));

    setEyePos(eyePos + focalPt);
    setFocalPt(focalPt);
    updateLookAt(); // Uses eyePos and focalPt to calculate a view matrix
}

// Input is a change in position
void rotate(glm::vec2 diff)
{
    theta += diff.x * rotateSpeed;
    phi -= diff.y * rotateSpeed;

    // Clamp to prevent user flipping camera at top of sphere
    phi = clamp(phi, 0.01f, 3.14f);
    updateCam();
}
void pan(glm::vec2 diff)
{
    // Up and right are the directions to pan when camera is facing forward
    glm::mat3 invView = glm::inverse(view);
    glm::vec3 dx = invView * glm::vec3(1.0f, 0.0f, 0.0f);
    glm::vec3 dy = invView * glm::vec3(0.0f, 1.0f, 0.0f);

    // Scale shift with scale so shift is relative to how far you are zoomed out
    focalPt += (dy * diff.y - dx * diff.x) * shiftSpeed;
    updateCam();
}