rotate objects like a trackball I know how to rotate objects around the center and there are rotation matrices for that as well, but, how do we rotate the object like a trackball, below is the example:

firstly I rotate around the y axis, and then the x axis, then I rotate it like a trackball, how is the third rotation achieved?
I don't know the specifics of what looks to be blender. But I would grab the view matrix of the camera. The rows of which should be orthonormal and give you forward, up, and right. Or columns depending on row or col major (technically its the inverse of the view, but that's just the transpose). Then just do some axes angle rotations around that. Something like this:
Vec3 dx = scale * (currPosX - prevPosX)
Vec3 dy = scale * (currPosY - prevPosY)
Matrix4x4 m1 = axesAngleRot(dx, camera up)
Matrix4x4 m2 = axesAngleRot(dy, camera right)

Matrix4x4 deltaRot = m1 * m2

Where scale is defined by you, and dx and dy give the delta coords of the mouse.
Then apply that to the object. You may have to tack on an extra transform to move back to origin (however you have that transform defined) before doing rotations. The cube you showed is at origin though.
Matrix4x4 finalTransform = inverseFinalTransform * deltaRot * finalTransform