Intuitive explanation for representing gradient in higher dimensions
Yes, you are thinking of multiple dimensions in the wrong way, but I'm not sure anybody knows the right way to think about them.
You don't represent complex networks with many parameters in 3D space. We start in low dimensional 2D and 3D space to develop an intuition about what happens with the gradient, how to move along it, and so on. Then, we go into thousands dimensional spaces, were we don't know what happens, and we apply our intuition - those spaces work the same way, just that we cannot visualize or comprehend them.
Your cost function will have thousands of dimensions, and your gradient will be the same. But we rely on our intuition from low dimensional spaces to manipulate them.