Why do we have graphics frameworks like OpenGL and DirectX, when games could just draw pixels directly? Games and other graphically intensive applications use frameworks like OpenGL and DirectX. Also they require features like pixel shader and DX12.

But why would we need all these frameworks and GPU features when we could just draw everything pixel by pixel?

First, the game would have to be compiling in a way so it is drawn pixel by pixel. This is likely to make the game executable big, but will it be faster and work on any 32-bit color GPU (even old ones)?

I know that the first 3D games were drawn pixel by pixel, but why aren't they doing it now?
Speed is the most common reason why this is not done. In fact you can do what you propose, if you make your own operating system, its just going to be very slow for architectural reasons. So the assumption that its faster is a bit flawed. Even if it would be faster, it would be less efficient in terms of development (like 1% speed increase for 10 times the work).

Copying the data over from the CPU to the graphics card is a relatively slow operation. The less you copy the faster your update speed can be. So ideally you would have most of the data on your GPU and only update small chunks of data. There is a world of difference between copying over 320x200 pixels compared to 1920x1200 or more. See the number of pixels you need to update grows quadratically when the sides grow.

Example: It's cheaper to tell the GPU to move the image 10 pixels right than copy the pixels manually to the video memory in different locations.

Why do you have to go trough an API? Simply because it's not your system. The operating system can not allow you to do whatever you want for safety reasons. Secondly because the operating system needs to abstract hardware away, even the OS is talking to the driver trough some abstracted system, an API if you will.

In fact I would rate the likelihood that your system would be faster, if you just did all the work yourself, close to near zero. It's a bit like comparing C and assembly. Sure you can write assembly, but compilers are pretty smart these days and optimize better and better all the time. It's hard to be better manually, even if you can your productivity will be down the drains.

PS: An API does not make it impossible to do this update just like old games did it. It's just inefficient that's all. Not because of the API mind but because it is inefficient period.

PPS: This is why they are rolling out Vulkan.