Cause of delay in CAN
Transfer of a max. sized CAN frame is roughly ~100µs (at 1MBit/s). Everything else I guess is SW. But Linux + phyton is not a Realtime environment or fast in first place. You need at least a core with FiFo scheduling policy, and get rid of python - use compiled language like C++. But even that might just bring a minor improvement. You might need a real RTOS (beneath Linux) to have control over Sysclock, Interrupts, DMA,...