What exactly is it about reality that is so difficult to recreate in a 3D simulation? I have to admit that, while watching footage of the enhanced 2020 version of the 2018 game Forza Horizon 4, when they are driving fast, I have to sometimes remind myself that I am watching a video game and not a video recording of reality. It's the closest I've ever seen to photo realism, but as soon as the car stops, or they go too slowly, or they do something slightly not expected, the illusion quickly falls apart.
It's just a bunch of smoke and mirrors and dead polygon objects after all. Not an actual photo-realistic 3D version of England where I can &quot;drive anywhere and do anything&quot;. But for small moments, it can fool me. Example gameplay footage: https://youtu.be/PqwQDRersIM?t=42
What exactly is the &quot;missing piece&quot; at this point? It's impossible for me to not be impressed by the game, but I still don't want to play it. I can tell that I would very quickly get bored of the illusion, and just grow ever more frustrated by the limitations of the reality as presented in the game.
For me, it's largely knowing that it's all dead 3D objects. There are no detailed rooms inside all of those buildings which I could enter and go through. There are no people either, or, if there are, they don't look or behave anything like real humans would. There are many visual details which just look &quot;off&quot;, and the more photo-realistic it looks overall, the more demanding my brain becomes and the more eager I am to point out any flaw.
If you then take a look at this random actual video, even that game looks like garbage in comparison: https://youtu.be/FMOb5-e3J7Y?t=49 (yes, I know that the settings/locations are quite different, but that doesn't really matter).
Something about reality is so much &quot;richer&quot; and &quot;real&quot; than what they can do with a 3D world, but I don't quite understand what exactly. I feel as if I should be more impressed than I actually am by that car game. It looks objectively amazing, yet at the same time, I have very little interest in actually playing it. And keep in mind that I'm a person who has been hunting for photo-realistic 3D simulations my entire life and who is endlessly fascinated by the concept.
One thing I can say for sure: it has almost nothing to do with the resolution of the game/camera. Even if you take a very old and cheap camera and go filming reality, it will look a million times better than the 8K game.
Can you explain to me why we don't have a simulation that looks and behaves just like the actual video I just linked to?
As @AnnoyinC already mentioned in his comment, we are simply lacking computational power and storage requirements (RAM).
For the visual effects, you need to calculate every ray of light that somehow ends up in the observer's eye after an arbitrary number of reflections on different surfaces. The reflections depend on the surface's microstructure. This alone is so extremely demanding that is far from possible to calculate it in real-time with current consumer hardware. The first steps into this direction are already taken with ray tracing support of modern graphic cards but it is still very limited.
And that's just the &quot;easy part&quot;.
It gets way more complicated if you want to have objects and fluids that behave physically correct. While each ray of light is more or less independent of other rays, physical objects influence each other. Also, simulating the correct reaction of an arbitrary object to an external force can already take days and weeks for a single timestep/frame since you need to solve extremely large equation systems that are often non-linear. If you want to know why I would suggest searching for a &quot;Finite-Element-Method&quot; or &quot;Computational Fluid Dynamics&quot; tutorial. These techniques are commonly used by engineers and separate a structure into a mesh of smaller interacting substructures. The results get closer to the behavior of the real structure/fluid the denser the mesh. However, doubling the mesh density in each of the 3 spatial directions comes at the price of needing 8 times more computational power and RAM. Now think of a tree waving in the wind with thousands of leaves where each leaf alone is a complex physical object... I think you get the picture.
To get around all these problems, a lot of approximations are used. While most of them are quite good illusions they will give wrong results at some point. However, that doesn't necessarily mean that we realize it.