What actually happens to footage that is upscaled or downscaled?
May i understand your question as

How do i have someone/something that understands the missing puzzles from
my upscaled images?

If so, you are talking some intelligence/experience in upscaling or AI-upscaling. Then, plz take a look of AI Upscalers; it mentions

An open source AI upscaler (super sampler) based on a generative adversarial network architecture.
An open source image enhancer that seeks to restore lost texture details from known types
Commercial super sampler that uses artificial intelligence

Good luck!