Is there an App or Extension that allows for per-window zoom/magnification?
What you ask isn't possible.

There are apps that will zoom a portion of the screen - I use the loupe in Xscope for this purpose and it does a very nice job of magnifying the area around the cursor, but it's not able to grab an entire window and zoom it in place.

The OS takes the content that the application draws and maps that onto the space where the window exists, so you would need to fundamentally alter the screen rendering on OS X to force an app that hadn't considered zoom levels to zoom only itself.

Unless you really like Xscope, my guess is you'll get in the habit of zooming the entire screen or possible find another software package to replace the one that won't zoom.

The other workaround would be to run that program in a virtual machine and force it to be full screen zoomed in that one OS and keep the rest of your programs running outside the VM. That would accomplish exactly what you ask at some level of time commitment to virtualize the OS and get that set up.