Why can anonymous users access private files using the system URL?
/sites/default/files/private/assets/image.jpg will never work because of an .htaccess directive prohibiting direct access to that directory.

Beyond that, this strikes me as an issue with OG.  I'd try executing a module_implements('file_download') and have a look at what modules might be interacting with determining if a given file is accessible.