How do I install an app with system-rights on a non-rooted smartphone? I would like to install logcat as system-app on a non-rooted phone.

Is that possible (using adb without root-access)?
No. A system app has to be either installed by root or signed with the same key that signed the ROM image.