How can we prevent AGI from doing drugs? I recently read some introductions to AI alignment, AIXI and decision theory things.
As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not causing something like the paperclip apocalypse.
Then a question comes to my mind that whatever the utility function would be, we need a computer to compute the utility and reward, so that there is no way to prevent AGI from seeking it to manipulate the utility function to always give the maximum reward.
Just like we humans know that we can give happiness to ourselves in chemical ways and some people actually do so.
Is there any way to prevent this from happening? Not just protecting the utility calculator physically from AGI (How can we sure it works forever?), but preventing AGI from thinking of it?
You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.
One thing I am sure though is that ML is known for &quot;cheating&quot; (see for exemple). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not &quot;cheat&quot;. Thus, the answer is mostly contained in your assumptions.
Whether we are able to build an AGI, what &quot;cheating&quot; we would have to overcome to do so and how we will be able to do so is mostly undetermined.