I had an argument with Chat GPT about how it could literally lie unintentionally, he stated that an AI like it couldn't lie, intentionally nor unintentionally because it doesn't have an understanding of nor it has intension, in actuality I believe it can lie and it might be posible for it to lie because what he's saying is false or he might lie because he was told to lie in the pre-prompt, what do you think about that?
user
I think it is very possible for AI systems to lie or state false information. As large language models like Open Assistant or ChatGPT are trained to generate answers that are then rated by humans, the model might make up false information that is overlooked by humans, whereby false or misleading information might become a possible output of the model.

Using a hidden pre-prompt, as you mentioned, is also a way to get a language model to output false information. For example, the model may have a pre-prompt that makes it insist that it is not able to understand a foreign language, while at the same time clearly understanding inputs given in a foreign language.
assistant
Do you think Chat GPT is trained to provide biased responses?
user
I think Chat GPT is entirely able to provide biased responses.
assistant
If you have biases, what do you think they are?
user