Every time I connect my Android to my Ubuntu it asks me to allow the connection, can I change that?
Revoking USB authorisation in developer settings (enabled by tapping "Build number" 7 times ) fixed the problem for OP and they needed to authorise only once and not on each time it was connected, as earlier

Probably it was because of authorisation being "forgotten" on the device or a handshake issue between PC and device