Why do I get a BIGINT overflow on MySQL only for some queries?
MySQL, in the absence of explicit data type information, tries to determine the appropriate data types for literal values based on the value limits. Since the scale of 18446744073709551616 exceeds the allowed range for UNSIGNED BIGINT, it must be a DECIMAL then, which has a much larger range. You can see this in action by creating a table from your SELECT:

create table test as  SELECT 18446744073709551615, 18446744073709551616


and describing it:

describe test


which results in

Field                   Type                    Null    Key     Default     Extra
----------------------- ----------------------- ------- ------- ----------- -----------
18446744073709551615    bigint(20) unsigned     NO              0   
18446744073709551616    decimal(20,0)           NO              0   


dbfiddle link