How to use max function in such condition?
Try it this way:

select userid from (
(select userid from guestuser order by userid DESC LIMIT 1)
union all 
(select userid from olduser ORDER BY userid DESC LIMIT 1)
) sq 
order by userid DESC LIMIT 1;