AppleScript - GUI Scripting and setting checkbox, radio button and field values
try this little syntax change:

on setfmFileOptions(app_name)

    -- choose the Developer Utilities menu item from the Tools menu of FileMaker Pro Advanced

    do_menu(app_name, "File", "File Options...")


    -- click the "Log in using" checkbox:
    tell application "System Events"
        tell process app_name
            repeat until (exists checkbox "Log in using:" of tab group 1 of window 1)
                delay 1
            end repeat
            click checkbox "Log in using:" of tab group 1 of window 1
        end tell
    end tell
end setfmFileOptions


You can't use window 1 without telling what process window 1 is in. You'll have to specify app_name as a parameter now.

So, like this:

setfmFileOptions("FileMaker Pro Advanced")