Postgres handle case to continue Insert other row if some row doesn't exists
Build a SELECT expression that does what you want, and then use INSERT INTO ... SELECT ... to insert it into the table.
To get a SELECT that does what you want, wrap the VALUES expression in parentheses, add an alias with field names for your sub-selects, then add a WHERE for the project and user IDs being NOT NULL. The final query becomes:
insert into projects_users select * from 
(VALUES (
            (Select id from users where username='user_id1'),
            (Select id from projects where name='project_id2'),
            True
        ),(
            (Select id from users where username='user_id3'),
            (Select id from projects where name='project_id4'),
            True
        )
) as r(user_id, project_id, is_admin)
WHERE user_id is not null and project_id is not null;