How to create a grading rubric that properly incentivizes and challenges students
In general, I would recommend keeping your grading rubric simple -- complicated rubrics have a way of causing confusion and sometimes resentment when they're misunderstood.

In the situation you describe, I would choose to compute the score as the average of that on the three required problems, weighted equally. If the student solved any of the optional problems, choose whichever looks the best and substitute this score for the lowest of the three required scores, but only if it improves the student's grade.