Can you get a single worker for a single die if you can only use one die roll?
You only need one die to get two workers
From the Castles of Burgundy rules, sourced here

Action &quot;Take worker tiles&quot;
Finally, a player may choose to use any die result to take 2 worker
tiles from the general supply. In this case, the die result itself is
meaningless.

The word 'die' is singular, indicating that you only need a single die to take this action.  This is in keeping with every other action in the game - every action takes exactly one die.
So any house rules are unnecessary.