How do I get Views content items in "my language" but fall back to "default" if the content isn't translated?
I think your requirements can be restructured into the following:


Show all content once.
Show the content in the current language, fall back to the other.


Then it becomes pretty easy:


Add the "Default Translation" filter to a view. That basically ensures that all your nodes are shown once because each node can only have one default translation.
Select "Interface text language selected for page" for "Rendering" language. That will attempt to display the site in the current language and fall back if that's not available. (Interface text sounds a bit misleading, but unless you explicitly enabled separate content language negotiation, that's the same as the content language).