To be useful, doesn't a test set often become a second dev set? I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:

You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally "done" you evaluate it on your test set. Your test performance comes back much worse than your dev performance.

So now you conclude, "My dev set must be too small, causing me to overfit my hyperparameters." So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.

But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.

To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set "too much" in the above way (increasing the dev set size in a cycle again and again), it is still "mostly" unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.

Do you think this is an accurate take?

Incidentally, I'm unsure if there's anything else you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options?
You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the Reusable Holdout. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.