To be useful, doesn't a test set often become a second dev set?
You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the Reusable Holdout. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.