How to overcome overfitting to single player styles in reinforcement learning?
Both of the solutions you suggest seem to be built around the intuition that it's good to ensure that there is sufficient variety in the experiences that you provide to your RL algorithm.

That intuition is good, but it should not come at (too much of) a cost in playing strength of the opponent. I'm afraid that your first solution may break down because of this; Tic Tac Toe is such a simple game, any agent that doesn't play optimally can really be viewed as a very poor agent... and I think minimax agents with very low limits on search depth may end up playing suboptimally. Your second solution seems better in this regard, that could help.

For this particular case of Tic Tac Toe, I suspect you should be able to train just fine against only optimal minimax agents, as long as you make sure that those optimal minimax agents break ties randomly. In some situations, there may be multiple different actions that are all "equally optimal". Then, you'll want to make sure that your minimax agents break ties randomly, rather than always deterministically picking the same action. This can be done, for example, by making sure to always shuffle your lists of legal moves in minimax after they are generated.