What are best practices for designing publication-quality figures?
Someone general advice:

do a Google search. There are already plenty of articles talking about all different aspects of creating good figures, and they probably do a better job than my answer here.
experiment. Just create a figure and look at the font size, linewidth, color selection and arrangement, etc. If you have an average person's understanding of aesthetics, you should be able to spot issues in your figures. Print the paper out, and see if everything is legible.
imitate. Find papers in top journals and important dissertations, and see how other people create figures. I am not in mathematics, but if you look at Science, Nature or top journals in my field like Physics Review Letter, you should get a good sense of the elements in a graph, how to convey data and ideas effectively, and how to make a graph look great.
look up guidelines. Journals in my field often have a very specific style guide (like this), which can be a very useful reference. They can include things like &quot;minimum linewidth is 0.5 pt&quot; and &quot;font size must be greater than 18 pt&quot;. They are generally good guidelines to follow.

Some specific things you can look into:

Don't be afraid to use large font size. They are probably not large enough.
using colors, and especially color maps, is tricky. There are many considerations that you may not be aware. You can easily find advice of how to use colors, and there are websites that help you select colors, so I will not repeat them here. Here are some discussions on colormap:

A Better Default Colormap for Matplotlib
Rainbow Color Map Critiques: An Overview and Annotated Bibliography
Color Map Advice for Scientific Visualization
A survey and task-based quality assessment of static 2D colormaps


make things simple. Don't overcrowd your figure.
use annotations where necessary. A figure with adequate annotations will often explain itself, even without captions.

Additional resources:

Look up a book named &quot;Trees, maps, and theorems&quot; in which there is a chapter about figures. (I have been to a great talk given by Jean-luc Doumont about how to give a good presentation.)
Figure Design (MIT EECS Communication Lab)
Choosing color palettes (Seaborn)