What is the process of inventing deep neural network models? How researchers deal with long training times?
Part of the answer is that new architectures are often variants of existing architectures. There are some rough heuristics people follow (e.g. using powers of 2 in layer sizes, changing layer sizes according to a schedule, adding normalization layers, etc.). However, even when building off an existing architecture, there can still be a lot of toying around with different configurations and hyperparameters to require copious training time.
Researchers working for a company will likely have the luxury of some serious compute, e.g. GPU clusters or cloud services like AWS. This can really make a huge difference over training a model on your PC GPU. Further, many models can be trained at once. Many academic institutions also employ cluster computing on dedicated GPU clusters - for example, here at the University of Utah our department has a cluster with 8 really excellent GPUs.
One last thing I will note, is that it's not always necessary to train the model to completion on the full training set to get a sense of which network configuration and hyperparameter settings are working better than others. If I have a massive dataset that will take days to train on, I may take a smaller subset of the training data and do some quick comparisons between different models to get a feel for what changes have the greatest impact on test performance. It's not completely reliable, but it can guide intuition to narrow down the number of models you want to train on the full dataset to convergence. For example, using this technique I was able to find that predictions from my model were fairly invariant to the exact layer sizes I was using and was far more sensitive to a hyperparameter in my optimization solver.