Building a model in two stages This question is regarding a use case related to predictive maintenance. The final model built (based on the steps mentioned below) is used to predict the failure of a particular component for a particular type of device. In the Machine Learning context, it's regression use case for which deep learning model(s) are being built.

The model building process consists of two stages: 


Initially based on the data (temporal in nature) from multiple devices, one single model has been built. The model has been cross validated with the data from the same set of devices (expected).
Now based on the model type and hyperparameters decided at #1, models are being built for individual devices (using data from respective devices. The temporal frequency of data at this stage is different from the frequency of the data used at #1). These models (for individual devices) will be deployed in production (i.e. one model per device). The model built at #1 will NOT be used in production.


Is this two stage process a valid approach of model building?

As per my understanding, instead of the two stage process and large number of individual models, a single model can be built with data from all devices (with matching frequency) in one shot. There is a possibility that devices differ from each other based on different characteristics. But those characteristics could be used as a feature while building the single model. 

Based on domain knowledge, I am more or less convinced that one single model can server multiple devices. But, my question is more about the validity of two stage approach:


Decide model type and hyperparameters in the first stage.
Build individual models with data from individual devices based on the model type and hyperparameters decided in the first stage.
This is a bad idea. The cross validation score that you obtained from your first model is valid only for that model. If you train a new model there is no guarantee that it will have similar performance, you have to test each and every model.

This could work only if you can guarantee that the data for the second model will be very similar to the data of the first model. You would have to be very sure of that.