which Model to apply on panel data where unique id has 6-8 records and total records are 2,000,000? I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply.
I have data from Q1-18 till Q4-2020
I have 2,000,000 rows and 200,000 unique id and 20 columns
For each id I have only 6-8 past quarter values, max quarter for each id are 8 quarters and for some id I have only 6 quarters where few quarter value are not available for that id
Below is the basic idea of what my data set look like
Quarter - respective business quarter for that year
Target - is the sales volume in ratio
I am trying to Predict - Target column for 2021 Q1 quarter
I have 8-10 different numeric columns and state , quarter and ID as category columns
I would appreciate if someone could suggest me what kind of modelling could be performed
Your data is a good candidate for a mixed effect model.
You have two potential random effects that look to be crossed: I see ID as one and state as another, as it seems that any ID can belong to any state.
Essentially, you can leverage the fact that some states and some IDs will have less data than others (and in general, some states and IDs will vary more than others). States and IDs who exhibit such characteristics will have their predictions bolstered by other states and IDs (called partial pooling). You also need to use time as a variable (perhaps convert quarter to 2019.25, 2019.5, etc. or use two time variables year, quarter) because you wish to predict the next period.
I think generalized linear mixed effect models can scale to 2 million observations. There are also machine learning methods that allow for mixed effects: mixed effect random forests (MixRF), mixed effect gradient boosting (GPBoost/mboost), and of course you can go full Bayesian as well (though you will likely need to use an approximation method since your data is large). The ML models may scale better.