Forecasts for construction equipment dealers I was wondering if it is possible to speed up this script? The source file has 90,000 rows, and it has to create a separate forecast for 1200 unique dealers using fbprophet. I'm new to python so I'm not sure if there is something obvious I'm missing, or if it's just going to have to take a really long time. If any sees a simple solution that would speed it up, I would appreciate any insight. Thank you
Additional details about code:
The code takes a CSV file that has 3 columns(Month, Dealer, and Sales) and creates a filter essentially for each unique dealer name, and then uses Fbprophet to forecast the future results of each dealer. Then it combines all the information in a CSV file. It's 90,000+ rows of data typically, and normally 1200 unique dealers that need to be forecasted. The historical data is 7 years of history.
from pandas import read_csv
from pandas import to_datetime
from pandas import DataFrame


from fbprophet import Prophet
from matplotlib import pyplot
import pandas as pd
# load data
data = read_csv('X.csv', header=0)
# prepare expected column names
data.columns = ['ds','Dealer', 'y']
data['ds']= to_datetime(data['ds'])

results = pd.DataFrame()

for dealer in data['Dealer'].unique():
    df_filtered = data[data['Dealer']==dealer]
    prophet_df = df_filtered[['ds', 'y']]
    # define the model
    model = Prophet()
    # fit the model
    model.fit(prophet_df)
    # define the period for which we want a prediction
    future = list()
    for h in range(11, 13):
        date = '2020-%02d' % h
        future.append([date])
    for i in range(1, 13):
        date = '2021-%02d' % i
        future.append([date])
    for j in range(1, 13):
        date = '2022-%02d' % j
        future.append([date])
    for k in range(1, 13):
        date = '2023-%02d' % k
        future.append([date])
    future = DataFrame(future)
    future.columns = ['ds']
    future['ds']= to_datetime(future['ds'])
    # use the model to make a forecast
    forecast = model.predict(future)
    forecast['Dealer'] = dealer
#     forecast.head(5)
    results = results.append(forecast)

results.to_csv('X1.csv')


Link to Sample Data: https://docs.google.com/spreadsheets/d/1lrkRQ9juDm7WZahsPV8oq2Rh7UFlFxCDpP4AgiTxTzU/edit?usp=sharing
Took some work but got a 25-30% speedup (down from roughly 5.2 to 3.8 seconds) on the sample data you provided along with cleaner code. I'm hoping that when you scale it across the full data the gains will be larger but tbd
So what are the changes?

Move whatever possible outside loops. Your future value never changes but you recreate it (relatively inefficiently) and do other DataFrame manipulation every loop which would be 1,200 times total. Not the biggest drain but it's there.

List comprehension can be a bit faster than going through regular loops and multiple assignments. Everything I could pull into a single function like pd.read_csv or pd.date_range or list comprehension I did

Slightly faster Dataframe filtering by setting df.index = &quot;dealer&quot; and using df.loc[np.in1d(df.index, dealer)][['ds', 'y']]. If you really want to squeeze out performance that's the fastest way I can find but probably not worth the hit to readability in normal code. Still, I left it in.


Are those three improvements? Sure. They'll add up over enough data but they're still shaving fractions of a second off. The real resource killer is Prophet doing the modeling/forecasting itself which leads to

Parallelizating Prophet with multiprocessing.Pool.imap. Instead of a loop creating the future range and then the filtering and then modeling and forecasting the data for each unique dealer now it's going to utilize multiple processors to work on each forcast more-or-less simultaneously. Probably not all 1,200 at once but more than 1. That's what accounts for most of the speedup and why hopefully you'll get even larger gains like maybe 2-4x (that's very loose guess, no promises) on the full data.

Rewrite:
from fbprophet import Prophet
import pandas as pd
import numpy as np
from multiprocessing import Pool, cpu_count


def forecast_data(args):
    (df, dealer, dates) = args

    model = Prophet()
    model.fit(df.loc[np.in1d(df.index, dealer)][['ds', 'y']])

    return model.predict(dates)


if __name__ == &quot;__main__&quot;:
    df = pd.read_csv('X.csv', header=0, names=['ds', 'dealer', 'y'],
                     parse_dates=['y'], index_col=&quot;dealer&quot;)

    date_range = pd.date_range('2020-11', '2024-12', freq='MS')
    date_range = pd.DataFrame(date_range, columns=['ds'])

    p = Pool(cpu_count())

    results = pd.concat(
        [result for result in p.imap(
            forecast_data,
            [(df, dealer, date_range) for dealer in df.index.unique()]
        )])

    results.to_csv('X2.csv')

P.S. If you wonder why forecast_data takes a tuple and unpacks it it's because I tried to write a memoized factory function that took df and dates are arguments and returned forecast(dealer) but that caused multiprocessing to have a pickle about using a local pickle.