Speeding up pandas apply
flag as boolean
If you change the flags from 'Y' and 'N' to True and False You can use boolean indexing. This should speed up a lot of things already
set
You check for each combination word in dest_words for word in source_words on a list of words. If the check matches, you convert to a set. The containment check would be sped up by checking against a list, but using set comparisons would speed this up a lot.
import typing

def partial_match(
    source_words: typing.Set[str], dest_words: typing.Set[str], index=None
) -&gt; typing.Tuple[typing.Any, typing.Optional[str]]:
    intersection = source_words &amp; dest_words
    if intersection:
        return index, &quot;, &quot;.join(intersection)
    return index, None

def exact_match(
    source_words: typing.Set[str], dest_words: typing.Set[str], index=None
) -&gt; typing.Tuple[typing.Any, typing.Optional[str]]:
    if source_words == dest_words:
        return index, &quot;, &quot;.join(source_words)
    return index, None

The reason I chose to return the index along with it is to be able to reconstruct the series easier when reassembling everything.
Don't touch the original data
You change your source data inplace (by adding columns). Better would be to leave this untouched, and keep the destination words etc in separate series.
Series.where
You can replace calls like this
if x['remove_bus_ending'] == 'Y':
        x['dest_words'] = x['dest_words_2']
    else:
        x['dest_words'] = x['dest_words_1']

with Series.where
a = pd.Series(list(&quot;abcd&quot;))
b = pd.Series(list(&quot;efgh&quot;))
c = pd.Series([True, True, False, True])
b.where(c, other=a)


0    e
1    f
2    c
3    h
dtype: object


If your data looks like this:
from io import StringIO

import pandas as pd

def setify(s):
    return s.str.strip(&quot;[]&quot;).str.split(&quot;, &quot;).apply(set)

df = pd.read_csv(StringIO(data_str), sep=&quot;\s\s+&quot;, index_col=False, engine='python')
df[&quot;source_words&quot;] = setify(df[&quot;source_words&quot;])
df[&quot;dest_words_1&quot;] = setify(df[&quot;dest_words_1&quot;])
df[&quot;dest_word_2&quot;] = setify(df[&quot;dest_word_2&quot;])
df[&quot;remove_bus_ending&quot;] = df[&quot;remove_bus_ending&quot;] == &quot;Y&quot;
df[&quot;partial_match_flag&quot;] = df[&quot;partial_match_flag&quot;] == &quot;Y&quot;
df[&quot;exact_match_flag&quot;] = df[&quot;exact_match_flag&quot;] == &quot;Y&quot;

intermediate dataframe
If you want to split the dataframe with arraysplit, you'll need to provide an intermediate form with the info you need:
df_intermediate = pd.concat(
    [
        df[&quot;dest_word_2&quot;]
        .where(df[&quot;remove_bus_ending&quot;], other=df[&quot;dest_words_1&quot;])
        .rename(&quot;dest_words&quot;),
        df[&quot;source_words&quot;],
    ],
    axis=1,
)

You can even split it immediately according to what matching is needed
df_intermediate_partial = df_intermediate.loc[df[&quot;partial_match_flag&quot;]]
df_intermediate_exact = df_intermediate.loc[df[&quot;exact_match_flag&quot;]]

applying the function
not parallel:
result_partial = list(
    map(
        partial_match,
        df_intermediate_partial[&quot;source_words&quot;],
        df_intermediate_partial[&quot;dest_words&quot;],
        df_intermediate_partial.index,
    )
)


results_exact = list(
    map(
        exact_match,
        df_intermediate_exact[&quot;source_words&quot;],
        df_intermediate_exact[&quot;dest_words&quot;],
        df_intermediate_exact.index,
    )
)

result = pd.Series(result_partial + results_exact)

This should be easy to parallelize. Since I'm no expert on that, I'll leave that to others.
context manager
Most of the examples I found in the multiprocessing documantation work with a context manager that takes care of the closing of the pool
with Pool(processes=4) as pool:
    ... # parallel part of the code