Issue translating large amounts of tweets using Google Translate I am working on translating large amounts of tweets using this deep-translator which uses the Google Translate API.
Initially everything was fine and tweets were translated with no problems whatsoever but I recently encountered an issue.
The issue with the data currently is that when there are non-English words in a particular tweet as in the attached image, Google Translate marks it as English and does not translate them. In the API I have set the source language to auto-detect the words then translate them to English.

The only workaround I have come up with to solve this is to turn the tweet into chunks and perform batch translations on them.
Example string:
&quot;NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM NEWS TRANSFER WINDOW UPDATE 손흥민 Son Award Link to Premier League Defen... TOTTENHAM NEWS TRANSFER WINDOW UPDATE Carabao Cup Win Final. 손흥민 Son Contract&quot;

Output after chunking the string into batches with a maximum of 5 words:
['NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM', 'NEWS TRANSFER WINDOW UPDATE 손흥민', 'Son Award Link to Premier', 'League Defen... TOTTENHAM NEWS TRANSFER', 'WINDOW UPDATE Carabao Cup Win', 'Final. 손흥민 Son Contract']

The issue is to translate the batch it takes about 13 seconds whereas if I translate the entire string(even though it won’t work) it takes under a second as the batched string performs 6 API request compared to the single API request for the normal string.
The time to translate will be quite high when I batch the strings and translate them.
Assuming it takes 13 seconds per tweet with 5000 tweets in the csv file I am currently working on it will take roughly 18 hours and I have csv files with significantly more tweets than this.
The chunk size of 5 is what I found to still be able to translate the words even if there is a single non-English word in the string. Anything more than that will still not translate the tweets.
Anyone have any workarounds that are faster than this?
So this was my work around:

Convert the list of strings that need to be translated into an indexed tuple with the first value being the index and the second value being the string:

s = [(0, 'NEW'), (1, 'YOUTUBE'), (2, 'VIDEO'), (3, 'OUT'), (4, 'NOW:TOTTENHAM'), (5, 'NEWS'), (6, 'TRANSFER'), (7, 'WINDOW'), (8, 'UPDATE'), (9, '손흥민'), (10, 'Son'), (11, 'Award'), (12, 'Link'), (13, 'to'), (14, 'Premier'), (15, 'League'), (16, 'Defen...'), (17, 'TOTTENHAM'), (18, 'NEWS'), (19, 'TRANSFER'), (20, 'WINDOW'), (21, 'UPDATE'), (22, 'Carabao'), (23, 'Cup'), (24, 'Win'), (25, 'Final.'), (26, '손흥민'), (27, 'Son'), (28, 'Contract')]


Next as the words that were causing issues when translating were ASCII I extracted them using this code:

(index, item) for (index, item) in s if not item.isascii()]
# result
[(9, '손흥민'), (26, '손흥민')]


Then I ran the translations on these words instead of the entire string or the chunks of strings.
After translating I then added them back in the list of tuples
Convert the list of tuples to a list of strings and concatenate them into a single string

This significantly reduced the time to translate the data.