How to assign a specific color to each taxonomy term in a chart created with the Charts module?
As you must have noticed, creating charts using the Views UI of the Charts module, you can only select 1 color for each chart.

However, if you would rework your chart to a Combination chart, you could set a color for each of the charts. Here are some more details about how I'd create the combination chart (with 1 color for each taxonomy term):


Use your current chart display as the parent chart for 3 child charts to be added to it (as explained below). Adapt its chart settings where needed, so that it won't get in the way of the 3 child charts to be added to it. For that, I'd make it a "line chart", possibly with some other refinements like color and label of that chart also.
Add a 1st (child) chart display with a views filter that limits the values to only your 1st taxonomy term, and specify a 1st color in its chart settings. Use the chart in bullet "1." as its parent chart.
Add a 2nd (child) chart display with a views filter that limits the values to only your 2nd taxonomy term, and specify a 2nd color in its chart settings. Use the chart in bullet "1." as its parent chart.
Add a 3rd (child) chart display with a views filter that limits the values to only your 3rd taxonomy term, and specify a 3rd color** in its chart settings. Use the chart in bullet "1." as its parent chart.


Here is an example of a column chart created as a combination chart:



And this is a variation of it in the format of an area chart (for "total") with 2 line charts:



As both examples illustrate, for each chart in such combination chart, you can assign any combination of colors ... Which is what this question was about, no?

Note: Using this approach, the parent chart in bullet "1." above will avoid that you run into the issue about "holes" in your X-axis data. Refer to my answer to the question about "How to create a combination chart with the Charts module if some X-axis values are missing?" for more details on that.