Title: Robust solution to plot large datasets with custom line formats using loops
Tags: python;pandas
Question: I am attempting to graph population and country data by year from the Comparative Political Data Set. Ex:
year    country   pop15_64
1960  Australia    6296.50
1961  Australia    6428.60
1962  Australia    6571.50
1960        USA  107919.4
1961        USA  109048.6
1962        USA  111177.3
The problem comes in attempting to see what line corresponds to what country. I have tried a number of different methods, and the easiest to differentiate seems to be plotting lines as dashes. Ex (snippet, not full):
df2.loc[df2['country'] == 'Cyprus'].plot(x='year', y='pop15_64', ax=ax, color='k')
df2.loc[df2['country'] == 'Czech Republic'].plot(x='year', y='pop15_64', ax=ax, color='g', dashes=[3, 3])
Resulting graph:
But the above is not robust. Adding to the dataset requires hard coding another line, as does removing a country. This loops robustly over the data:
fig, ax = plt.subplots()
plt.xlabel('Year')
plt.ylabel('Population')
for countries in df2['country'].unique():
df2[df2['country']==countries].plot.line(x='year',
y='pop15_64',
ax=ax,
label=countries,
title='Country Population by Year')
ax.legend(handlelength=5)
plt.show()
Resulting graph:
...but I see no way of customizing lines within those loops without breaking it. For example, this:
for countries, x in df2['country'].unique():
df2[df2['country']==countries].plot.line(x='year',
y='pop15_64',
ax=ax,
dashes=[x+2, x],
label=countries,
title='Country Population by Year')
...results in a "ValueError: too many values to unpack (expected 2)"
What else I tried:
.groupby - couldn't get groupby and iterables to work together in one plot
iterables - had difficulty getting unique countries without losing the underlying data, still had line customization issues.
I admit that while I have experience in C, C++, and C#, I am a novice at python and more so with pandas. It requires a restructuring in how one thinks about problems. As a result, applying the many vaguely similar StackOverflow Q&amp;A about custom lines and legends has not resulted in solutions like it might for someone with more experience. I hope someone knows an answer, and that this might help more than just me.
Update: after Peter Leimbigler's answer I modified the above for this tentative solution:
for i, countries in enumerate(df2['country'].unique()):
df2[df2['country']==countries].plot.line(x='year',
y='pop15_64',
ax=ax,
dashes=[3+i, i/2, abs(i-2), i/2],
label=countries)
Comment: @QuangHoang Look at that first graph. Do you see any outliers? For various types of data, that info is useful. In the second graph, how would you tell which line is the outlier?

Comment: Honestly, I don't see the point of plot 20 lines in the same axis.

Here is the accepted answer: I would suggest the following change to your attempt at changing dash properties within a loop:
for i, countries in enumerate(df2['country'].unique()):
This will need further stylistic adjustment, but should be a good starting point.
More broadly, to echo Quang Hoang's comment, it's very difficult to present this many lines on one plot without visual confusion. Two possible workarounds:
Group the dataset by greater region (North &amp; South America, SE Asia, etc.) and build several plots.
Build an interactive line plot with zooming, panning, and toggling of countries, in something like Bokeh, Plotly Dash, or Holoviews.
Comment for this answer: Thanks very much for the quick response! Re: QH's comment to my OP, I appreciate that this is a confusing graph, but for the amount of variables I am working with on this dataset it is in some cases a necessity. Country population was just for this question. Sometimes outliers are easier to notice on one graph. That said, I appreciate the suggestions, and will keep them in mind. I am going to take a close look at Bokeh etc. I have been hunting for a way to make this more interactive than basic matplotlib.
Comment for this answer: @LabGecko glad to help! I see from your other comment that your goal is to identify outliers quickly, in which case it does make sense to give each country a unique line, even if a bunch of them end up on top of each other.


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2019:00:05/27/19', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><pandas>', 'Title': 'Robust solution to plot large datasets with custom line formats using loops', 'ViewCount': 32, 'id': 56328158, 'user_AboutMe': None, 'user_AccountId': 3263053.0, 'user_DisplayName': 'LabGecko', 'user_Reputation': 182.0, 'user_Views': 44.0, 'user_id': 2748201.0}}