How to find ranking of textbooks by popularity used in a course?
This question is interesting, unfortunately the question is also too broad.  (Course of an specific school or just the subject? By country or which country? At university, MBA or Doctorate level?, etc). But the way to find a possible ranking would be to see best selling books per topic. For example:


Cambridge:https://www.cambridge.org/ar/academic/collections/classical-studies-btu
Amazon's'Best Sellers in Academic &amp; Commercial Writing Reference':
https://www.amazon.com/Best-Sellers-Books-Academic-Commercial-Writing-Reference/zgbs/books/12017 
ALIBRI: https://www.alibris.com/search/books/subject/Academic


There you can add filters per subject, which do can do also with many other bookselling sites. This however will give you best sellers per subject/topic in wide.

But you could also search for the program's syllabus and the books are listed there. If it's for your current school and the school sells books, just go ask for their most sold books for your research.

However consider that:


A textbook used is not necessarily the one suggested to be bought.
Rather than textbooks, students can use online materials instead.
In some countries, teachers force students to buy certain books
and certain editions (which they receive some revenue from).
Popular for teachers is not necessarily popular for students.
Piracy always adds a level of error int he estimations.