What do Contextual Filters filter against?
Generally speaking, a view is used to show data for a list of entities. View filters are used to select (or filter) which entities are handled by the view; contextual filters are used to select the entities basing on the values passed in the URL.
For example, the Files view that comes with Drupal 8 has two pages: one whose path is /admin/content/files and one whose path is /admin/content/files/usage/%.



The second page shows data (the entity label and the entity type for the entity containing a reference to the file; the module managing the file relationship; the number of times the file is used by the entity) for the file whose ID is passed in the view path.



When I visit /admin/content/files/usage/3 on the test site I created, the view shows the following data.



On /admin/content/files/usage/2, the view shows the following data.



Using as another example a default view that is already enabled on /admin/structure/views, the Taxonomy term view has a page shown on /taxonomy/term/% which uses the Content: Has taxonomy term ID contextual filter.



It shows all the nodes using the taxonomy term whose ID is passed in the URL. In the test site I am using, /taxonomy/term/1 shows all the nodes using the taxonomy term whose ID is 1 (the test taxonomy term), while /taxonomy/term/2 shows all the nodes using the taxonomy term whose ID is 2 (the view taxonomy term).







Contextual filters are still filters, as they select (or filter) the entities from which the data is taken. The difference is that their values are passed from the view URL, not from a form like the filters set in the Filter criteria section.
The number of entities from which the data is taken and what entities are handled by the contextual filter is hard-coded in the class implementing the contextual filter: Content: Has taxonomy term ID selects all the nodes using the taxonomy term whose ID is equal to the value passed in the URL, which could be zero or more than one node, while Content: ID selects only the node whose ID is passed in the URL; User: Created selects all the user accounts that have been created at the time passed in the URL.
The category of the contextual filter (another value set by the the class implementing the contextual filter), which is shown in the contextual filter name, helps to understand which entities a contextual filter selects: Content: Has taxonomy term ID and Content: ID select nodes, while User: Created selects users.