Create custom endpoints by extending the Resource Base class
The serialization class is the input or output of normalization and denormalization, or in other words, the object that you get when you denormalize data or the object that you normalize on the way out. Drupal 8 core provides one example of this \Drupal\rest\Plugin\rest\EntityResource. If the object is an entity you can use core:

  serialization_class = "Drupal\Core\Entity\Entity",


In order to use your arbitrary non-entity, you would also probably need to implement a Normalizer, but that is possible as well.

If your goal is to create a non-REST API with a custom endpoint that does not correspond to a REST operation (such as login or getProperty), then you should look at the Services module.