Where can I set cardinality for a custom field?
Yes, you can limit a custom field to single value by defining a cardinality in the annotations of the field type.

I don't know if you can find this in the docs, but you can use the core comment field as example:

/**
 * Plugin implementation of the 'comment' field type.
 *
 * @FieldType(
 *   id = "comment",
 *   label = @Translation("Comments"),
 *   description = @Translation("This field manages configuration and presentation of comments on an entity."),
 *   list_class = "\Drupal\comment\CommentFieldItemList",
 *   default_widget = "comment_default",
 *   default_formatter = "comment_default",
 *   cardinality = 1,
 * )
 */
class CommentItem extends FieldItemBase implements CommentItemInterface {


When you add a comment field to a content type the cardinality form element is hidden and instead this message is displayed:


  Allowed number of values 
  
  This field cardinality is set to 1 and cannot be configured.