How do I create a bundle under my custom content entity type in Drupal 8?
Yes, you declare a config entity type to store the bundle definitions. For example that for Custom Block Types (bundle of block_content):

Drupal\block_content\Entity\BlockContentType

namespace Drupal\block_content\Entity;

use Drupal\Core\Config\Entity\ConfigEntityBundleBase;
use Drupal\block_content\BlockContentTypeInterface;

/**
 * Defines the custom block type entity.
 *
 * @ConfigEntityType(
 *   id = "block_content_type",
 *   label = @Translation("Custom block type"),
 *   handlers = {
 *     "form" = {
 *       "default" = "Drupal\block_content\BlockContentTypeForm",
 *       "add" = "Drupal\block_content\BlockContentTypeForm",
 *       "edit" = "Drupal\block_content\BlockContentTypeForm",
 *       "delete" = "Drupal\block_content\Form\BlockContentTypeDeleteForm"
 *     },
 *     "list_builder" = "Drupal\block_content\BlockContentTypeListBuilder"
 *   },
 *   admin_permission = "administer blocks",
 *   config_prefix = "type",
 *   bundle_of = "block_content",
 *   entity_keys = {
 *     "id" = "id",
 *     "label" = "label"
 *   },
 *   links = {
 *     "delete-form" = "/admin/structure/block/block-content/manage/{block_content_type}/delete",
 *     "edit-form" = "/admin/structure/block/block-content/manage/{block_content_type}",
 *     "collection" = "/admin/structure/block/block-content/types",
 *   },
 *     ...
 * )
 */
class BlockContentType extends ConfigEntityBundleBase implements BlockContentTypeInterface {


Which is then referenced in the content entity type:

Drupal\block_content\Entity\BlockContent

*   bundle_entity_type = "block_content_type",


If you use Drupal Console to generate a custom content entity type and you say yes to the question if you want bundles, this config entity type is set up for you automatically.