How do I set an entity base field as NOT NULL?
There is no API for that at the moment, every field that is specified as a key happens to be set to not null, but that's IMHO actually a bug and I would not rely on that.

Instead, the only thing you can do is provide a storage schema handler and customize the schema there.

See \Drupal\node\NodeStorageSchema::getSharedTableFieldSchema() for example.