How to add security questions in Drupal 8?
Well at this time you have 2 modules but only for Drupal 7:

Security Questions


  Security Questions provides administrator configurable challenge
  questions for use during the log in and password reset processes.
  Think of it like most bank website logins.


Password reset


  The password_reset module allows for passwords to be reset without
  involving e-mail addresses through the use of security questions.


There is not a 8.x branch at this time, so your best option is study the code and migrate one of them.