Challenges with Key with Gaps?
What you see is completely normal and expected.


  Note: Because smallserial, serial and bigserial are implemented using sequences, there may be "holes" or gaps in the sequence of
  values which appears in the column, even if no rows are ever deleted.
  A value allocated from the sequence is still "used up" even if a row
  containing that value is never successfully inserted into the table
  column. This may happen, for example, if the inserting transaction
  rolls back.


Unless you really need a gapless series of values, you don't have to worry about it.  This is how most (well over 99%, I guess) database tables work.

If you need it, be careful if you have concurrent inserts on the table - you will need some locking to exclude the possibility of choosing the same value by two or more sessions running in parallel.