Why can't you add an Identity to an existing colum in SQL Server?
So what is so special about an Identity column that this can't be
easily added in retrospect?

An identity does not insert values on a table by itself, it depends on an INSERT command and the old rows on your table would require an UPDATE. If you try to UPDATE an identity column, you get an error like this:

Msg 8102, Level 16, State 1, Line 8 Cannot update identity column
'ColumnName'.

Every feature has a purpose and limitations. The purpose of the identity is not for updating values.


Why is this not just a single int/bigint flag somewhere that is being
incremented every time you insert a new row?

In fact it is something like that. The CREATE TABLE (Transact-SQL) IDENTITY (Property) says:

SQL Server might cache identity values for performance reasons and
some of the assigned values can be lost during a database failure or
server restart.

So your IDENTITY column has something like a flag that asks SQL Server for one of those values when inserting a new row.
From the Remarks section of the IDENTITY doc you can see the feature has some restrictions and the reason is:

These restrictions are part of the design in order to improve
performance, and because they are acceptable in many common
situations.