What are my locking options for a Merge statement?
You need to give the query processor a more efficient access path to locate StudentTotalMarks records.  As written, the query requires a full scan of the table with a residual predicate [StudentID] = [@StudentId] applied to each row:



The engine takes U (update) locks when reading as a basic defence against a common cause of conversion deadlocks.  This behaviour means the second execution blocks when trying to obtain a U lock on the row already locked with an X (exclusive) lock by the first execution.

The following index provides a better access path, avoiding taking unnecessary U locks:

CREATE UNIQUE INDEX uq1 
ON dbo.StudentTotalMarks (StudentID) 
INCLUDE (StudentMarks);


The query plan now includes a seek operation on StudentID = [@StudentId], so U locks are only requested on target rows:



The index is not required to be UNIQUE to solve the issue at hand (though the INCLUDE is required to make it a covering index for this query).

Making StudentID the PRIMARY KEY of the StudentTotalMarks table would also solve the access path problem (and the apparently redundant Id column could be removed).  You should always enforce alternate keys with a UNIQUE or PRIMARY KEY constraint (and avoid adding meaningless surrogate keys without good reason).