How to return 0 for record count when nothing returns for that hour using datepart?
You can use a table constructor of all hours (0-23) and join to that from your DATEPART function.

DB FIDDLE

SELECT 
   h.h as DatePartHour, 
   --DATEPART(hour,TempDateTime) AS TempDatePart,
   COUNT(DISTINCT TempNumber) AS RecordCount
FROM #Temp
   full join (values(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),(11),(12),(13),(14),(15),(16),(17),(18),(19),(20),(21),(22),(23),(0)) h(h)
   on h.h = DATEPART(hour,TempDateTime)
GROUP BY  h.h