Table clean up taking too long
You have a performance problem, so investigate it like a performance problem. Use a methodology like Waits and Queues. This will reveal what is the performance bottleneck, including any blocking that may occur. Likely culprits are blocking (your query is blocked by... you and did not do anything in the past 6 hours), log growth, just poor performance (lack of indexes, including on the ##temp_table).

As a side note the use of SET ROWCOUNT to restrict deletes is deprecated and should be replaced with usage of TOP keyword.