zero-or-one to zero-or-one
You could go with your own idea of an asymmetric schema by removing one of the foreign keys from the current set-up, or, to keep things symmetric, you could remove both foreign keys and introduce a junction table with a unique constraint on each reference.

So, it would be like this:

CREATE TABLE dbo.Hazard
(
  HazardId int IDENTITY(1,1) NOT NULL CONSTRAINT PK_Hazard PRIMARY KEY CLUSTERED,
  Details varchar(max) NULL
);

CREATE TABLE dbo.Task
(
  TaskId int IDENTITY(1,1) NOT NULL CONSTRAINT PK_Task PRIMARY KEY CLUSTERED,
  Details varchar(max) NULL,
);

CREATE TABLE dbo.HazardTask
(
  HazardId int NOT NULL
    CONSTRAINT FK_HazardTask_Hazard FOREIGN KEY REFERENCES dbo.Hazard (HazardId)
    CONSTRAINT UQ_HazardTask_Hazard UNIQUE,
  TaskId int NOT NULL
    CONSTRAINT FK_HazardTask_Task FOREIGN KEY REFERENCES dbo.Task (TaskId)
    CONSTRAINT UQ_HazardTask_Task UNIQUE
);


You could additionally declare (HazardId, TaskId) to be the primary key if you need to reference these combinations from another table. For the purpose of keeping the pairs unique, however, the primary key is unnecessary, it is enough that each ID is unique.