The second bomb, which dropped 70 years ago during World War II, hit the Herefordshire home of his father, Ernest Hursey, who was in charge of security at the Rotherwas factory at the time.
Mr Hursey said it was "fate" he was in the back of the house in the early hours because the family had visitors and the front bedrooms were taken.
He said: "While [time has] healed, it hasn't really stopped the picture of the aircraft coming over and dropping its bombs.
"What's in the bomb bounced out of the factory area along the ground into our house... It will always remain very prominent in my mind."
Thousands of women worked at the Royal Ordnance Factory during the two world wars, said Herefordshire Lore, which publishes local memories in a newsletter and online.
It was believed 17 people died in the factory in the incident on 27 July 1942, the group added.
Mr Hursey said: "About 6 o'clock in the morning - a lovely sunny morning, beautiful - and the sirens went and of course in Hereford nobody took any notice of the sirens.
"I saw the bomb bays were open and two bombs dropped out.
"One landed sort of in the middle of a row of sheds and exploded and the other one went into the end shed nearest our house, which was probably about 200, 300 yards away."
Mr Hursey said he did not know if he was knocked unconscious, but "over a period of time... everything was so quiet".
He added: "I couldn't understand why [there was] no noise, nothing was happening and then eventually I heard somebody running down the road and shouting 'is there anybody about?'
"They came and gradually dug me out of all the debris."
His father Ernest and French mother Bertha - both in their late 50s - died along with his 22-year-old brother Ron, who was on leave from the Army, and his brother Ernest's wife and mother-in-law.
Ernest junior, an RAF pilot, was killed in action in his mid 20s about a year after the Rotherwas tragedy.
The family house was demolished after the Rotherwas factory incident, but Ken Hursey - the youngest brother - "just had a couple of little scratches".
After the bombing the apprentice electrician, who had left school at 14, lived with his uncle, Bob Hursey, and his wife before later joining the Fleet Air Arm and the police in 1949.
Now aged 86, Ken Hursey has been married to Margaret for 67 years and has three children and four grandchildren.
But he clearly remembers his "great" parents who met in France while his father was serving in the Army during World War I.
He said: "We had a wonderful upbringing... I wouldn't have changed anything.
"Something happens and it causes a lot of upset, terrible strife, but you've just got to get on with it and get over it."
Asked if he ever thought about the pilot, Mr Hursey said: "I can't bear any animosity against them. They were only doing their job, the same as my brother who was in the RAF."
Ken Hursey was 16 when two 250kg bombs were dropped on a munitions factory, killing several members of his family.