A former police officer has revealed how the authorities have known for more than eight years the vast scale on which media organisations employed private detectives to obtain the personal information of thousands of individuals, including the families and friends of murder victims.

The Independent has conducted a detailed examination of the files seized as part of Operation Motorman in 2003, and has been told by the lead investigator on that inquiry that his team was forbidden from interviewing journalists who were paying for criminal records checks, vehicle registration searches, and other illegal practices.

Among the targets of these searches were the victims of some of the most notorious crimes and tragedies of the past 15 years. Many of the investigations were perfectly legal, but many others, it is clear, were well outside the law.

The Motorman files reveal that the Sunday Express used private investigators to obtain the private telephone number of the parents of Holly Wells, shortly after she was murdered in Soham by Ian Huntley. In a statement last night, Express Newspapers said it "has never instructed private investigators to obtain information illegally. We have always and will continue to uphold the highest level of journalistic standards".

The parents of the murdered schoolgirl Sarah Payne were targeted by the same investigator, who was hired by two national newspaper groups – News International and Trinity Mirror – and separately by a celebrity magazine, Best, which is owned by the National Magazine Company. The same agency was also used by the News of the World to target the parents of Milly Dowler, and by The People and NOTW to obtain private numbers for the family of Stuart Lubbock, whose body was found in Michael Barrymore's swimming pool. The People used similar tactics to target the families of children who were victims of the Dunblane massacre.

Operation Motorman was set up by the Information Commissioner's Office (ICO) to look into widespread breaches of data protection laws by the media. In a signed witness statement given to The Independent, Motorman's original lead investigator, a retired police inspector with 30 years' experience, accuses the authorities of serious failings, and of being too "frightened" to question journalists.

"I feel the investigation should have been conducted a lot more vigorously, a lot more thoroughly and it may have revealed a lot more information," he said. "I was disappointed and somewhat disillusioned with the senior management because I felt as though they were burying their heads in the sand. It was like being on an ostrich farm."

He claimed that had investigators been allowed to interview journalists at the time, the phone-hacking scandal and other serious breaches of privacy by the media may have been uncovered years earlier. "The biggest question that needed answering was, why did the reporters want all these numbers and what were they doing with them?" His comments reflect badly on the ICO, and the Press Complaints Commission, which was given early notification of the evidence in the Motorman files. "We weren't allowed to talk to journalists," he said. "It was fear – they were frightened."

The PCC said last night that it had never been given sight of the Motorman evidence but had strengthened its code and issued industry guidelines which had led to an improvement in standards. All the information has been in the hands of the authorities since 2003, when a team from the ICO seized the material from the home of private detective Stephen Whittamore. Whittamore and three other members of his private investigation network were given conditional discharges when Motorman came to court in 2005. No journalists were charged, although the files contain prima facie evidence of thousands of criminal offences. Thousands of victims disclosed in the paperwork have never even been told they were targeted.

News International spent £193 and then a further £105, hiring Whittamore's company JJ Services to carry out investigations into "Sarah Payne", a few months after her murder in 2000. The People also paid for the ex-directory number of Sarah's family home in Surrey. Whittamore was engaged by Best magazine to obtain the same number. At the same time Best, which is now owned by Hearst Magazines UK, asked for three more ex-directory searches relating to Pam Warren, a survivor of the Paddington rail crash of 1999, who was so disfigured she had to wear a face mask. Hearst declined to comment. The families of Dunblane massacre victims Aimie Adam and Matthew Birnie also appear as subjects in the Motorman files, following requests for ex-directory numbers by The People.

When a major tragedy occurred, Whittamore was often the first person that tabloid newsrooms would call. NOTW spent more than £200 using him to locate the parents and other relatives of the murdered schoolgirl Milly Dowler, whose mobile phone was later hacked by Glenn Mulcaire.

John Whittingdale, chairman of the Culture, Media and Sport Committee, said: "There was an absolute lack of any wish on the part of the police or the ICO or those looking into it to start delving into the prosecution of newspapers and journalists. [The ICO] took a list of hundreds and hundreds of journalists' names. Yes, there's a public interest defence but they didn't even bother to go and ask whether that was what they employed Whittamore for."

The Operation Motorman investigator, who has requested anonymity, has written to Lord Leveson asking to give evidence to his inquiry into media standards. The inquiry has expressed interest in him giving oral evidence or submitting a witness statement. He has also been interviewed by Strathclyde Police, which is investigating criminal activity by journalists in Scotland. The Whittamore files have also recently been requested by the Metropolitan Police's Operation Tuleta team, which is investigating the use of computer hacking by journalists.

Many searches will have been carried out legitimately, but the files show the grand scale on which newspapers were using private detectives to gain access to the police national computer and the records of the DVLA in order to obtain details of criminal records and vehicle registrations. Such offences could carry a jail sentence for encouraging a police officer or DVLA employee to commit misconduct in public office. JJ Services was hired to "blag" personal information (by impersonating individuals or officials) from organisations ranging from hospitals to hotels, gyms and banks. Blags are not always illegal – for instance, when information is freely volunteered by an individual without reference to a database.

Newspapers and magazines also used the agency to illicitly obtain thousands of private telephone numbers, often including the details held by telephone companies under the category "Friends and Family".

In total, the Whittamore files reference 17,489 orders from media organisations. Some 1,028 are in the so-called "blue book", which was essentially dedicated to News International. The "red book" contained 6,774 jobs, most on behalf of Trinity Mirror titles. The "green book", which includes work from Associated Newspapers titles, Express Newspapers and some celebrity magazines, has 2,227 references. And the "yellow book", which is miscellaneous, has 7,460 orders.

In 2006, the Information Commissioner's Office published a report What Price Privacy?, giving some details of what it had discovered. The ICO did not identify victims and, in a follow-up report, printed a league table of titles that had used Whittamore's service, showing a total of 3,757 transactions.

The senior investigator described the report as "very inaccurate", citing the apparent under-reporting.

Christopher Graham, the Information Commissioner, said the ICO stood by its reports and that the placing of these documents before Parliament "was a far more effective method of raising awareness of the illegal trade in personal information" than attempting to prosecute journalists. "The ICO has always been clear that our decision not to pursue legal action against any of the journalists linked to the Operation Motorman investigation was based on a lack of evidence that the journalists who had received information from Mr Whittamore had directly asked him to obtain the information illegally. Without this evidence the ICO could not justify chasing every possible prosecution as this would have taken a disproportionate amount of time and resource and was unlikely to lead to any meaningful results."

The ICO said the lower figure in its report was a result of grouping multiple requests by a journalist as a single transaction.

The most alarming inquiries in the Motorman files – and those which would appear to be among the least justifiable in the public interest – are those which involve intrusions into the privacy of victims of serious crime. In 2003, following a drive-by murder in Birmingham, a reporter on The People employed Whittamore to carry out a series of ex-directory checks and other searches on the relatives and associates of the victims, Charlene Ellis and Letisha Shakespeare. He obtained an ex-directory number for the parents of Charlene and sent a total bill to the paper for £355.50.

Other tragedies were subjects of searches. When Stuart Lubbock was found drowned in the swimming pool of Michael Barrymore, NOTW paid Whittamore for the ex-directory number of Claire Wicks, his girlfriend and mother of their daughters. The People made similar inquiries for the private number of the dead man's father, Terry.

Blags were Whittamore's speciality. Charging £100 a time, he repeatedly posed as someone else to obtain information from organisations. Among the targets were the Guide Association, the Royal National Orthopaedic Hospital, the Bel Air Beverly Hills hotel and the investment bank Goldman Sachs.

Some of the tasks performed by Whittamore for newspapers are entirely legal and a number of others have been carried out to obtain information in the public interest, such as in exposing corruption and other criminal activity. In some cases, editors may not have been aware their reporters were engaging private detectives.

Some subjects were political figures. Anji Hunter, former adviser to Tony Blair, came under close scrutiny following her break-up with the landscape gardener Nick Cornwall. The Daily Mail asked Whittamore to obtain the address of Hunter and Cornwall. It then asked for the "Family and Friends" numbers listed by the couple, through which Whittamore discovered the address of Cornwall's parents. The reporter then requested the "Family and Friends" numbers of the parents – 15 numbers, many of them in Liverpool – and did a similar exercise on one of those to produce a further 10 numbers. Whittamore and his network of associates typically obtained such numbers by blagging them from staff at BT. There is no evidence that the Daily Mail engaged in phone hacking, and no article actually appeared. Associated Newspapers, which owns the Daily Mail, said last night that it had banned the use of private investigators in 2006.

Much of the work in the files targets celebrities or appears to be salacious gossip that has no clear public interest justification. The files contain scores of invoices paid to JJ Services, and among those paid by News International are jobs which he records in his files under such descriptions as "Love Rat Mum", "Sex in Unusual Places", "Emma's Sexy Secrets" and "Bonking Tory". News International declined to comment.

A spokesman for Trinity Mirror said: "Since the publication of the Information Commissioner's What Price Privacy Now? report it has been widely known that a number of media organisations, including some of our titles, used the services of Steve Whittamore. We have not used Steve Whittamore since that report. We are engaging fully with the Leveson Inquiry as we will with any other inquiries from appropriate regulatory or legal authorities."

Only a tiny proportion of victims have been told they were targeted. The ICO investigator and his senior colleague interviewed a small sample, including Ian Hislop, Lenny Henry, Hugh Grant, Chris Tarrant and Charlotte Church.

There are around 400 named journalists in the files, from investigative reporters and newsdesk executives to showbiz hacks and diary writers. For some, Whittamore's services were not just a useful tool but almost an addiction. One reporter used him 422 times. Another carried out 191 transactions, requesting dozens of vehicle searches, more than a dozen criminal records checks, several blags and numerous Friends and Family inquiries. Yet the Motorman team were told not to speak to any journalist.

The whistleblower's story: They were too afraid of the press to let us interview journalists

It was incredible. Even as we were doing the search I could see how big it was and that night when we retired to the hotel I spent about four hours browsing through it and the more I browsed the more apparent it became how big it was going to be.

We were down there for two or three days. We came back and the first thing I did was arrange an informal meeting.

When we enlightened them with what we'd found I was subsequently told, within a few days, that we [the investigations unit] weren't allowed to talk to journalists and that he [the Information Commissioner] would deal with the press. It was fear, they were frightened.

We told them what our plan of action was. We intended to put together 30 or 40 prosecution packages and then go for conspiracy, which would involve the blaggers, the private detectives, the corrupt sellers of the information, right up to the journalists.

When I mentioned the press, I still remember the words which one of them said: "We can't take them on, they're too big for us."

I remember thinking, "It's our job to take them on and if we can't take them on, who does take them on?" As an ex-police officer for 30 years and a detective, there's nothing worse than having a damned good case and somebody tells you, "You can't go and interview the suspect".

If you don't ask questions you don't get answers. I feel that had we been given the opportunity to interview some of the reporters we might have got a hint about this hacking because it was totally unknown at that time.

The biggest question that needed answering was why did the reporters want all these numbers and what were they doing with them?

I knew about phone-tapping but I knew there were complicated issues involved in phone tapping so we dismissed that. Had we been given the opportunity to investigate more thoroughly and interview journalists it may well have identified that phone-hacking existed – instead of waiting for the Mulcaire case to break. If we had identified this in 2003 then perhaps a lot of this would never have happened.

I was not present at the Whittamore court case but I was told that the first thing the judge said was a comment about not seeing any journalists in the dock.

If newspapers and reporters had seen the ICO going into their premises or arranging to interview journalists I think it would have sent a lot stronger message out than publishing a report 13 months later.

I feel the investigation should have been conducted a lot more vigorously, a lot more thoroughly, and it may have revealed a lot more information. I know it's difficult to tell all the victims but isn't that the Information Commissioner's job?

There are thousands of people out there who still don't know they've been victims. When I was in the police I was always taught that your victim is your most important person.

I was disappointed and somewhat disillusioned with the senior management because I felt as though they were burying their heads in the sand. It was like being on an ostrich farm.

The impression being given is that they never prosecuted the press because they were so disappointed with the [Whittamore] result of a conditional discharge.

But by then it was too late and, in any case, we knew virtually from the start of that inquiry that no journalist was ever going to get prosecuted.

To be honest it made a bit of a farce of the investigation.

Were these methods illegal?

The transactions which are contained in Stephen Whittamore's files range from area and occupancy searches to criminal records checks and inquiries into vehicle registrations.

Area and occupancy searches

Area and address occupancy searches may have been procured illicitly but would provide no prospect of a prosecution, even when there was no public interest defence in requesting the information. This is because the requester of the information could claim an expectation that the investigator would acquire the details through legitimate means, such as by consulting an electoral roll in a public library, for instance.

Ex-directory checks, phone 'conversions' and friends and family searches

This would be information obtained from a phone company and might be in breach of Section 55 of the Data Protection Act (which came into effect in March 2000 and carries a maximum fine of £5,000) unless Whittamore had obtained them from a friend or relative. These searches could be legally defended if the inquiry was in the public interest.

Searches of the police national computer or the DVLA database