The Times Educational Supplement (TES) says measures to guard against grade inflation were not appropriate for this year's English exams.
Teaching unions have accused Ofqual of "regulatory failure" and say it is more evidence of flawed exam marking.
But the regulator insists it applied its measures correctly.
It comes as state and private school heads call for an independent inquiry into the problems.
Last week Ofqual said it felt the way this year's English GCSE exams were graded was fair, despite the  grade boundaries being moved significantly part-way through the year.
Teachers have complained that pupils achieving exactly the same marks would have received different grades depending on what time of year they sat the exam.
And many who were predicted a grade C, obtained a grade D in their summer exams.
Ofqual found that June's grade boundaries were correct, while January's were "too lenient".
A key document sent to exam boards, which is on the Ofqual website, sets out how it expects exam boards to use a process known as "comparable outcomes" to ensure that grade standards are maintained between GCSEs year-on-year.
It sets out five conditions for when such processes should be used to alter the way papers are marked.
According to the TES report, this year's English GCSE and the circumstances around it failed to meet four of the conditions.
The first of Ofqual's five conditions was that the cohort - or pupils from that year- in a subject "must be similar, in terms of ability, to those of previous years".
But last week's report on English GCSEs noted that the "attainment profile" for 2012 dropped, as grammar and independent school pupils were replaced with extra, lower-achieving comprehensive pupils.
The second condition was that the qualification must be "fit for purpose".
On Monday, Education Secretary Michael Gove told Parliament that this year's English GCSE was "not fit for purpose".
The third condition was that the "nature of the qualification" must be the same.
Ofqual's report last week stated that "these qualifications are different from previous English qualifications in a number of ways".
Finally, the regulator said in the letter that comparable outcomes must only be applied where "previous grades were appropriate".
Ofqual's report said that the English GCSE grades in January were "too generous".
The only condition that appears to have been met is that teaching standards have remained largely consistent.
The guidelines also state: "Students should not be advantaged or disadvantaged simply because they were the first to sit a new set of examinations."
The watchdog has interpreted its own conditions differently, and is insisting that the comparable outcomes approach that prevented grades from rising in June was correctly implemented.
But Russell Hobby, general secretary of the NAHT heads' union, said: "Ofqual doesn't seem to have applied the concept of comparable outcomes properly.
"In this instance there is a regulatory failure. It is a big mess."
And Brian Lightman, general secretary of the Association of School and College Leaders (ASCL) said this appeared to be "another piece of evidence that procedures applied to this exam was not fit for purpose".
Meanwhile, Ofqual said: "The comparable outcomes approach was used correctly for GCSE English this year and we stand by this decision.
"Comparable outcomes are not intended to produce identical results year on year.
"What's important is that outcomes can be compared when taking into consideration a range of factors which may affect the results. More details on the use of this approach can be found in our report."
But Dr Christopher Ray, chairman of the Head Masters' and Headmistresses' Conference (HMC), said the issue was evidence of a deeper problem with Ofqual.
"The verdict of Ofqual's initial report and the reasoning to support it fall well short of answering the questions raised in the minds of schools and pupils."
Shadow education secretary Stephen Twigg said: "It is clear that pupils, parents and education professionals, across the spectrum of schools, feel that pupils have been done a disservice.
"Whilst the Education Secretary Michael Gove says he is 'saddened' by the injustice that has been served to thousands of pupils, he is showing how out-of-touch he is with pupil opinion by refusing to take action. Labour supports calls for an independent inquiry to get to the bottom of this mess."
The Commons Education Committee is set to start an inquiry into the GCSE grading problems next week.
England exams regulator Ofqual breached its own rules in allowing controversial changes to the way English GCSEs were graded this summer, it is claimed.