Challenges for Libraries in Difficult Economic Times: Evidence from the UK

Michael Jubb firstname.lastname@example.com
Research Information Network
96 Euston RoadNW1 2DBLondonUnited Kingdom

Challenges for Libraries in Difficult Economic Times: Evidence from the UK
iSSN: 1435-5205. P132-151721965818610B8A6BE03CA7B22D8733Elibrarieslibrary usersfinancedigitaluniversity
The last decade has been a period of unprecedented change for university libraries.The rapid growth in numbers of students and staff across the university sector has been accompanied by the move to a substantially digital environment with some fundamental changes in how libraries and their users operate.During this period, most university libraries have seen continued growth in their budgets in real terms, but libraries now face a period in which they will have to cope with continued rapid, perhaps transformational, change, accompanied by reductions in their budgets.This paper is based on evidence gathered and analysed by the Research Information Network on the financial position of university libraries in the UK, and how they are dealing with the prospect of cuts.It highlights two related challenges for libraries: developing and enhancing relationships with their users, and developing new services to maximise their value to users.

Introduction

The last decade has been a period of unprecedented change for university libraries.The rapid growth in numbers of students and staff across the university sector has been accompanied by the move to a substantially digital environment with some fundamental changes in how libraries and their users operate.Further change is on the way, with unpredictable implications for students, academic staff, and for libraries.

As they have responded to new developments over the past decade, and changed their operations, most university libraries have seen continued growth in their budgets in real terms.The next few years are likely to prove much more difficult in financial terms.Libraries therefore face a period in which they will have to cope with continued rapid, perhaps transformational, change, accompanied by reductions in their budgets.

This paper is based on evidence gathered analysed by the Research Information Network on how university libraries in the UK in particular are dealing with this new situation.It highlights two related challenges for libraries.

First, users' demands for content have increased enormously over the past decade, in response to the greatly increased supply of content available to them 24/7.But the demand is unrelated to price, because users -at least in the UK -do not pay; the library does.At the same time, many usersparticularly academics -believe that libraries and the services they provide have become less relevant to their needs.Hence many are less willing than previously to see library budgets sustained at the expense (as they see it) of their own departmental budgets.Disintermediation thus has the power to hit libraries in a number of different ways, and a key challenge is to develop and enhance relationships between libraries and their users.

Second, research libraries in particular face a tension between sustaining their traditional roles of looking after and providing access to scholarly content on the one hand, and providing more content and new services in a digital environment.In what remains a hybrid world, at least some of the traditional roles remain valuable to many users.But developing new services is essential if libraries are to maximise their value to users in a rapidlychanging world.

Budgets and Finance

Between 1998 and 2008, expenditure on all university libraries in the UK grew by 30% in real terms, 1 with even faster growth -at nearly 48% -shown by the libraries of the research-intensive universities represented by Research Libraries UK (RLUK). 2 Numbers of staff and students also increased, however; and income and expenditure on research grew even faster.Hence, as shown in Figure 1, the proportion of total university expenditure that went to support libraries fell: from 3.4% to 2.8% across all UK universities, and from 3.2% to 2.6% across the RLUK libraries.So libraries represent a declining share of university budgets.Over the past two years, however, libraries have begun to experience a much harsher financial climate.A survey for the Charleston Observatory in 2009 showed that libraries across the world were facing cuts.In the US, the growth in library budgets in the years up to 2008 seems to have been somewhat slower than in the UK: statistics from the ARL suggest that its university members saw an increase of 12% in real terms in their budgets between 2001 and 2007.But two-thirds of them experienced cuts in 2008-09 and 2009-10, with a mean of 5% and a maximum of over 20%.The Charleston survey 3 showed a similar picture, indicating that over 45% of US libraries had experienced cuts in 2009, and over 40% were anticipating cuts in future years.

As shown in Figure 2, budget cuts have been rather slower to arrive in the UK, but over 53% of libraries expect cuts in 2010-11.And they are especially gloomy about prospects for the future, with nearly 57% predicting further cuts in 2011-12.In such circumstances, there is renewed attention to the balance of expenditure between staffing, content and services.The proportions of library budgets spent on these three key elements vary hugely across the university sector in the UK, as we shall see, but there is a widespread view that in libraries where staff represent a relatively high proportion of expenditure, that proportion will have to fall.

Searching for Savings

In seeking to address the new economic situation, library directors are planning to make cuts in services, staffing, infrastructure and content provision.

Staffing

Staffing represents over 50% of expenditure overall across all UK university libraries, although there are significant differences between individual libraries -proportions vary between under 30% and over 70% -and groups of libraries.The proportion tends to be highest in smaller colleges and in specialist institutions.Across all UK libraries, staff numbers rose by about 15% between 1998 and 2008, and, as shown in Figure 3, expenditure on staffing rose in real terms even faster, by over 30%, with the sharpest rise -at 45% -in the research-intensive universities.Again, however, it should be noted that the numbers of academic staff and students -the prime users of university libraries -grew almost twice as fast as library staff numbers in this period.In a context where the demands on libraries and their staff have increased, UK university library directors believe that their staffing levels have already been pared down.But senior managers in universities in the UK and the US are planning cuts in staffing, including academic staff, and it is not surprising that they see scope for cuts in libraries too.Hence nearly three-quarters Michael Jubb of UK library directors -a significantly higher proportion than their US counterparts -believe that they will have to make cuts in the numbers of library staff.

But doing more with fewer staff represents a significant challenge for library directors, and many have spoken of the need to think in terms of re-engineering and introducing new staffing structures in order to help in ensuring that they focus resources on those areas that have the biggest impact: speedy delivery of the information resources and services that accurately meet the needs of students and academics.Reductions in staffing levels should thus ideally be based on increased understanding of the relationships between activities, costs and impact, and be combined with more effective performance management so that library staff have a clear view of their roles, along with a willingness to take on new ones and to learn new skills.

Libraries are familiar with the experience of restructuring, for example in merging with IT services.But there are tensions between the desire for fully thought-through re-engineering to deliver more effective and efficient performance on the one hand, and the need to reduce staff numbers quickly in response to budget cuts on the other.Evidence so far is that the favoured methods for achieving cuts in the current crisis is by introducing recruitment freezes and not replacing staff who leave.

Services and Infrastructure

Service levels are of course closely related to staffing, but librarians are understandably reluctant to cut services to staff and students.Responses to the Charleston survey and focus group discussions, however, show that UK library directors across the sector believe that budget cuts at the levels they are being asked to consider cannot be achieved without a significant impact on services.Reductions in opening hours, particularly at weekends and during vacations, are options being considered by many libraries, and some directors wonder whether they can sustain the 24-hour opening that has become common, especially during the exam season.Some libraries are already reducing opening hours, even though this is unwelcome to many students.

Other options being considered include reductions in subject support for academic staff and students, and in information skills training, in addition to cuts in the purchase of books and of journals.And budget cuts many make it difficult to sustain newer developments such as promoting changes in scholarly communications, or supporting the information gathering required for research assessment purposes.It is also clear that some libraries are postponing IT and other projects for the development of their infrastructure of services, along with plans for new building work.

Information Content

Expenditure on information content of all kinds represents about 35% of all library expenditure across the UK university library sector, and that proportion has been relatively stable over the past decade.At a number of universities, there have been policies to protect the budgets for information resources above all else, partly in order to sustain and improve the student experience.Again there are significant differences between individual libraries and groups of libraries, essentially representing the mirror image of the different proportions of their expenditure accounted for by staff.The proportion tends to be highest -at around 37% -in universities established before 1992, although it is slightly lower among the most researchintensive universities.

Michael Jubb

The relatively stable proportion of expenditure on content implies, of course, increases in actual expenditure in real terms.But here experiences differ across the sector, as shown in Figure 5.In the research-intensive universities expenditure rose by 52%, but in the newer universities, after rising by 5% in the years up to 2002, expenditure on content has actually declined in real terms since then, and in 2008 was actually 2% lower than it was in 1998.

Books

Expenditure on books, both printed and electronic, fell from 12% to 9% of overall library expenditure across all UK universities between 1998 and 2008, and actual expenditure fell by 10% in real terms.These falls occurred even though students have a powerful voice in demanding that universities should provide access through their libraries to the books included on course reading lists.US university libraries experienced similar falls, though it should be noted that US students are generally more prepared than their UK counterparts to purchase copies of key texts for themselves.

There is a widespread belief -or hope -that e-books may ease the problems that libraries face in meeting the demands for books -particularly for text-books and course readers, and important experiments are under way in the UK in the form of an e-books observatory project. 4Many librarians express a sense of inevitability about a transition to e-books, and this feeds a view, represented in the Charleston survey and in RIN focus groups, that cuts in their provision of content will fall disproportionately on printed books.

But librarians are also aware of the constraints against a wholesale move to e-books.Many monographs and other books are not yet available electronically; business models and publishers' policies on pricing and accessibility are complex, inconsistent and difficult to comprehend; and despite the arrival of the i-pad and other devices, the technical problems relating to reading long books on screen have not yet been resolved to universal satisfaction.It is thus difficult for libraries to customise collections to meet the needs of their users, and the many different platforms for e-books make it less than straightforward for libraries and users to navigate through the various interfaces.These constraints, coupled with concerns about preservation of scholarly materials in digital form for the long term, have limited the take-up of e-books; and there is also a widespread view that libraries need to work together with publishers to promote innovative thinking on new models and routes to content.

Journals

It is well-known that expenditure on scholarly journals and other serials has risen fast in recent years.Across the UK it rose by 59% between 1998 and 2008, and by 87% in the research-intensive universities.It now represents nearly 19% of overall library expenditure, having risen from just over 15% in 1998, and in many well-established university libraries it accounts for over a quarter of the overall budget.

Big deals have increased enormously the range of titles available to members of universities in all parts of the sector.They have also changed expectations.Readers -academics in particular -now expect to have immediate access to most of the journals in their specialist area, and libraries find that nearly all of the titles to which they now subscribe are read, even if only by small groups of readers.Any suggestion of major cuts in the portfolios of journals to which libraries subscribe thus gives rise to vocal hostility.

Nevertheless, there is a widespread belief that current levels of journal provision may be unsustainable, especially for some smaller institutions and those which are teaching-led.There is intense pressure to reduce costs by abandoning print copies of journals, and cancelling subscriptions to lowly-used titles.Libraries are thus looking intensely at usage data before making decisions on renewals, and in some less-well-funded institutions especially, the prospect is that some big deals will be cancelled.

Libraries are also looking very closely at the costs of the big deals and how they might be reduced, for they view the seemingly inexorable increases in the prices charged by publishers as unsustainable.For UK libraries, most of the deals with the larger publishers are negotiated through regional consortia or centrally through the Joint Information Systems Committee (JISC), under its National Electronic Site Licensing initiative (NESLi2).Individual libraries then decide whether or not to subscribe to the deal.Although the negotiations are conducted by professionals, some members of our focus groups -especially those from teaching-led universities -are concerned that the deals do not represent good value for them, and are considering the possibility of negotiating individually with publishers on their own account.

But there are also concerns about the confidentiality clauses that seek to prevent universities from sharing with each other information about the costs of the deals they reach.Many library directors consider themselves involved in a war of attrition with publishers, and there are some suggestions of concerted action to tackle the lack of transparency about the terms of big deals.Nevertheless, librarians are aware of the risks if they try to break ranks, not least because the UK is a relatively small part of the global market for publishers.In other parts of the world, especially China and India, publishers are experiencing growth in their markets.

Hence only by acting in concert with others both in the UK and overseas do libraries see much chance of achieving significant savings in their deals with publishers.There is thus in the UK increasing interest in the possibility of site licences covering the whole university sector and especially in the pilot initiative to that end in Scotland (SHEDL).Achieving such deals across the sector as a whole will not be straightforward, especially in a financial climate which is likely to intensify opposition from senior managers in universities towards 'top-slicing' of the funds provided to individual universities.Thus the diverse positions and financial prospects of different universities may make it more difficult for the sector as a whole to work together towards common goals, and students and staff in smaller or less-well-resourced institutions may find that they have access to a reduced range of journals.Nevertheless, it is important that libraries should work together and with publishers to find ways of reducing the costs of licence deals and the restrictions on access often associated with them.

New Strategies

Academic libraries in all parts of the sector have developed in the past decade new kinds of activities in supporting research, teaching and learning, and senior librarians are very much aware of the need to sustain the momentum of innovation in developing new approaches and new services to support changes in institutional missions and in the workflows of both students and staff.But the scope for doing so may become increasingly constrained by financial cuts and loss of key staff; sustaining momentum will depend increasingly on co-operation across the sector.

Content

Even the most well-resourced libraries cannot hope to acquire and provide access to all the content that their academics and students might want.Any such aim is especially otiose in an environment where users can find content from multifarious sources on the internet.Selective development of their collections thus remains a fundamental part of the role of academic libraries.But there is increasing interest in moving from a 'just-in-case' to a 'just-in-time' approach, and from library-controlled to user-generated acquisitions, particularly in the light of evidence that the latter circulate at least as widely -and often more so -than the former.

The shift in university libraries away from print and towards digital content is driven by powerful forces including the constraints of space and finance, and above all user demands.In the short term, however, most university libraries in the UK are likely to remain hybrid, with both print and digital content.For despite the progress of digitisation and the increasing preponderance of Michael Jubb born-digital content, we are still far from the position where all the content of interest to students and researchers is available digitally, and many of them are reluctant to abandon print altogether.

In a hybrid world, some libraries are interested in exploring how they might build co-operative, distributed models for library collections -both digital and in print -that are shared across a group of libraries, regionally, nationally, or defined by their subject interests.Low-use print material can thus be shared by users across a number of libraries, and locally managed collections replaced by co-operative agreements.For digital material, there is increasing interest in moving towards co-operative collection-development plans and co-operatively managed digital aggregations, or even 'cloud-sourced' collections.

One potential way of reducing the pressure on acquisition budgets and of enhancing access to content, of course, is through the provision of open access, through either the green or the gold route.Librarians in the UK have played a leading role in promoting open access initiatives, through the establishment of institutional repositories and of arrangements for the payment of open access publication fees.In the very long term, it is possible that open access may help in reducing the pressure on library budgets.But over the next 3-5 years at least, open access initiatives will continue to represent additional burdens on libraries, for the costs of running repositories and promoting their use, or of paying publication fees, are not being offset by any significant reductions in subscription costs for scholarly journals.

Services

The shift from print to digital resources provides libraries with the opportunity to focus more on providing services than on managing collections of content.Many libraries are keen to develop in this way, with new services to help users create, manage and manipulate information.Libraries are thus interested in helping to develop more effective arrangements for managing, curating, sharing and preserving data created or gathered by researchers.Similarly, they have also been active in addressing the increasing concerns about lack of understanding and skills among academics as well as students in handling an ever-more complex information environment.

If they are to develop new services or enhance existing ones, however, libraries must tackle three key challenges.

First, they must reduce if not eliminate what is routine in order to make space for new activities, for it is unlikely that additional resources or funding will be available.Outsourcing of what can be done more efficiently or effectively by others is likely to be part of the answer in areas including cataloguing and the hosting of library websites.Second, they must ensure that users are fully engaged in the development and implementation of new services.Third, they must develop new models of working co-operatively to exploit the resources and expertise of their colleagues in the sector as a whole.A recent project sponsored by SCONUL to develop a business case for shared services for all UK university libraries is a significant example of work of this kind.

Staffing

Increasingly, librarians are being required to take on a variety of roles and activities.The ICT infrastructure for libraries is becoming ever more complex, and libraries are likely to need more computer scientists and informaticians to work along with subject experts.

There is a growing recognition among senior managers in universities as well as information professionals that both academics and students require better support if they are to handle information resources effectively.But effective support depends increasingly on a mixture of subject specialist and information expertise; and there is no consensus on how moves of this kind might be initiated, nor on how professionals of this kind might be organised and where they might be located within institutional structures.

A minority view among senior managers is that libraries and librarians are key to these kinds of developments: that they already understand and have a broad view of the issues which other players lack, and that some of them are already involved in providing relevant services as well as acting as partners in pilot projects.A more widespread view is that while many librarians have talked about developing their contribution to the research effort in these
• • • Michael Jubb
ways, there are only weak linkages between the expertise and the services now needed and the traditional skillsets of librarians.Even those who take a more positive view acknowledge that if librarians are to develop their roles in this way, there will be a huge need for training and capacity-building; and that might involve re-envisioning and rebuilding the role of subject specialist librarians.

The Value of Libraries

There is a strong feeling among senior librarians that they have failed effectively to communicate the value of their services to those who fund and use them.They believe that many senior managers in universities, as well as academics and students, have outdated views and expectations about the services that libraries now provide.There is thus an increasing risk that much of what libraries actually do may be invisible in a virtual environment.This may leave libraries especially vulnerable when institutions have to take difficult financial decisions, with cuts targeted at specific departments or areas of activity.Moreover, while the student voice in support of libraries may be particularly powerful, it tends to be focused on a rather narrow set of issues relating to ready access to course texts and the need for extended opening hours.

In current circumstances it is particularly important that libraries should be able to show not only that they are operating efficiently, but that they provide services with demonstrable links to success in achieving institutional goals.Return on investment is thus an increasingly important issue.It is important therefore that libraries are proactive in seeking to understand user behaviour and workflows; and in rigorously analysing and demonstrating the value of their activities in improving students' experience and in supporting teaching, learning and research.

The focus of performance indicators up to now has tended to be on inputs and outputs that are relatively straightforward to measure, rather than addressing the much harder issues relating to impact and value.Nevertheless, it is essential that more is done to analyse the relationships between library activities on the one hand, and learning and research outcomes on the other.

Work of this kind is in its relatively early stages, and it is fraught with difficulties.Gathering and analysing evidence of value is notoriously difficult.A number of different approaches have been adopted, and there is no single answer.A key question is 'value for whom?'In relation to libraries, approaches to gathering evidence of value for students or academics may well differ from approaches to value for funders or for universities.Similarly, approaches to the value of existing services may not be appropriate in gathering evidence of possible changes either to the nature or to the level (positive or negative) of those services.And there are notorious difficulties presented in assessing changes in value over time.

One simple approach is to analyse expenditure and usage, to ascertain the unit cost per usage and its variation between different resources and institutions, or over time.Analysis of the costs per download of e-journals in UK university libraries shows some intriguing patterns.As shown in Table 1, downloads rose by over 160% between 2004 and 2008, and by 250% in the research-intensive universities.When levels of usage are put alongside expenditure on e-journals in individual universities, again the results are intriguing.They show a very strong correlation between volumes of downloads and expenditure, with only a few outliers, as shown in Figure 6.This finding seems to indicate that universities as a whole are spending their money wisely.But when we look at specific disciplines and institutions, and analyse levels of usage against institutional size, even within a single discipline, we find that the intensity of use for each potential reader varies by a factor of over four, which raises interesting questions for individual libraries.

Trying to assess the impact of usage is much more difficult.For the linkages between use of information resources provided by libraries and research or learning outcomes are difficult to pin down, and chains of reasoning may raise as many questions as they seek to answer.One approach is to try to calculate the return on investment (ROI) for individual libraries.Recent studies led by Carol Tenopir suggest that the ROI varies from between 15.5:1 to under 1:1 (i.e. a negative return), depending on such factors as the balance between teaching and research, and the subject mix, in each university.The high rates of return are for the libraries of research-intensive universities with a focus on science, technology and medicine, where winning research grants and contracts is a key measure of success for the university as a whole.Low or even negative rates of return are for the libraries of teaching-led universities, or those where the focus is on the arts and humanities.

ses 5 show that there is a strong correlation between numbers of downloads of e-journal articles in individual universities and success in research as measured by the number of PhDs awarded, the value of research grants and contracts received, the numbers of papers published, and field-normalised citation impact.

Correlations do not necessarily imply causal relationships, of course.But by using a partial least squares regression and path modelling technique, we have built a model that seeks to predict levels of each of three variablesexpenditure, use, and research outcomes -on the basis of the other two, and to quantify how good they are as predictors of each other.The analysis of 113 UK universities suggests that:

both expenditure on e-journals and success in research are strong predictors of future use of e-journals (indeed, expenditure is a precondition for use); success in research tends to lead to higher levels of expenditure on e-journals in the future, but there is no strong link between expenditure and subsequent research success; the linkage between levels of use and expenditure is not strong: high levels of use do not seem to be a direct driver of high levels of future expenditure on e-journals (perhaps because journals are generally supplied as bundled deals and not priced according to usage); there is a strong feedback loop between levels of usage and research success.

The most powerful linkage by far in this modelling is between levels of usage and research performance: higher usage is a strong predictor of subsequent research success.But the modelling also shows that higher expenditure leads to more usage, and that there are thus good reasons for sustaining current levels of expenditure.

These findings are suggestive rather than conclusive, but it is important that such studies should be continued, so that libraries can sustain the effort to show not only that they are operating efficiently, but that they provide services with demonstrable links to success in achieving institutional goals.Libraries need to be more proactive in seeking to understand user behaviour and workflows; and in rigorously analysing and demonstrating the value of their activities in improving students' experience and in supporting teaching, learning and research.There is a need to go beyond performance indicators that focus on inputs and outputs which are relatively straightforward to measure, and to address the much harder issues relating to impact and value.That implies detailed investigations of the relationships between library activities on the one hand, and learning and research outcomes on the other.

Conclusion

Academic libraries have demonstrated their ability to lead and to adapt their roles in response to changing circumstances.They play a crucial role in supporting the teaching, learning and research missions of their universities.The quality of libraries and their services is an especially important part of the student experience, and students are increasingly vocal about any perceived shortcomings.

Librarians across the sector in the UK recognise that they must respond to the pressures of a new period of financial stringency; and that they must demonstrate that their services are both efficient and effective in delivering value to the staff and students in their universities.They also recognise that they must find ways of doing things differently; many see budget cuts as an opportunity to re-think what the library does and what it means.But they have as yet few concrete proposals that will transform services or yield large-scale efficiency savings, at least within the bounds of individual universities.

In these circumstances, it is important for libraries to share ideas and experiences, and to test what works and what does not.There is a risk that the increasing diversity of the HE sector may inhibit working together to achieve common goals.But it is crucial for libraries to exploit the potential for cooperation in developing a range of shared services in order to enhance efficiency, as well as the scope and quality of what they provide to both academic staff and students.

Libraries need to do all this not least in order to sustain a momentum in the development of new services that meet the changing needs of users and the missions and strategies of their host universities.Co-operation and partnership -with other libraries; with other information service providers; and with the staff, students and senior managers of their host universitieswill be watchwords for libraries as they develop their strategies for the future.Working closely together with, and influencing, partners across the university sector will be essential if libraries are to exploit the current financial difficulties as an opportunity for change.

Notes

1 All figures relating to budgets and expenditure in UK libraries are calculated from SCONUL statistics.See http://vamp.diglib.shrivenham.cranfield.ac.uk/statistics/sconul-annual-library-statistics.

Fig. 1 :
1
Fig. 1: Library expenditure as a percentage of overall institution expenditure 1998-2008.

Fig. 2 :
2

Fig. 3 :
3
Fig. 3: Indexed real terms expenditure on staff 1998-2008.

Fig. 4 :
4
Fig. 4: Information provision expenditure as a percentage of overall library expenditure 1998-2008.

Fig. 5 :
5
Fig. 5: Expenditure on information provision 1998-2008.

Table 1 :
1

Average for sectorYear20042005200620072008Russell Group783,8701,377,6031,846,1212,211,2452,795,825Other Pre-1992 universities439,813632,144665,926819,3351,001,521Post-1992 universities and283,760322,251443,027521,350592,253other institutionsTotal432,693632,758772,600930,4151,134,165Index 2004=100Russell Group100175.7235.5282.1356.7Other Pre-1992 universities100143.7151.4186.3227.7Post-1992 universities and100117.1156.1183.7208.7other institutionsTotal100146.2178.6215.0262.1As shown in Table 2, the rise in usage has been accompanied by a fall incost per use, from £1.19 to £0.70 (thus by 40%) across the sector as a whole,with the biggest fall again among the research-intensive Russell Groupuniversities.

Table 2 :
2

Average for sectorYear20042005200620072008Russell Group£1.73£0.99£0.82£0.74£0.66Other Pre-1992£1.20£0.96£0.98£0.91£0.81universitiesPost-1992 universities and£1.01£0.85£0.73£0.68£0.65other institutionsTotal£1.19£0.91£0.83£0.77£0.70Index 2004=100Russell Group100.057.247.442.838.2Other Pre-1992100.080.081.775.867.5universitiesPost-1992 universities and100.084.272.367.364.4other institutionsTotal100.076.569.764.758.8
ARL statistics suggest that growth was lower in the US, where the budgets of the university library members of ARL grew by 12% in real terms between 2001 and 2007. See http://www.arl.org/stats/annualsurveys/arlstats/statxls.shtml.
The economic downturn and libraries: survey findings, CIBER, University College London, 2009, available at http://www.ucl.ac.uk/infostudies/research/ciber/ charleston-survey.pdf. David Nicholas, et al., 'The impact of the economic downturn on libraries: with special reference to university Libraries', The Journal of Academic Librarianship (2010), doi:10.1016/j.acalib.2010.06.001.
 National e-Books Observatory Project, Final Report, 2009, available at http://www. jiscebooksproject.org/reports/finalreport.
E-journals: their use value and impact, RIN 2009, available at http://www.rin. ac.uk/system/files/attachments/E-journals-report.pdf.
Liber Quarterly Volume 20 Issue 2 2010 The Research Information Network has taken a rather different approach, working with the CIBER team at University College London to look at relationships between levels of usage on the one hand and of different kinds of research outcomes for universities in the UK on the other.These analy-Michael Jubb


Clarifying differences between review designs and methods

David Gough
James Thomas
Sandy Oliver
Clarifying differences between review designs and methods
C O M M E N T A R Y Open AccessAggregation configurationComplex reviewsMappingMethodologyMixed methods reviewsResearch methodsScoping reviewsSynthesisSystematic reviewsTaxonomy of reviews
This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews. Terminology is necessary for planning, describing, appraising, and using reviews, building infrastructure to enable the conduct and use of reviews, and for further developing review methodology. There is insufficient consensus on terminology for a typology of reviews to be produced and any such attempt is likely to be limited by the overlapping nature of the dimensions along which reviews vary. It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation. Three such main dimensions are proposed: (1) aims and approaches (including what the review is aiming to achieve, the theoretical and ideological assumptions, and the use of theory and logics of aggregation and configuration in synthesis); (2) structure and components (including the number and type of mapping and synthesis components and how they relate); and (3) breadth and depth and the extent of 'work done' in addressing a research issue (including the breadth of review questions, the detail with which they are addressed, and the amount the review progresses a research agenda). This then provides an overarching strategy to encompass more detailed descriptions of methodology and may lead in time to a more overarching system of terminology for systematic reviews.

Background

Research studies vary in many ways including the types of research questions they are asking, the reasons these questions are being asked, the theoretical and ideological perspectives underlying these questions, and in the research methods that they employ. Systematic reviews are a form of research; they are (and the theoretical and ideological perspectives underlying these methods) a way of bringing together what is known from the research literature using explicit and accountable methods [1]. Systematic methods of review have been successfully developed particularly for questions concerning the impact of interventions; these synthesize the findings of studies which use experimental controlled designs. Yet the logic of systematic methods for reviewing the literature can be applied to all areas of research; therefore there can be as much variation in systematic reviews as is found in primary research [2,3]. This paper discusses some of the important conceptual and practical differences between different types of systematic review. It does not aim to provide an overall taxonomy of all types of reviews; the rate of development of new approaches to reviewing is too fast and the overlap of approaches too great for that to be helpful. Instead, the paper argues that, for the present at least, it is more useful to identify the key dimensions on which reviews differ and to examine the multitude of different combinations of those dimensions. The paper also does not aim to describe all of the myriad actual and potential differences between reviews; this would be a task too large even for a book let alone a paper. The focus instead is on three major types of dimensions of difference. The first dimension is the aims and approaches of reviews; particularly in terms of their methodologies (their ontological and epistemological foundations and methods of synthesis). The second dimension is the structure and components of reviews. The third dimension is the breadth, depth, and extent of the work done by a review in engaging with a research issue. Once these three aspects of a review are clear, consideration can be given to more specific methodological issues such as methods of searching, identifying, coding, appraising, and synthesizing evidence. The aim of this paper is to clarify some of the major conceptual distinctions between reviews to assist the selection, evaluation, and development of methods for reviewing.

Clarifying the nature of variation in reviews As forms of research, systematic reviews are undertaken according to explicit methods. The term 'systematic' distinguishes them from reviews undertaken without clear and accountable methods.

The history of systematic reviews is relatively recent [4,5] and despite early work on meta-ethnography [6], the field has been dominated by the development and application of statistical meta-analysis of controlled trials to synthesize the evidence on the effectiveness of health and social interventions. Over the past 10 years, other methods for reviewing have been developed. Some of these methods aim to extend effectiveness reviews with information from qualitative studies [7]. The qualitative information may be used to inform decisions made in the statistical synthesis or be part of a mixed methods synthesis (discussed later). Other approaches have been developed from a perspective which, instead of the statistical aggregation of data from controlled trials, emphasize the central role that theory can play in synthesizing existing research [8,9], address the complexity of interventions [10], and the importance of understanding research within its social and paradigmatic context [11]. The growth in methods has not been accompanied by a clear typology of reviews. The result is a complex web of terminology [2,12].

The lack of clarity about the range of methods of review has consequences which can limit their development and subsequent use. Knowledge or consensus about the details of specific methods may be lacking, creating the danger of the over-generalization or inappropriate application of the terminology being used. Also, the branding of different types of review can lead to over-generalizations and simplification with assumptions being made about differences between reviews that only apply to particular stages of a review or that are matters of degree rather than absolute differences. For example, concepts of quality assurance can differ depending upon the nature of the research question being asked. Similarly, infrastructure systems developed to enable the better reporting and critical appraisal of reviews, such as PRISMA [13], and for registration of reviews, such as PROSPERO [14] currently apply predominantly to a subset of reviews, the defining criteria of which may not be fully clear.

A further problem is that systematic reviews have attracted criticism on the assumption that systematic reviewing is applicable only to empirical quantitative research [15]. In this way, polarized debates about the utility and relevance of different research paradigms may further complicate terminological issues and conceptual understandings about how reviews actually differ from one another. All of these difficulties are heightened because review methods are undergoing a period of rapid development and so the methods being described are often being updated and refined.

Knowledge about the nature and strengths of different forms of review is necessary for: appropriate choice of review methods by those undertaking reviews; consideration of the importance of different issues of quality and relevance for each stage of a review; appropriate and accurate reporting and accountability of such review methods; interpretation of reviews; commissioning of reviews; development of procedures for assessing and undertaking reviews; and development of new methods.

Clarifying the nature of the similarities and differences between reviews is a first step to avoiding these potential limitations. A typology of review methods might be a solution. There are many diverse approaches to reviews that can be easily distinguished, such as statistical metaanalysis and meta-ethnography. A more detailed examination, however, reveals that the types of review currently described often have commonalities that vary across types of review and at different stages of a review. Three of these dimensions are described here. Exploring these dimensions also reveals how reviews differ in degree along these overlapping dimensions rather than falling into clear categories.

Review aims and approaches

Primary research and research reviews vary in their ontological, epistemological, ideological, and theoretical stance, their research paradigm, and the issues that they aim to address. In reviews, this variation occurs in both the method of review and the type of primary research that they consider. As reviews will include primary studies that address the focus of the review question, it is not surprising that review methods also tend to reflect many of the approaches, assumptions, and methodological challenges of the primary research that they include.

One indication of the aim and approach of a study is the research question which the study aims to answer. Questions commonly addressed by systematic reviews include: what is the effect of this intervention (addressed by, for example, the statistical meta-analysis of experimental trials); what is the accuracy of this diagnostic tool (addressed by, for example, meta-analysis of evaluations of diagnostic tests); what is the cost of this intervention (addressed by, for example, a synthesis of cost-benefit analyses); what is the meaning or process of a phenomena (addressed by, for example, conceptual synthesis such as meta-ethnography or a critical interpretative synthesis of ethnographic studies); what is the effect of this complex intervention (addressed by, for example, multi-component mixed methods reviews); what is the effect of this approach to social policy in this context (addressed by, for example, realist synthesis of evidence of efficacy and relevance across different policy areas); and what are the attributes of this intervention or activity (addressed by, for example, framework synthesis framed by dimensions explicitly linked to particular perspectives).

Although different questions drive the review process and suggest different methods for reviewing (and methods of studies included) there is considerable overlap in the review methods that people may select to answer these questions; thus the review question alone does not provide a complete basis for generating a typology of review methods.

Role of theory

There is no agreed typology of research questions in the health and social sciences. In the absence of such a typology, one way to distinguish research is in the extent that it is concerned with generating, exploring, or testing theory [16].

In addressing an impact question using statistical metaanalysis, the approach is predominantly the empirical testing of a theory that the intervention works. The theory being tested may be based on a detailed theory of change (logic model) or be a 'black box' where the mechanisms by which change may be affected are not articulated. The review may, in addition to testing theory, include methods to generate hypotheses about causal relations. Testing often (though not always) wants to add up or aggregate data from large representative samples to obtain a more precise estimate of effect. In the context of such reviews, searching aims to identify a representative sample of studies, usually by attempting to include all relevant studies in order to avoid bias from study selection (sometimes called 'exhaustive' searching). Theoretical work in such analyses is undertaken predominantly before and after the review, not during the review, and is concerned with developing the hypothesis and interpreting the findings.

In research examining processes or meanings the approach is predominantly about developing or exploring theory. This may not require representative samples of studies (as in aggregative reviews) but does require variation to enable new conceptual understandings to be generated. Searching for studies in these reviews adopts a theoretical approach to searching to identify a sufficient and appropriate range of studies either through a rolling sampling of studies according to a framework that is developed inductively from the emerging literature (akin to theoretical sampling in primary research) [17]; or through a sampling framework based on an existing body of literature (akin to purposive sampling in primary research) [18]. In both primary research and reviews, theoretical work is undertaken during the process of the research; and, just as with the theory testing reviews, the nature of the concepts may be relatively simple or very complex.

Aggregative and configurative reviews

The distinction between research that tests and research that generates theory also equates to the distinction between review types made by Voils, Sandelowski and colleagues [19,20] (although we have been very influenced by these authors the detail of our use of these terms may differ in places). Reviews that are collecting empirical data to describe and test predefined concepts can be thought of as using an 'aggregative' logic. The primary research and reviews are adding up (aggregating) and averaging empirical observations to make empirical statements (within predefined conceptual positions). In contrast, reviews that are trying to interpret and understand the world are interpreting and arranging (configuring) information and are developing concepts ( Figure 1). This heuristic also maps onto the way that the review is intended to inform knowledge. Aggregative research tends to be about seeking evidence to inform decisions whilst configuring research is seeking concepts to provide enlightenment through new ways of understanding.

Aggregative reviews are often concerned with using predefined concepts and then testing these using predefined (a priori) methods. Configuring reviews can be more exploratory and, although the basic methodology is determined (or at least assumed) in advance, specific methods are sometimes adapted and selected (iteratively) as the research proceeds. Aggregative reviews are likely to be combining similar forms of data and so be interested in the homogeneity of studies. Configurative reviews are more likely to be interested in identifying patterns provided by heterogeneity [12]. The logic of aggregation relies on identifying studies that support one another and so give the reviewer greater certainty about the magnitude and variance of the phenomenon under investigation. As already discussed in the previous section, the approach to searching for studies to include (the search strategy) is attempting to be exhaustive or, if not exhaustive, then at least avoiding bias in the way that studies are found. Configuring reviews have the different purpose of aiming to find sufficient cases to explore patterns and so are not necessarily attempting to be exhaustive in their searching. (Most reviews contain elements of both aggregation and configuration and so some may require an unbiased set of studies as well as sufficient heterogeneity to permit the exploration of differences between them).

Aggregating and configuring reviews also vary in their approach to quality assurance. All reviews aim to avoid drawing misleading conclusions because of problems in the studies they contain. Aggregative reviews are concerned with a priori methods and their quality assurance processes assess compliance with those methods. As the basis of quality assurance is known a priori, many aspects of this can be incorporated into the inclusion criteria of the review and then can be further checked at a later quality assurance stage. The inclusion criteria may, for example, require only certain types of study with specific methodological features. There is less consensus in the practice of quality assessment in configurative reviews; some adopt a similar strategy to those employed in aggregative reviews, whereas others reject the idea that the quality of a study can be assessed through an examination of its method, and instead prioritize other issues, such as relevance to the review and the contribution the study can make in the review synthesis to testing or generating theory [21][22][23]. Some of the differences between aggregating and configuring reviews are shown in Figure 1.

Although the logics of aggregating and configuring research findings demand different methods for reviewing, a review often includes components of both. A metaanalysis may contain a post hoc interpretation of statistical associations which may be configured to generate hypotheses for future testing. A configurative synthesis may include some components where data are aggregated (for example, framework synthesis) [24,25]. Examples of reviews that are predominantly aggregative, configurative, or with high degrees of both aggregation and configuring are given in Table 1 (and for a slightly different take on this heuristic see Sandelowski et al. [20]).

Similarly, the nature of a review question, the assumptions underlying the question (or conceptual framework), and whether the review aggregates or configures the results of other studies may strongly suggest which methods of review are appropriate, but this is not always the case. Several methods of review are applicable to a wide range of review approaches. Both thematic [26] and framework synthesis [24,25] which identify themes within narrative data can, for example, be used with both aggregative and configurative approaches to synthesis.

Reviews that are predominantly aggregative may have similar epistemological and methodological assumptions to much quantitative research and there may be similar assumptions between predominantly configurative reviews and qualitative research. However, the quantitative/qualitative distinction is not precise and does not reflect the differences in the aggregative and configurative research processes; quantitative reviews may use configurative processes and qualitative reviews can use aggregative processes. Some authors also use the terms conceptual synthesis for reviews that are predominantly configurative, but the process of configuring in a review does not have to be limited to concepts; it can also be the arrangement of numbers (as in subgroup analyses of statistical meta-analysis). The term 'interpretative synthesis' is also used to describe reviews where meanings are interpreted from the included studies. However, aggregative reviews also include interpretation, before inspection of the studies to develop criteria for including studies, and after synthesis of the findings to develop implications for policy, practice, and further research. Thus, the aggregate/configure framework cannot be thought of as another way of expressing the qualitative/ quantitative 'divide'; it has a more specific meaning Cost benefit How effective is the benefit of an intervention relative to its cost?

Prevalence How extensive is this condition?

Configurative

Meta-ethnography [4] What theories can be generated from the conceptual literature?

Critical interpretative synthesis [8] What theories can be generated from the conceptual literature?

Meta narrative review [11] How to understand the development of research on an issue within and across different research traditions?

Configuring and aggregative

Realist synthesis [9] What is the effect of a social policy in different policy areas?

Framework synthesis [25] What are the attributes of an intervention or activity?

concerning the logic of synthesis, and many reviews have elements of both aggregation and configuration.

Further ideological and theoretical assumptions

In addition to the above is a range of issues about whose questions are being asked and the implicit ideological and theoretical assumptions driving both them and the review itself. These assumptions determine the specific choices made in operationalizing the review question and thus determine the manner in which the review is undertaken, including the research studies included and how they are analyzed. Ensuring that these assumptions are transparent is therefore important both for the execution of the review and for accountability. Reviews may be undertaken to inform decision-making by non-academic users of research such as policymakers, practitioners, and other members of the public and so there may be a wide range of different perspectives that can inform a review [27,28]. The perspectives driving the review will also influence the findings of the review and thereby clarify what is known and not known (within those perspectives) and thus inform what further primary research is required. Both reviewer and user perspectives can thus have an ongoing influence in developing user-led research agendas. There may be many different agendas and thus a plurality of both primary research and reviews of research on any given issue. A further fundamental issue that is related to the types of questions being asked and the ideological and theoretical assumptions underlying them is the ontological and epistemological position taken by the reviewers. Aggregative reviews tend to assume that there is (often within disciplinary specifications/boundaries) a reality about which empirical statements can be made even if this reality is socially constructed (generalizations); in other words they take a 'realist' philosophical position (a broader concept than the specific method of 'realist synthesis'). Some configurative reviews may not require such realist assumptions. They take a more relativist idealist position; the interest is not in seeking a single 'correct' answer but in examining the variation and complexity of different conceptualizations [12,29]. These philosophical differences can be important in understanding the approach taken by different reviewers just as they are in understanding variation in approach (and debates about research methods) in primary research. These differences also relate to how reviews are used. Aggregative reviews are often used to make empirical statements (within agreed conceptual perspectives) to inform decision making instrumentally whilst configuring reviews are often used to develop concepts and enlightenment [30].

Structure and components of reviews

As well as varying in their questions, aims, and philosophical approach, reviews also vary in their structure. They can be single reviews that synthesize a specific literature to answer the review question. They may be maps of what research has been undertaken that are products in their own right and also a stage on the way to one or more syntheses. Reviews can also contain multiple components equating to conducting many reviews or to reviewing many reviews.

Systematic maps

To some degree, most reviews describe the studies they contain and thus provide a map or account of the research field. Some reviews go further than this and more explicitly identify aspects of the studies that help describe the research field in some detail; the focus and extent of such description varying with the aims of the map. Maps are useful products in their own right but can also be used to inform the process of synthesis and the interpretation of the synthesis [3,30]. Instead of automatically undertaking a synthesis of all included studies, an analysis of the map may lead to a decision to synthesize only a subset of studies, or to conduct several syntheses in different areas of the one map. A broader initial review question and a narrower subsequent review question allows the synthesis of a narrower subset of studies to be understood within the wider literature described in terms of research topics, primary research methods, or both. It also allows broader review questions to create a map for a series of reviews ( Figure 2) or mixed methods reviews (Figure 3). In sum, maps have three main purposes of: (i) describing the nature of a research field; (ii) to inform the conduct of a synthesis; and (iii) to interpret the findings of a synthesis [3,31]. The term 'scoping review' is also sometimes used in a number of different ways to describe (often non-systematic) maps and/or syntheses that rapidly examine the nature of the literature on a topic area [32,33]; sometimes as part of the planning for a systematic review.

Broad review question

Mixed methods reviews

The inclusion criteria of a review may allow all types of primary research or only studies with specific methods that are considered most appropriate to best address the review question. Including several different methods of primary research in a review can create challenges in the synthesis stage. For example, a review asking about the impact of some life experience may examine both randomized controlled trials and large data sets on naturally occurring phenomena (such as in large scale cohort studies). Another strategy is to have sub-reviews that ask questions about different aspects of an issue and which are likely to consider different primary research [34,35]. For example, a statistical meta-analysis of impact studies compared with a conceptual synthesis of people's views of the issue being evaluated [34,35]. The two sub-reviews can then be combined and contrasted in a third synthesis as in Figure 3. Mixed methods reviews have many similarities with mixed methods in primary research and there are therefore numerous ways in which the products of different synthesis methods may be combined [35]. Mixed knowledge reviews use a similar approach but combine data from previous research with other forms of data; for example a survey of practice knowledge about an issue (Figure 4).

Another example of a mixed methods review is realist synthesis [9] that examines the usefulness of mid-level policy interventions across different areas of social policy by unpacking the implicit models of change, followed by an iterative process of identifying and analyzing the evidence in support of each part of that model. This is quite similar to a theory-driven aggregative review (or series of reviews) that aggregatively test different parts of a causal model. The first part of the process is a form of configuration in clarifying the nature of the theory and what needs to be empirically tested; the second part is the aggregative testing of those subcomponents of the theory. The difference between this method and more 'standard' systematic review methods is that the search for empirical evidence is more of an iterative, investigative process of tracking down and interpreting evidence. Realist synthesis will also consider a broad range of empirical evidence and will assess its value in terms of its contribution rather than according to some preset criteria. The approach therefore differs from the predominantly a priori strategy used in either standard 'black box' or in theory driven aggregative reviews. There have also been attempts to combine aggregative 'what works' reviews with realist reviews [36]. These innovations are exploring how best to develop the breadth, generalizability and policy relevance of aggregative reviews without losing their methodological protection against bias.

There are also reviews that use other pre-existing reviews as their source of data. These reviews of reviews may draw on the data of previous reviews either by using the findings of previous reviews or by drilling down to using data from the primary studies in the reviews [37]. Information drawn from many reviews can also be mined to understand more about a research field or research methods in meta-epidemiology [38]. As reviews of reviews and meta-epidemiology both use reviews as their data, they are sometimes both described as types of 'meta reviews'. This terminology may not be helpful as it links together two approaches to reviews which have little in common apart from the shared type of data source. A further term is 'meta evaluation'. This

Broad review question

Subsynthesis 2 can refer to the formative or summative evaluation of primary evaluation studies or can be a summative statement of the findings of evaluations which is a form of aggregative review (See Gough et al. in preparation,and [39]).

Mixed synthesis

Breadth, depth, and 'work done' by reviews Primary research studies and reviews may be read as isolated products yet they are usually one step in larger or longer-term research enterprises. A research study usually addresses a macro research issue and a specific focused sub-issue that is addressed by its specific data and analysis [16]. This specific focus can be broad or narrow in scope and deep or not so deep in the detail in which it is examined.

Breadth of question

Many single component aggregative reviews aim for homogeneity in the focus and method of included studies. They select narrowly defined review questions to ensure a narrow methodological focus of research findings. Although well justified, these decisions may lead to each review providing a very narrow view of both research and the issue that is being addressed. A user of such reviews may need to take account of multiple narrow reviews in order to help them determine the most appropriate course of action.

The need for a broader view is raised by complex questions. One example is assessing the impact of complex interventions. There are often many variants of an intervention, but even within one particular highly specified intervention there may be variations in terms of the frequency, duration, degree, engagement, and fidelity of delivery [40]. All of this variation may result in different effects on different participants in different contexts. The variation may also impact differentially within the hypothesized program theory of how the intervention impacts on different causal pathways. Reviews therefore need a strategy for how they can engage with this complexity. One strategy is to achieve breadth through multi-component reviews; for example, a broad map which can provide the context for interpreting a narrower synthesis, a series of related reviews, or mixed methods reviews. Other strategies include 'mega reviews' , where the results from very many primary studies or meta-analyses are aggregated statistically (for example, [41,42]) and multivariate analyses, where moderator variables are used to identify the 'active ingredients' of interventions (for example, [43,44]). Whether breadth is achieved within a single review, from a sequence of reviews, from reviews of reviews, or from relating to the primary and review work of others, the cycle of primary research production and synthesis is part of a wider circle of engagement and response to users of research [45].

Review resources and breadth and depth of review

The resources required for a systematic review are not fixed. With different amounts of resource one can achieve different types of review. Broad reviews such as mixed methods and other multi-component reviews are likely to require more resources, all else being constant, than narrow single method reviews. Thus, in addition to the breadth of review is the issue of its depth, or the detail with which it is undertaken. A broad review may not have greater resources than a narrow review in which case those resources are spread more thinly and each aspect of that breadth may be undertaken with less depth.

When time and other resources are very restricted then a rapid review may be undertaken where some aspect of the review will be limited; for example, breadth of review question, sources searched, data coded, quality and relevance assurance measures, and depth of analysis [46,47]. Many students, for example, undertake literature reviews that may be informed by systematic review principles of rigor and transparency of reporting; some of these maybe relatively modest exercises whilst others make up a substantial component of the thesis. If rigor of execution and reporting are reduced too far then it may be more appropriate to characterize the work as non systematic scoping than as a systematic review.

Reviews thus vary in the extent that they engage with a research issue. The enterprise may range in size from, for instance, a specific program theory to a whole field of research. The enterprise may be under study by one research team, by a broader group such as a review group in an international collaboration or be the focus of study by many researchers internationally. The enterprises may be led academic disciplines, applied review collaborations, by priority setting agendas, and by forums to enable different perspectives to be engaged in research agendas. Whatever the nature of the strategic content or process of these macro research issues, reviews vary in the extent that they plan to contribute to such more macro questions. Reviews thus vary in the extent that this research work is done within a review; rather than before and after a review (by primary studies or by other reviews).

Reviews can be undertaken with different levels of skill, efficiency, and automated tools [48] and so resources do not equate exactly with the 'work done' in progressing a research issue. In general, a broad review with relatively little depth (providing a systematic overview) may be comparable in work done to a detailed narrow review (as in many current statistical metaanalyses). A multi-component review addressing complex questions using both aggregative and configuring methods may be attempting to achieve more work, though there may be challenges in terms of maintaining consistency or transparency of detail in each component of the review. In contrast, a rapid review has few resources and so is attempting less than other reviews but there may be dangers that the limited scope (and limited contribution to the broader research agenda) is not understood by funders and users of the review. How best to use available resources is a strategic issue depending upon the nature of the review question, the state of the research available on that issue and the knowledge about that state of the research. It is an issue of being fit for purpose. A review doing comparatively little 'work' may be exactly what is needed in one situation but not in another.

Conclusion

Explicit accountable methods are required for primary research and reviews of research. This logic applies to all research questions and thus multiple methods for reviews of research are required, just as they are required for primary research. These differences in types of reviews reflect the richness of primary research not only in the range of variation but also in the philosophical and methodological challenges that they pose including the mixing of different types of methods. The dominance of one form of review question and review method and the branding of some other forms of review does not clearly describe the variation in review designs and methods and the similarities and differences between these methods. Clarity about the dimensions along which reviews vary provides a way to develop review methods further and to make critical judgments necessary for the commission, production, evaluation, and use of reviews. This paper has argued for the need for clarity in describing the design and methods of systematic reviews along many dimensions; and that particularly useful dimensions for planning, describing, and evaluating reviews are:

1. Review aims and approach: (i) approach of the review: ontological, epistemological, theoretical, and ideological assumptions of the reviewers and users of the review including any theoretical mode; (ii) review question: the type of answer that is being sought (and the type of information that would answer it); and (ii) aggregation and configuration: the relative use of these logics and strategies in the different review components (and the positioning of theory in the review process, the degree of homogeneity of data, and the iteration of review method). 2. Structure and components of reviews: (iv) the systematic map and synthesis components of the review; and (v) the relation between these components. 3. Breadth, depth, and 'work done' by reviews: (vi) macro research strategy: the positioning of the review (and resources and the work aimed to be done) within the state of what is already known and other research planned by the review team and others; and (vii) the resources used to achieve this.

Clarifying some of the main dimensions along which reviews vary can provide a framework within which description of more detailed aspects of methodology can occur; for example, the specific strategies used for searching, identifying, coding, and synthesizing evidence and the use of specific methods and techniques ranging from review management software to text mining to statistical and narrative methods of analysis. Such clearer descriptions may lead in time to a more overarching system of terminology for systematic reviews.

Figure 3 Figure 4
34A Mixed knowledge review.

Table 1
1Examples of review typesPredominant review type Review questions

Aggregative

'What works?' reviews
What is the effect of a health or social
intervention?

Diagnostic test
What is the accuracy of this diagnostic
tool?

http://www.systematicreviewsjournal.com/content/1/1/28
AcknowledgementsThe authors wish to acknowledge the support and intellectual contribution of their previous and current colleagues at the EPPI-Centre. They also wish to acknowledge the support of their major funders as many of the ideas in this paper were developed whilst working on research supported by their grants; this includes the Economic and Social Research Council, the Department of Health, and the Department for Education. The views expressed here are those of the authors and are not necessarily those of our funders.Competing interestsThe authors declare that they have no competing interests.Authors' contributionsAll three authors have made substantial contributions to the conception of the ideas in this paper, have been involved in drafting or revising it critically for important intellectual content, and have given final approval of the version to be published.
The Handbook of Research Synthesis. H Cooper, L Hedges, New York: Russell Sage FoundationCooper H, Hedges L: The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994.

Dimensions of difference in evidence reviews (Overview; I. Questions, evidence and methods; II.Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal. D Gough, Gough D: Dimensions of difference in evidence reviews (Overview; I. Questions, evidence and methods; II.Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal;

Series of six posters presented at National Centre for Research Methods meeting. Manchester. LondonEPPI-CentreV. Communication, interpretation and application. Series of six posters presented at National Centre for Research Methods meeting, Manchester. London: EPPI-Centre; January 2007. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1919

Commonality and diversity in reviews. D Gough, J ; Thomas, D Gough, Oliver S , Thomas J London, Introduction to Systematic Reviews. 2012Gough D, Thomas J: Commonality and diversity in reviews. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:35-65.

A brief history of research synthesis. I Chalmers, L Hedges, H Cooper, Eval Health Professions. 25Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Professions 2002, 25:12-37.

Formalising syntheses of medical knowledge: the rise of meta-analysis and systematic reviews. I Bohlin, Perspect Sci in press. in pressBohlin I: Formalising syntheses of medical knowledge: the rise of meta-analysis and systematic reviews. Perspect Sci in press, in press.

Hare RD: Meta-ethnography: synthesizing qualitative studies. G Noblit, Sage PublicationsNewbury Park NYNoblit G: Hare RD: Meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

In Cochrane Handbook for Systematic Reviews of Interventions. J Noyes, J Popay, A Pearson, K Hannes, A Booth, Higgins JPT, Green S. Version 5.1.0The Cochrane CollaborationNoyes J, Popay J, Pearson A, Hannes K, Booth A: Qualitative research and Cochrane reviews. In Cochrane Handbook for Systematic Reviews of Interventions. Edited by Higgins JPT, Green S. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. www.cochrane-handbook.org.

Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. M Dixon-Woods, D Cavers, S Agarwal, E Annandale, A Arthur, J Harvey, R Hsu, S Katbamna, R Olsen, L Smith, R Riley, A J Sutton, BMC Med Res Methodol. 635Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol 2006, 6:35.

Evidenced-based policy: a realist perspective. R Pawson, London: SagePawson R: Evidenced-based policy: a realist perspective. London: Sage; 2006.

Can we systematically review studies that evaluate complex interventions. S Shepperd, S Lewin, S Struas, M Clarke, M Eccles, R Fitzpatrick, G Wong, A Sheikh, PLoS Med. 68Shepperd S, Lewin S, Struas S, Clarke M, Eccles M, Fitzpatrick R, Wong G, Sheikh A: Can we systematically review studies that evaluate complex interventions? PLoS Med 2009, 6:8.

Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. T Greenhalgh, G Robert, F Macfarlane, P Bate, O Kyriakidou, R Peacock, Soc Sci Med. 61Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med 2005, 61:417-430.

Methods for the synthesis of qualitative research: a critical review. E Barnett-Page, J Thomas, BMC Med Res Methodol. 959Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol 2009, 9:59.

The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. D Moher, A Liberati, J Tetzlaff, D G Altman, PLoS Med. 66Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med 2009, 6:6.

PLoS Medicine Editors: Best practice in systematic reviews: The importance of protocols and registration. PLoS Med. 82PLoS Medicine Editors: Best practice in systematic reviews: The importance of protocols and registration. PLoS Med 2011, 8:2.

Introduction: evidence and practice. In Evidence-based Practice in Education. G Thomas, Pring R, Thomas G. BuckinghamOpen University PressThomas G: Introduction: evidence and practice. In Evidence-based Practice in Education. Edited by Pring R, Thomas G. Buckingham: Open University Press; 2004:44-62.

Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. London: Teaching and Learning Research Programme. D Gough, S Oliver, M Newman, K Bird, Gough D, Oliver S, Newman M, Bird K: Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. London: Teaching and Learning Research Programme; 2009.

A Strauss, J Corbin, Basics of qualitative research, grounded theory procedures and techniques. London: SageStrauss A, Corbin J: Basics of qualitative research, grounded theory procedures and techniques. London: Sage; 1990.

Qualitative Data Analysis. London: Sage. M Miles, A Huberman, Miles M, Huberman A: Qualitative Data Analysis. London: Sage; 1994.

Making sense of qualitative and quantitative findings in mixed research synthesis studies. C I Voils, M Sandelowski, J Barroso, V Hasselblad, Field Methods. 20Voils CI, Sandelowski M, Barroso J, Hasselblad V: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods 2008, 20:3-25.

Mapping the Mixed Methods-Mixed Research Synthesis Terrain. M Sandelowski, C J Voils, J Leeman, J L Crandlee, 10.1177/1558689811427913Journal of Mixed Methods Research. Sandelowski M, Voils CJ, Leeman J, Crandlee JL: Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research 2011, doi:10.1177/1558689811427913.

Types and Quality of Knowledge in Social Care. London: Social Care Institute for Excellence. R Pawson, A Boaz, L Grayson, A Long, C Barnes, Pawson R, Boaz A, Grayson L, Long A, Barnes C: Types and Quality of Knowledge in Social Care. London: Social Care Institute for Excellence; 2003.

Expressions of excellence and the assessment of applied and practice-based research. A Oancea, J Furlong, Res Pap Educ. 22Oancea A, Furlong J: Expressions of excellence and the assessment of applied and practice-based research. Res Pap Educ 2007, 22:119-137.

In Introduction to Systematic Reviews. A Harden, D Gough, Gough D, Oliver S, Thomas J. London: Sage2012Quality and relevance appraisalHarden A, Gough D: Quality and relevance appraisal. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:153-178.

Methods for the thematic synthesis of qualitative research in systematic reviews. J Thomas, A Harden, BMC Med Res Methodol. 845Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008, 8:45.

A multidimensional conceptual framework for analyzing public involvement in health services research. S Oliver, R W Rees, Clarke-Jones , L Milne, R Oakley, A R Gabbay, J Stein, K Buchanan, P Gyte, G , Heal Expect. 11Oliver S, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, Stein K, Buchanan P, Gyte G: A multidimensional conceptual framework for analyzing public involvement in health services research. Heal Expect 2008, 11:72-84.

A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. C Carroll, A Booth, K Cooper, BMC Med Res Methodol. 1129Carroll C, Booth A, Cooper K: A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol 2011, 11:29.

Stakeholder perspectives and participation in reviews. R Rees, S Oliver, Introduction to Systematic Reviews. Gough D, Oliver S, Thomas J. London: Sage2012Rees R, Oliver S: Stakeholder perspectives and participation in reviews. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:17-34.

Getting started with a review. S Oliver, K Dickson, M Newman, Introduction to Systematic Reviews. Gough D, Oliver S, Thomas J. London: Sage2012Oliver S, Dickson K, Newman M: Getting started with a review. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:66-82.

Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. London: Government Chief Social Researcher's Office. L Spencer, J Ritchie, J Lewis, L Dillon, Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. London: Government Chief Social Researcher's Office; 2003.

The many meanings of research utilisation. C Weiss, Public Adm Rev. 29Weiss C: The many meanings of research utilisation. Public Adm Rev 1979, 29:426-431.

A Descriptive Mapping of Health Promotion Studies in Young People EPPI Research Report. G Peersman, London: EPI-CentrePeersman G: A Descriptive Mapping of Health Promotion Studies in Young People EPPI Research Report. London: EPI-Centre; 1996.

Scoping Studies: towards a methodological framework. H Arksey, O Malley, L , Int J Soc Res Methodol. 8Arksey H, O'Malley L: Scoping Studies: towards a methodological framework. Int J Soc Res Methodol 2005, 8:19-32.

Scoping studies: advancing the methodology. D Levac, H Colquhoun, O 'brien, K K , Implement Sci. 569Levac D, Colquhoun H, O'Brien KK: Scoping studies: advancing the methodology. Implement Sci 2010, 5:69.

Integrating qualitative research with trials in systematic reviews: an example from public health. J Thomas, A Harden, A Oakley, S Oliver, K Sutcliffe, R Rees, G Brunton, J Kavanagh, Brit Med J. 328Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. Brit Med J 2004, 328:1010-1012.

Mixed methods and systematic reviews: examples and emerging issues. A Harden, J Thomas, Handbook of Mixed Methods in the Social and Behavioral Sciences. Tashakkori A, Teddlie C. London: Sage20102nd editionHarden A, Thomas J: Mixed methods and systematic reviews: examples and emerging issues. In Handbook of Mixed Methods in the Social and Behavioral Sciences. 2nd edition. Edited by Tashakkori A, Teddlie C. London: Sage; 2010:749-774.

Nijssen Combining campbell standards and the realist evaluation approach: the best of two worlds. M Leontien, Van Der Knaap, F L Leeuw, S Bogaerts, T J Laura, J Eval. 29Leontien M, van der Knaap, Leeuw FL, Bogaerts S, Laura TJ: Nijssen Combining campbell standards and the realist evaluation approach: the best of two worlds?. J Eval 2008, 29:48-57.

Methodology in conducting a systematic review of systematic reviews of healthcare interventions. V Smith, D Devane, C M Begley, M Clarke, BMC Med Res Methodol. 1115Smith V, Devane D, Begley CM, Clarke M: Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol 2011, 11:15.

RCTs for policy interventions: a review of reviews and meta-regression. S Oliver, A M Bagnall, J Thomas, J Shepherd, A Sowden, I White, J Dinnes, R Rees, J Colquitt, K Oliver, Z Garrett, Health Technol Assess. 1416Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z: RCTs for policy interventions: a review of reviews and meta-regression. Health Technol Assess 2010, 14:16.

An introduction to meta-evaluation. M Scriven, 2Educational Products ReportScriven M: An introduction to meta-evaluation. Educational Products Report 1969, 2:36-38.

A conceptual framework for implementation fidelity. C Carroll, M Patterson, S Wood, A Booth, J Rick, S Balain, Implement Sci. 240Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S: A conceptual framework for implementation fidelity. Implement Sci 2007, 2:40.

Meta-analysis of psychotherapy outcome studies. M L Smith, G V Glass, Am Psychol. 32Smith ML, Glass GV: Meta-analysis of psychotherapy outcome studies. Am Psychol 1977, 32:752-760.

Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. J Hattie, London: RoutledgeHattie J: Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. London: Routledge; 2008.

Mosteller F: Meta-analysis for Explanation: A Casebook. T D Cook, H Cooper, D S Cordray, H Hartmann, L V Hedges, R J Light, T A Louis, New York: Russell Sage FoundationCook TD, Cooper H, Cordray DS, Hartmann H, Hedges LV, Light RJ, Louis TA, Mosteller F: Meta-analysis for Explanation: A Casebook. New York: Russell Sage Foundation; 1992.

Explaining heterogeneity in meta-analysis: a comparison of methods. S G Thompson, S J Sharp, Stat Med. 18Thompson SG, Sharp SJ: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med 1999, 18:2693-2708.

Making a difference with systematic reviews. In Introduction to Systematic Reviews. R Stewart, S Oliver, Gough D, Oliver S, Thomas J. London: Sage2012Stewart R, Oliver S: Making a difference with systematic reviews. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:227-244.

Government Social Research Unit: Rapid Evidence Assessment Toolkit. Government Social Research Unit: Rapid Evidence Assessment Toolkit. 2008. http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid- evidence-assessment

Issues in conducting and disseminating brief reviews of evidence. P C Abrami, E Borokhovski, R M Bernard, C A Wade, R Tamim, T Persson, M A Surkes, Evidence & Policy: A Journal of Research, Debate and Practice. 6Abrami PC, Borokhovski E, Bernard RM, Wade CA, Tamim R, Persson T, Surkes MA: Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice 2010, 6:371-389.

Information management in reviews. J Brunton, J Thomas, Introduction to Systematic Reviews. Gough D, Oliver S, Thomas J. London: Sage2012Brunton J, Thomas J: Information management in reviews. In Introduction to Systematic Reviews. Edited by Gough D, Oliver S, Thomas J. London: Sage; 2012:83-106.

Eppi-Centre, 10.1186/2046-4053-1-28Evidence for Policy and Practice Information and Coordinating Centre. Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre): http://eppi.ioe.ac.uk doi:10.1186/2046-4053-1-28

Clarifying differences between review designs and methods. Gough, Systematic Reviews. 28Cite this article asCite this article as: Gough et al.: Clarifying differences between review designs and methods. Systematic Reviews 2012 1:28.