Poverty in Poland and the United States: A Comparison of Key Characteristics, Composition, and Prospects
5 May 2016

Bozena Leven
Economics Department
School of Business
The College of New Jersey
EwingUSA

Poverty in Poland and the United States: A Comparison of Key Characteristics, Composition, and Prospects
5 May 201644C789660AEA09B46B7A9D91A912540310.4236/me.2016.75057Received 20 February 2016; accepted 2 May 2016;Measurement and Analysis of Poverty, Comparative Studies of Poverty
This study provides a comparative approach to assess the state of Polish poverty.To do so, poverty in Poland and the United States is presented in terms of selected general, shared characteristics and particularly vulnerable societal groups.Using official, primarily income based, statistics for Poland and the United States, we conclude that a higher percentage of the overall population in the United States is poor and, relative to median country-specific incomes, that the US poverty is deeper than in Poland.In addition, more children, the elderly, and the disabled are poor in America than in Poland; the breadth of poverty is wider in the United States and, perhaps most importantly, may in some important ways also be more intractable.

Introduction

While poverty exists in all the countries, its extent, depth and characteristics differ greatly.Comparative studies of poverty are difficult because poverty in Africa, Europe or the United States poses very different realities for those living there.Comparing poverty is also challenging as individual countries rely on differing statistical methodologies and data on poverty are notoriously problematic.While the characteristics and causes of poverty are diverse and often country specific, however, there are some basic commonalities generally shared by the poor across countries, as they tend to belong to similar societal groups.

In this study we broadly describe the characteristics of poverty in Poland and compare it to poverty in the United States.Analyzing Polish poverty is a relatively new endeavor, because prior to its transition in 1990 to a market economy, Poland under socialism was "poverty free" according to official state sources.Pre-transition, the State provided cradle to grave assistance, taking care of all basic needs, including health care, education, and subsidized housing and day care.Full employment was a key economic objective of the State and all prices (including food and energy) were set for prolonged periods of time.Consequently, there was no open unemployment or inflation and no need for welfare or social benefits.And, since all wages were controlled by the State, Poland experienced relatively equal income distribution, though relevant data for that period are not always reliable.

As the transition to a market economy progressed, liberalization and privatization reforms led to unemployment, inflation, income polarization and rising poverty.Rising income inequalities were exacerbated as Poland's economy grew and private ownership expanded, leading to wealth accumulation and newly established foreign, as well as private, firms, which often paid premium wages for selected skills.Rapid growth and development in the late 1990s and early 2000, combined with an accelerated integration of technology, created opportunities for selected workers, while leaving multiple societal groups with a sense of real and/or relative deprivation.

Significantly, rising inequality and poverty in Poland were a cornerstone of the recent 2015 presidential elections, and will likely serve as a metric against which the current government will be assessed in the short and medium-term by Poland's electorate.Poles, who feel marginalized, either in absolute or relative terms voted for politicians who spoke to this sense of marginalization.Today's Poland is sharply divided between those who benefited from transition and those who were left behind, as is clearly reflected in a deep, persistent policy divergence between the country's two main political parties.Considering Poland's poverty relative to one of the wealthiest countries in the world may broaden and enrich the perspectives of those now focused on this issue.

The choice of the US for developing a different perspective on Poland's poverty is dictated by several reasons.To begin with, most Poles believe that the United States is the world's richest country and enjoys the highest standard of living (despite often contrary statistical data) and know little about its economic problems.These misperceptions are based, in part, on historical reasons.For years, Poles were exposed to a steady stream of propaganda critical of America from a widely distrusted State media propped up by Poland's eastern neighbor.At the same time, American entertainment outlets presented a picture of the United States as a nation of abundance.The resulting positive image of the United States was reinforced by the power of the dollar on the domestic black currency market in the pre-transition period.Savings maintained in dollars at that time permitted those Poles who had accumulated even small amounts to substantially increase their standard of living.

Today, this same wealth perception is continuously reinforced by the media, as American made entertainment still dominates Polish television and cinema.And this image is unlikely to change in the near future, as expansive, stringent American visa requirements-in marked contrast with the open borders within the EU-actively hinder many Poles from experiencing a different reality by visiting or working in the United States.

Comparing Poland to the US in any economic aspect is inherently problematic.Recognizing the major differences in size, population and levels of development in the US and Poland, we rely exclusively on relative poverty measures for each country in our analysis.A better understanding of how poverty in the United States and Poland differ, and where they are similar, may help inform how Polish society evaluates its own problems and its current anti-poverty policies.Since some of data used are based on census or surveys for comparability purposes most of the recent data used cover years 2010-2013.

The Literature on Poverty-Review

There are numerous definitions of poverty, focusing on absolute and relative measures.Absolute poverty measures poverty in relation to the amount of money necessary to meet basic needs such as food, clothing, and shelter and is not concerned with broader quality of life or income inequality [1].Relative poverty defines poverty in relation to the economic wellbeing of other members of the society: people are poor if they fall below selected standards in a country [2].Both concepts, however, are largely concerned with income and consumption.

In addition to problems in defining poverty, literature on poverty often includes an analysis of the methodological differences and related problems embedded in the relevant statistical data.Several major international organizations that collect primary poverty data have devoted substantial resources to developing unique methods for assessing poverty.The most extensive poverty studies originate from various UN departments, the World Bank, the OECD, and other international organizations that research and combat poverty.For example, The Annual Human Development Report, funded as part of the United Nations Development Program, is a major source of international poverty data and analysis that has frequently revised its poverty measures to address numerous methodological problems.

While measuring poverty in any single country is challenging, cross country comparisons are even more problematic since they require some degree of compatibility of the indicators used.When selecting poverty indicators for cross country comparisons, the literature focuses on a proper selection of poverty thresholds for individual country, which then determine and define the extent and depth of poverty.Use of such thresholds alone is problematic as described by Atkinson [3], due to differences in the purchasing power of incomes across countries, and the common failure to adequately account for non-cash, state provided subsidies/benefits, such as health care, education and housing, or other benefits available in different countries.Thus, cross country poverty analysis often involves individual country measures beyond income-based indicators.Unfortunately, since these benefits differ by country, Smeeding [4], they add to the complexitiy of those cross-country studies.Overall, while there is a consensus in the literature that income-based indicators need to be supplemented by other complex measures, there is no consensus on a better multi-dimensional approach, as discussed by Sen [5].

Given those poverty data problems, in addition to average income-based statistics, cross-country comparisons rely also on medians and other percentiles describing income distribution, particularly when averages are less representative due to severe income inequalities.Furthermore, aside from income, another frequently employed approach involves spending indicators Sahn [6], which shed more light on households' ability to finance their basic needs but require use of numerous assumptions about spending patterns and non-cash benefits.Another poverty measure beyond those linked to income incorporates additional data on life expectancy, health, nutrition, and education through the Human Development Index (HDI), which is reported annually for all countries in the Annual Human Development Reports.Because As the HDI index requires preselecting weights for each indicator, however, it is sometimes also considered to be also arbitrary [7].

Given these methodological challenges, a multi-dimensional approach best captures situations in which a country specific monetary and non-monetary benefits are used to assess the depth and extent of poverty [8].In addtion, a number of poverty studies identify societal groups most frequently affected by poverty within specific countries, and then compare them to the same societal groups across countries [9].The societal groups examined typcially include children, women, and the elderly.For example, the World Bank and the Luxembourg Center for Population, Poverty and Public Policy and UNICEF [5] have conducted various studies of societal groups most affected by poverty, including complex research on the demographics of poverty, child poverty and its gender component [10].

In this study we rely on several income based measures of poverty for both countries, as well as a comparison of the extent of poverty in similar societal groups.To the best of our knowledge, no study generally compares poverty in Poland and the United States using the measures described below, or compares our selected societal groups.

Differences in Measuring Poverty Rates: Poland vs. US

There is substantially more data and studies on poverty in the United States than in Poland.However, as transition progresses and income inequality has increased, the topic of poverty has become more important in Polish language literature, published primarily by the government funded Main Statistical Office (GUS), private foundations [11] and some academic institutions.

As with all such studies, both Polish and US poverty data pose a series of methodological problems.In particular, the United States Bureau of Census measures poverty by comparing income to as elected poverty threshold for households of a given size, using it uniformly across the United States, despite major regional cost of living differences.In 2012, the average poverty threshold for a household of four was $23,492 annually.This income-based measure ignores household housing conditions, debt levels, financial assets (including savings or property), capital gains, or non-cash benefits (such as public housing, Medicaid, or food stamps).

In Poland, the major income-based poverty measure is calculated as 60 percent of the median income, also adjusted for household size.In 2012, that threshold for a household of four was 1863 PLN per month, or about $7460 yearly [11].As in the US, this measure also ignores real and financial household assets (though these are significantly lower than in the United States), access to subsidized housing (more common in Poland than the US) and is also not geographically adjusted (although Poland is significantly smaller in size, and less regionally diversified, then the United States).

However, the three-to-one income ratio of those rough poverty measures in the United States and Poland closely matches the ratio of their respective GDP per capita, which is $44,439 versus $14,710 [12].Using the above income-based approach in 2012 poverty rates in the United States and Poland were 15 and 6.8 percent, respectively.

In addition to these income-based poverty statistics, both Poland and the United States also employ additional ways of measuring poverty by incorporating housing quality, the ownership of various consumer goods, and the ability to provide adequate heating.But, as previously noted, these country-specific measures are not used in a systemic way and therefore are problematic when comparatively assessing economic quality of life.

Aside from measuring absolute poverty, international comparisons rely on various relative poverty measures, in which the "depth" of poverty is considered.One such measure involves the percentage of population/households receiving less than 60 percent of the average income [13].In 2011, those numbers were 17.1 percent and 11.1 percent, respectively, for the United States and Poland [14].Another relative poverty indicator accounts for the percentage of people with income 50 percent below the poverty line, which are often referred as those living in deep poverty.In the United States, 6.6 percent of population lived in deep poverty in 2012 and in Poland that number was 6.1 percent (using a rough equivalent poverty measure of the percentage of the population living below the "existential minimum") [15].This "existential minimum" denotes the lowest income that enables the purchase of a selected modest basket of identified consumer goods and services.

In addition to absolute and relative poverty rates, poverty gap measures are also sometimes considered, when calculated as the difference between the poverty line and the median income of those below the poverty line.The poverty gap in 2012 was 20.6 percent for Poland and 27.5 percent for the United States [16].

For the reasons previously noted, these statistical poverty income linked measures do not imply that the poor in one country are comparatively better off than their counterparts in another.This is particularly true where, as here, there is a high degree of income inequality between the United States-GINI coefficient in 2011 was 45.0, and Poland with a 32.7 percent GINI coefficient [17].

Nevertheless, these data do show that, by country-specific measures, the incidence of poverty in the United States is higher than it is in Poland by every indicator presented.Moreover, as appears below, this overall conclusion also applies to societal groups that are typically overrepresented among the poor.

Who Are the Poor: Poland vs. the US

Age

According to the literature, the largest poverty clusters are typically found among those who are the most economically vulnerable: children and the elderly.As indicated by Table 1, the incidence of poverty for children in the United States is higher than for children in Poland, regardless of the income threshold used.Furthermore, as per Table 2, in 2011, 16.6 percent of American children lived in poverty set at a threshold of 40 percent of average income and these numbers increase to 21.1 percent when the poverty threshold is set at 50 percent of average income and to 31.1 percent when 60 percent of average income is used.

When using the US national poverty threshold for 2012, American children living in poverty represented fully 23.7 percent of the total U.S. population, constituted 34.6 percent of all Americans in poverty, and accounted for 35 percent of Americans living in deep poverty.Using a roughly equivalent Polish measure (i.e., 40 percent of average income), 7.5 percent of Poland's children are poor (as opposed to 16.6 percent in the United States).This difference between countries diminishes as the poverty line threshold increases from 40 to 50 and 60 percent of average income, indicating that poverty is shallower among Polish children.

When the characteristics of children living in poverty are examined there appears to be a key difference between the two countries, in the US the race of the child appears to be a key factor, while in Poland the size of a household matters.As indicated in Table 3, almost three quarters of all children living in poverty in the US almost three quarters are Black or Hispanic.

While Poland's homogenous society does not have an analogous racial component, household size (number of children) does play a role in poverty status.In 2010, 34 percent of Polish families with 4 or more children fell below the poverty threshold [13].By contrast, the number of children in a household does not appear to be related to poverty in the United States.

The Elderly

Measuring poverty among the elderly through income-based statistics is particularly problematic, because in both countries members of this group often rely on various transfer payments and non-cash benefits unavailable to other groups that are not captured in official income data.Moreover, these benefits differ substantially in Poland and the United States.While seniors in both countries have access to health care, in Poland that access is free of charge (although quality and wait times can be problematic) while American seniors often pay high outof-pocket costs but are far less constrained by long wait times for care.In addition, many seniors in the United States supplement their social security income by working past their retirement age, which is rather unheard of in Poland.Poland's current retirement age is also lower than the US and often profession specific, inflating the number of Poles eligible for senior benefits.Furthermore, official poverty data in both countries often ignore savings and accumulated non-financial assets that directly impact the well-being of seniors, which further complicates comparative poverty studies of that group.

These concerns about measuring poverty among US seniors prompted the US Census Bureau to develop an alternative measure, which deducts health care expenses from income.This is known as the supplemental poverty measure and, under it, poverty in the United States among the elderly increased from 9 percent in 2007 to 15 percent using unadjusted data in 2011 [18].

Poland provides universal health care and no attempts have been made to adjust income of the elderly for health care expenses.The reliance of some seniors on private health care and rising co-payments for prescription drugs would tend to bias Poland's poverty measures downwards, as compared to the United States.In Poland, approximately 4 percent of all people over 65 were classified as poor in 2012, using a poverty threshold of 60 percent of median income adjusted for household size.As a percentage of all poor, Poland's seniors held a 14.7 percent share in 2011, which doubled from a 7.3 poverty share in 2005.However, this increase is partially methodological and, given Poland's increasing income polarization and income growth, use of median income as a poverty threshold indicates an increase in relative, not absolute, poverty [4].

There are some housing market differences between Poland and the US that can also affect the accuracy of poverty measurements in this age group.In Poland, many seniors pay relatively lower housing costs than other societal groups, which is a legacy of the socialist period.The relatively limited mortgage market and regulatory restrictions on Polish rentals left most seniors in apartments inherited from the pre-transition period, which are either leased or purchased from the State at below market rates as a part of the privatization process.Thus, although housing costs are not captured by poverty statistics in either country, they tend to more negatively affect the elderly in the United States, such that Poland's data on poverty among seniors likely suffers from some degree of upward bias.While poverty data among the elderly in both countries contain some serious limitations, regardless of the measures used the elderly in Poland and the United States constitute a substantial share of all poor in both countries and their actual poverty rates are likely to exceed those reported using income threshold poverty measures.

Poverty and Education

In addition to age, there is broad consensus in the literature that education, a key factor influencing income, also determines the incidence of poverty.While lack of education contributes to poverty, an individual's access to quality education is related to income status.Consequently, education is both a cause and effect of poverty.In the United States and Poland primary and secondary education is funded mostly by local communities; therefore, the quality of teachers, teaching materials, and course specialization (e.g., specific offerings in computer science, foreign languages, music, etc.) are linked to the affluence of a particular community.Generally, aside from government subsidies, low income communities in both countries are less able to fund education at the same level as high income communities.

Measuring and comparing the quality of primary and secondary education in different countries is an inherently difficult task.One generally applied measure involves the PISA score used internationally to rank the quality of education.Using PISA rankings, Poland scored 9 th in 2012, with a mean score of 518, while the United States ranked 31 st with a mean score of 481 [12].

The quality of primary and secondary education, in turn, impacts access to tertiary education, which is itself a key factor inversely linked to poverty.In that respect, American universities are ranked among the best in the world, whereas Poland lags far behind in this category.Aside from quality, there are, however, some characteristics of tertiary education that favor Poland, particularly in terms of enrollments and costs.In 2012, almost half of Poland' full time students attended public, tuition free universities.For those attending private schools, tuition ranged between $800 -$3000, representing less than 25 percent of Poland's average annual income (approximately $13,000, with the median being $12,700), only 10 percent of Polish students financed their education through loans and, surprisingly, 80 percent of those receiving loans attended public schools.

Overall, higher education in Poland is far more accessible for less affluent groups than it is in the US.Moreover, for Poland's less privileged access to higher education is increasing.During the 1994-2010 period, the difference in high education enrollments between the 1 st and the 5 th income quintile increased from 13.3 percentage points in 1994 to 40.5 percentage points in 2005, falling to 30.5 percentage points in 2008 [19].Aside from the changing labor market in Poland, which makes higher education more important over time, transition led to the formation of private colleges and universities in medium and small size cities.Given Poland's relatively severe legal restrictions on the rental market (and consequent high rents), shortening physical distances between students and universities increased access to higher education in smaller towns, which are often located in most impoverished areas.

The high quality of higher education in the United States comes with high costs, as tuition averaged $15,000 for public, and over $35,000 for private, schools in 2014.These costs substantially exceed international averages, and constitute a high entry barrier for many students.As a result, many more American than Polish students finance their education through loans and accumulate substantial debt.For example, in 2012, the average student loan balance in the United States was $24,301, with one-quarter of borrowers owing more than $28,000 and 10 percent of borrowers owing more than $54,000 upon graduation [20].

Over the last 25 years the relative increase in Polish tertiary education enrollments outpaced US equivalents in relative terms; as the demand for new skills escalated, the supply of Polish universities increased.Specifically, between 1995 and 2010, tertiary enrollment rates within the 20 -29 years old age group increased from 15 percent to 30 percent, with a greater number of women entering colleges and universities.During that same time period college participation rates in the United States also increased, but at a slower pace, rising from 19 to 25 percent [21].These limited data suggest that Poland's higher educational system, though lagging the United States in quality, is less costly and, therefore, more accessible to a relatively higher number of students.Since education may well be the single most important factor affecting poverty levels, Poland's relative advantage here has likely played, and will continue to play, an important role in explaining the above-cited differences in poverty between the two countries.

Poverty and Disability

Disability is another characteristic linked to poverty.In 2010 18.7 percent of the U.S. population had some form of disability and 12.6 percent were considered severely disabled.For those disabled persons aged 18 to 64, the poverty rate was 28.4 percent (4.3 million); that is, more than twice the poverty rate of 12.5 percent (22 million) for Americans in that age category who were not disabled in 2012.

The poverty rate for disabled Polish adults was 9.6 percent in 2010 [13], and declined to just 3 percent by 2012.That decline is largely attributable to changes in Poland's disability legislation, which substantially tightened the definition of disability and eligibility criteria for obtaining that status [22].The relatively low poverty rate among Poland's disabled also results, in part, from statistical methodology.In Poland, disability benefits are typically replaced with state pensions upon a disabled person attaining retirement age, which removes that person in poverty from the "disabled" to the "pensioner" category.By contrast, disabled Americans maintain their disabled status regardless of age and are often not eligible for full Social Security benefits, but instead receive a much lower SSA supplemental payment.

As in the other categories, comparing poverty among people with disabilities in both countries poses challenges.Aside from data problems that impede accurate calculations, it is difficult to establish equivalent, direct and indirect benefit values received by each disabled population.Despite those constraints, two basic conclusions can be drawn: disability is an important factor affecting poverty in both countries; and there are relatively fewer disabled Poles than disabled Americans living in poverty.

Geography of Poverty

Across the world poverty tends to cluster geographically.In the last 25 years, Poland's growth and development most positively affected large cities and its western regions.Some of the reasons for this geographical disparity of development are rooted in historical differences in regional development, urbanization, the quality of infrastructure, concentration of foreign direct investment, and availability of skilled labor.In Poland, which is divided into 16 districts (vojevodships), the highest incidence of poverty, regardless of the poverty measures used, is concentrated in two regions considered to be Poland's least developed: Swietokrzyskie, (11.9 percent) and Warminsko-Mazurskie (11.7 percent).The lowest poverty rates are in Opolskie (4.7 percent) and Lubuskie (4.8 percent), which are located in the most developed part of the country [23].

The two areas with the highest poverty rate exhibit high structural unemployment rooted in the elimination of large state farms and manufacturing companies (along with their related supply chains) through privatization, and the predominance of small towns in those regions (i.e., with populations under 50,000).This is particularly apparent in Warmiensko-Mazurskie, a region that relied heavily on state-guaranteed jobs, along with free or heavily subsidized housing and multi-layered social services for all employees.Many of those state jobs in agriculture and manufacturing required limited education, and therefore became obsolete as Poland transitioned to a new market economy, causing structural unemployment to skyrocket in these regions.

Another high poverty region in Poland-the Kieleckie region-shares the same problems rooted in privatization, structural unemployment of unskilled labor and poor infrastructure.The structural unemployment here is also linked to heavy industry firms (outside of coal and steel) that could not compete when Poland's opened its borders to western markets.Closingstate farms in a world with slowly improving infrastructure limited private sector formation and the inflow of domestic and foreign investment leading, in turn, to long-term structural un-employment and poverty.

By contrast, Poland's regions with the lowest incidence of poverty (Slaskie, Lubuskie, and Opolskie) are among the most developed, the most urbanized, and enjoy the best infrastructure.These regions also benefit from their proximity to Germany, which has facilitated significant (albeit often temporary) migrations and consequent transfers of remittances, as well as FDI inflows, to those regions.In combination, these remittances, FDI and developed infrastructure further increased urbanization, fueled growth, spurred employment, and contributed to a decline in regional poverty.

Poverty in the United States is also clustered geographically.In 2010, the average US poverty rate was 15.9 percent, and the states with the highest poverty rates included Mississippi (22.6),New Mexico (21.5), and Louisiana (20.4).The lowest rates were in New Hampshire (8.8),Maryland (10.1), and New Jersey (10.4), respectively [24].

Factors affecting the regionalization of Poland's poverty; i.e., poor infrastructure and a low degree of urbanization-do not appear to play a discernible role in the geographic distribution of poverty in the United States.In fact, urbanization levels in the states with the lowest poverty rates, such as New Hampshire (60.3 percent), are lower than in the two states with the highest poverty rates-New Mexico (77.4 percent) and Louisiana (73.2 percent).Additionally, poverty clusters exist in such highly urbanized locations as the Houston, Texas metropolitan area, with a record 37.7 percent incidence of poverty in 2011, the Fresno, California metro area-25.8percent, and the El Paso, Texas metro area-24.7 percent [24] Detroit, Michigan-44 percent in poverty (of whom 18.5 percent were in deep poverty), and East St Louis, Illinois-50.8percent in poverty (of whom 19.5 percent were in deep poverty) [25].

The reasons for regional clusters of poverty in both countries differ.In Poland, the geographical distribution of poverty is mainly a legacy of its central planning heritage; that is, concentrated privatization of state farms and heavy industry, and the related slow development of modern infrastructure and the private sector in those areas.In the United States, geographical poverty clusters seem unrelated to infrastructure.Instead, they are concentrated in areas with high minority populations, which is a factor completely absent in Poland.For example, states like California, where Hispanics constitute 60.6 percent of the total population, and portions of Texas (Maverick County, Webb County, and Starr County, being 96.8, 96.4, and 96.1 percent Hispanic, respectively) are each burdened with large poverty clusters [6].However, regional poverty in Poland and the United States do share a common characteristic-structural unemployment.And in both countries, that shared feature reflects changes in markets: in Poland structural unemployment is caused by privatization of state farm and factories and a declining demand for unskilled and semi skilled labor.In the US, the causes of structural unemployment generally reflect a decline in manufacturing caused by cheaper imports and the changing composition of labor market demand.

Unique Characteristics of the Poor: Poland vs. US

As indicated above, one key factor associated with poverty in the United States is race.In 2012, 27 percent of African Americans and 25 percent of Hispanics lived below the official poverty threshold [7]; Hispanics are more than twice as likely, and African Americans almost three times more likely, to live in deep poverty than non-Hispanic whites.Specifically, in 2012, 12.7 percent of blacks (almost 5.1 million), 10.1 percent of Hispanics (almost 5.4 million), and 4.3 percent of non-Hispanic whites (8.4 million) were living in deep poverty.

Another characteristic of the poor in the United States is their marital status.In 2012, almost 31 percent of households headed by a single woman were living below the poverty line-being nearly five times the 6.3 percent poverty rate for families headed by married couples [7].Sixteen percent of households headed by a single male were living in poverty.

As Poland's society is homogeneous, race is not a factor among the poor.At the same time, family structure plainly matters, and Poland's family structure is relatively intact, with only 12 percent of families being headed by single mothers [10].More importantly, of those families only 6 percent are in poverty [13].What distinguishes Polish poverty in terms of family structure is its size-as mentioned above those families with 4 or more children have a significantly higher incidence of poverty.

Consequently some of the key roots of poverty in the US and Poland are very different: while race and marital status are important predictors of US poverty, in Poland, those are not are irrelevant factors.Instead, family size and place of residence tend to explain some of the poverty causes.

Conclusions

Based on official poverty data in the United States and Poland, there are substantial differences, by societal groups, in the composition of the poor.Overall, a higher percentage of the overall US population is poor than in Poland and, relative to median country-specific incomes, US poverty is deeper.Among the groups that tend to be more vulnerable to poverty, relatively more American children and a higher percentage of persons aged 65 and older are poor than in Poland.In addition, more Americans than Poles are considered disabled, and in this group, a higher percentage of disabled Americans live in poverty.

As often discussed in the literature poverty indicators have severe limitation, as they often fail to provide a basis for comparing the standards of living of the societal groups across countries, do not accurately capture the purchasing power of income, and ignore numerous non-cash benefits that affect how well, or poorly, people actually live.

These poverty measures do, however, provide us with the general understanding that on relative terms poverty is wider in the United States than in Poland and, perhaps more importantly, suggest that poverty in America may also, in some important ways, be more difficult to address through government policies.In the United States the link between minority status, single parent households, and poverty is widely recognized.In Poland, large families are most likely to experience poverty.If family size in Poland is easier to address than the incidence of single parent households in minority communities, poverty in the US may also prove be more intractable.

Table 1 .
1
US and Poland's key poverty measures.
Measure/in percentage in 2012USPolandIncome determined poverty156.8Relative poverty17.111.1Deep poverty6.66.1Poverty gap20.627.5Of all poor, those under 1822 *14.1Incidence of poverty among those under 1819.110.9

Table 2 .
2

Percentage ofPoverty line at 40 percentPoverty line at 50 percentPoverty line at 60 percentchildrenof average incomeof average incomeof average incomeUS16.621.131.1Poland7.514.522.9Source: http://www.unicef-irc.org/publications/pdf/rc10_eng.pdf

Table 3 .
3
US children living in poverty, 2010.
CategoryPercentAll children under 1822.0White only, non-Hispanic12.4Black38.2Hispanic35.0Asian13.6

The Definition and Measurement of Poverty. A Hagenaars, K De Vos, 10.2307/145776Journal of Human Resources. 231988

An Evaluation of Subjective Poverty Definitions: Comparing Results from the US and the Netherlands. K D Vos, T I Garner, 10.1111/j.1475-4991.1991.tb00371.xReview of Income and Wealth. 371991

On the Measurement of Poverty. A Atkinson, 10.2307/1911028Econometrica. 551987

Poverty, Inequality and Family Living Standards across Seven Nations. T Smeeding, 10.1111/j.1475-4991.1993.tb00458.xReview of Income and Wealth. 391993

Poverty: An Ordinal Approach to Measurement. A Sen, 10.2307/1912718Econometrica: Journal of the Econometric Society. 441976

Exploring Alternative Measures of Welfare in the Absence of Expenditure Data. Review of Income and Wealth. D E Sahn, D Stifel, 10.1111/j.0034-6586.2003.00100.x200349

Composite Indices of Physical Quality of Life. R Rati, 10.1016/0304-3878(82)90005-0Journal of Development Economics. 111982

On Measuring Poverty. D Thon, 10.1111/j.1475-4991.1979.tb00117.xReview of Income and Wealth. 251979

Social Income Transfers and Poverty: A Cross-Country Analysis for OECD Countries. K Caminada, G Kees, K Ferry, 10.1111/j.1468-2397.2011.00815.xInternational Journal of Social Welfare. 212011

Child Poverty in Upper-Income Countries: Lessons from Luxemburg Poverty Study. J Jannati, C Gornick, 2010Luxemburg Center for PopulationLuxemburg

Ubostwoi Wyklucznie Spoleczne (Poverty and Social Exclusion). R Szarfenberg, 2012Friedrich Ebert FoundationWarsaw

United Nations Development Program. New YorkUS GDP Trading Economies2010. 2014Human Development Report

. Ubostwo W Polsce, 2010. 2012Data World Bank

. Wskazniki Zagrozenia, 2012

. Data Unicef Official, 2013

M Branko, Global Inequality Recalculated: The Effect of New 2005 PPP Estimates. Washington DC. B. LevenPoverty and Opportunity2009. 20135061

Higher Education. 2010

. Tamarind Financial Planning. 2013

Income and Poverty in the United State (2013) Current Population Reports. US Department of Commerce

. Wprost, 2013

Ubostwo Wsrod Dzieci w Polsce (Poverty among Children in Poland. Departament Analiz Ekonomicznych i Prognoz. Warszawa2008

. Poverty, 2010 and 2011. 2010

MI) Poverty Rate Data. Michigan Detroit, 2012


Improving the use of research evidence in guideline development: 3. Group composition and consultation process

Atle Fretheim email@example.com
Norwegian Knowledge Centre for the Health Services
St. Olavs plassP.O. Box 7004N-0130OsloNorway

Holger J Schünemann holgerjschünemannemail@example.com
INFORMA
Istitituto Regina Elena
Via Elio Chianesi 5300144RomeS.C. EpidemiologiaItaly

Andrew D Oxman
Norwegian Knowledge Centre for the Health Services
St. Olavs plassP.O. Box 7004N-0130OsloNorway

Improving the use of research evidence in guideline development: 3. Group composition and consultation process
10.1186/1478-4505-4-15BioMed Central Page 1 of 6 (page number not for citation purposes) Health Research Policy and Systems Review Andrew D Oxman -firstname.lastname@example.com * Corresponding author
Background:The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the third of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.Objective: In this review we address the composition of guideline development groups and consultation processes during guideline development.Methods:We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.Key questions and answers: What should be the composition of a WHO-panel that is set up to develop recommendations?The existing empirical evidence suggests that panel composition has an impact on the content of the recommendations that are made. There is limited research evidence to guide the exact composition of a panel. Based on logical arguments and the experience of other organisations we recommend the following:• Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers.• Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing.• Groups should include or have access to content experts.• To work well a group needs an effective leader, capable of guiding the group in terms of the task and process, and capable of facilitating collaboration and balanced contribution from all of the group members.

• Because many group members will not be familiar with the methods and processes that are used in developing recommendations, groups should be offered training and support to help ensure understanding and facilitate active participation.

What groups should be consulted when a panel is being set up?

We did not identify methodological research that addressed this question, but based on logical arguments and the experience of other organisations we recommend that as many relevant stakeholder groups as practical should be consulted to identify suitable candidates with an appropriate mix of perspectives, technical skills and expertise, as well as to obtain a balanced representation with respect to regions and gender.

What methods should WHO use to ensure appropriate consultations?

We did not find any references that addressed issues related to this question. Based on logical arguments and the experience of other organisations we believe that consultations may be desirable at several stages in the process of developing guidelines or recommendations, including:

• Identifying and setting priorities for guidelines and recommendations • Commenting on the scope of the guidelines or recommendations • Commenting on the evidence that is used to inform guidelines or recommendations • Commenting on drafts of the guidelines or recommendations • Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations.

• Key stakeholder organisations should be contacted directly whenever possible.

• Consultation processes should be transparent and should encourage feedback from interested parties.

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the third of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

Health care recommendations that are systematically and transparently developed and well informed by the best available evidence require several types of evidence and judgements. Judgements must be made about:

• The expected effects of the options that are being considered,

• Factors that might modify the expected effects in specific settings,

• Needs, risks and resources in specific settings,

• Ethical, legal and political constraints, and • The balance between the expected benefits harms and costs if a recommendation is implemented.

A group developing recommendations must be capable of assessing the evidence that is available to inform these judgements and to make all of these different types of judgements. In this paper we address the following questions:

• What should be the composition of a WHO-panel that is set up to develop recommendations?

• What groups should be consulted when a panel is being set up?

• What methods should WHO use to ensure appropriate consultations?

Questions related to group processes or integrating values and consumer involvement are addressed in two other papers in this series [1,2].

What WHO is doing now

Expert committees are sometimes used by WHO to provide guidance. The Director General selects committee members from WHO's expert advisory panels. The members of these panels are primarily included based on "their technical ability and experience". When an expert committee is assembled it should have: "equitable geographical representation, gender balance, a balance of experts from developed and developing countries, representation of different trends of thought, approached and practical experience in various parts of the world, and an appropriate interdisciplinary balance" [3].

Establishing an expert committee is a formal process with regulations established by the World Health Assembly. Many WHO recommendations are not developed by expert committees. Less formal procedures that are not subject to the Regulations for Expert Advisory Panels and Committees are frequently used to convene groups that develop guidelines or recommendations. "Consultations" or "proceedings" also frequently provide the basis for recommendations. Consumers or representatives of the general public are rarely included in groups that develop recommendations.

The Guidelines for WHO Guidelines state that the "Technical Guidelines Development group" should be multidisciplinary with around 8-12 individuals representing stakeholders (professionals, disease experts, primary care/ public health, end users, and patients) as well as methodologists [4]. However, up to now WHO has published few recommendations that have adhered to these guidelines.

Broad consultations do not appear to be commonly used to identify potential members of expert committees or other groups that develop guidelines or recommendations. Consultations at other stages during the process of developing recommendations are also uncommon, apart from peer review of draft reports. Occasionally, draft recommendations may be circulated more widely.

What other organisations are doing

In a recent international survey of 152 units that support the use of research in developing guidelines and health policy, most respondents reported that their guidelines development panels consisted of several stakeholders or expert groups -often including end-users and consumers [5]. Another review of guidelines on hypertension and hyperlipidaemia, found that stakeholder involvement was much lower in guidelines sponsored by specialty societies than in guidelines sponsored by other groups [6]. A third review of 18 prominent guideline organisations in Australia, Canada, Europe, New Zealand and the U.S. found that guideline development groups typically consist of 10 to 20 members and the number of disciplines per group is often three to five [7]. Most of these programs invite methodological experts (epidemiologists, library scientists and others) and patient representatives. Most have permanent staff providing editorial support.

Many agencies have issued guidelines for developing guidelines [8][9][10][11] (see also our review of Guidelines for Guidelines [12]), and all such documents that we are aware of recommend convening multi-disciplinary groups. This includes consumers, professionals working in the field, and individuals with the necessary methodological skills (e.g. epidemiologists, economists). Procedures for recruitment of panel members are usually prescribed or suggested, with emphasis on extensive consultation with relevant stakeholder-groups.

In contrast to approaches that rely heavily on clinical experts or research experts, as exemplified by many specialty societies [6] and the WHO, the National Institute for Health and Clinical Excellence (NICE) in the UK does not necessarily include experts in their guideline development groups (GDGs): "Experts attending a GDG are present because of their knowledge in a particular area. Therefore, it is important that they sit within the group and enter fully into any discussion. However, they are not full members of the group; they do not have voting rights and should not be involved in the final wording of recommendations." The role of the professional members in the GDG is to "represent the perspective(s) of the health care workers involved" [10]. In an external evaluation carried out by WHO, it was concluded that NICE is "internationally a leading agency", and that the organisation "has developed a well-deserved reputation for innovation and methodological development" [13].

Most guidelines for guidelines highlight the importance of having an effective leader. This person has a key role in facilitating "the interpersonal aspects of the group processes" and ensuring "that the group works in a spirit of collaboration, with a balanced contribution from all members" [10].

Some agencies arrange training for members of GDGsparticularly, but not solely, aimed at facilitating the active participation of consumer representatives [14]. Another suggested approach to ensure consumer involvement is to establish separate focus groups for this purpose [10].

Wide consultation in the course of developing recommendations may be done in various ways, for instance by hosting open meetings to discuss guideline drafts [8], or by posting guideline drafts on the web [10]. Peer-review is also commonly used. In the survey of 152 units that support the use of research in development of guidelines and health policy, most reported involvement of target-users in the selection of topics, e.g. in priority-setting groups, through surveys or by reviewing draft lists of priority topics [5]. Most respondents also reported having consumers involved at various stages of the development process, often by review of draft guidelines or reports [5]. In its manual for guideline developers, NICE specifies several stages during guideline development where consultations with stakeholders should take place:

• When the draft scope of the guideline has been prepared • During the selection of panel members

• When the full draft version of the guidelines is completed

Methods

The methods used to prepare this review are described in the introduction to this series [15]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We searched PubMed and three databases of methodological literature (the Cochrane Methodology Register [16], the US National Guideline Clearinghouse [17], and the Guidelines International Network [18]) for existing systematic reviews and relevant methodological research that address these questions. We did not conduct systematic reviews ourselves. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

In our literature search on panel composition we used the terms "group composition" or "panel composition" or "'consumer involvement' and guidelines". We also checked the reference lists of key papers and contacted key researchers in the field. In our search on consultation processes, we used the term "guidelines and consultation and process".

Findings

What should be the composition of a WHO-panel that is set up to develop recommendations?

We identified relatively few articles on group composition. A key paper was a comprehensive report by Murphy and colleagues from 1998, who reviewed the research literature on group composition and clinical guideline development [19]. This systematic review identified several studies that compared recommendations by groups with different compositions, and several comparisons of judgements made by homogenous subgroups of mixed groups [19]. The authors found that "these studies, although few in number, show that differences in group composition may lead to different judgements. More specifically, members of a specialty are more likely to advocate techniques that involve their specialty." Their conclusion was that "these studies confirm that the composition of groups is important in determining the decision reached."

Knowing that groups with different compositions produce different recommendations does not necessarily tell us what group composition will provide the most appro-priate recommendations. Arguments for using multidisciplinary groups are largely based on logic. For example, "Individuals' biases may be better balanced in multidisciplinary groups, and such balance may produce more valid guidelines" [20]. A report from the U.S. Institute of Medicine (IOM) put forward three arguments for multidisciplinary groups: 1) Multidisciplinary participation increases the probability that all relevant scientific evidence will be located and critically evaluated; 2) Such participation increases the likelihood that practical problems with using guidelines will be identified and addressed; 3) Participation helps build a sense of involvement or "ownership" among different audiences for the guidelines [21].

Arguments against having narrowly focused expert groups are based in part on research that have compared expert recommendations to systematic reviews and which have investigated the relationship between expertise and systematic reviews. In one study comparisons were made between recommendations of clinical experts in textbooks and major medical journals and results of meta-analyses of randomized controlled trials of treatments for myocardial infarction [22]. The investigators found that clinical experts often made recommendations that were not consistent with available research findings. Another study found strong correlations between the expertise of authors of reviews and the methods that were used in the reviews [23]. Expertise was associated with stronger opinions prior to conducting a review, less time spent conducting a review, and the use of less systematic and transparent methods.

The systematic review by Murphy and colleagues included studies of the effects that heterogeneity has on group judgement, and concluded that "The weight of evidence suggest that heterogeneity in a decision-making group can lead to a better performance than homogeneity. There is, however, also some evidence that heterogeneity may have an adverse effect because conflict may arise between diverse participants" [19].

In addition, the review found few studies of the extent to which the particular individuals that participate in a group affect the groups' decisions. The authors concluded that "the selection of individuals has some, but not a great deal, of influence on outcome", based on this limited research.

Finally, the review included research related to the optimal size of groups. The authors remark that "having more group members will increase the reliability of group judgement", while "large groups may cause coordination problems". They base their conclusion mainly on research within social and organisational psychology: "It is likely that below about six participants, reliability will decline quite rapidly, while above about 12, improvements in reliability will be subject to diminishing returns."

We have identified one recent study not included in the review by Murphy et al. where recommendations made by groups of different composition were compared [24], as well as four studies comparing judgements made by different subgroups in mixed groups [25][26][27][28]. The findings, which are consistent with the conclusions of the review, indicate that clinical experts have a lower threshold for rating the procedures they perform as being appropriate. Another study, in which clinicians were surveyed about the appropriateness of coronary angiography for various indications, found a similar relationship [29].

We did not find any studies on the impact of group composition on public health or health systems recommendations. However, there is some evidence that suggests that the same relationships between expertise and recommendations that have been found for clinical recommendations are also found for public health recommendations; i.e. that expert recommendations are frequently not consistent with the available research evidence [30]. Moreover, those making public health and health policy recommendations may frequently not systematically consider potential adverse effects of public health and health policy interventions.

We found several systematic reviews of consumer involvement [31][32][33][34][35]. There are a number of relevant arguments for including consumers in groups that develop recommendations and descriptions of practical experience. However, a review of comparative studies of interventions to promote consumer involvement [31], found a "lack of evidence from comparative studies to inform decisions about desirable and adverse effects of consumer involvement in collective decisions about health care or how to achieve effective consumer involvement". The authors of another systematic review, which included non-comparative studies of how to involve consumers in setting the research and development agenda for health systems [32], recommend collaborating with "well-networked consumers and providing them with information, resources and support to empower them in key roles for consulting with their peers." They also recommended "consultations should engage consumer groups directly and repeatedly in facilitated debate." See also our review on how to integrate values and involve consumers in guideline development [2].

What groups should be consulted when a panel is being set up?

We did not identify papers addressing this question.

What methods should WHO use to ensure appropriate consultations?

We did not identify any research findings that could inform the answer to this question

Discussion

Based on the findings from the reports we have identified, there is sufficient evidence to conclude that how a panel is composed can have an important impact on conclusions drawn by a group when making recommendations for health care. In particular, clinical experts are more likely to recommend procedures linked to their own specialty than others. Furthermore, experts in a field frequently do not employ systematic methods when reviewing evidence and developing recommendations. These findings support the current recommendation in the Guidelines for WHO Guidelines: Panels should be multidisciplinary, including a broad representation of stakeholders, as well as methodologists [4].

The research evidence to guide panel composition and consultation processes is limited. However logical arguments and the experience of other organisations suggest that • Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers.

• Special attention should be paid to the selection of a group leader who has a crucial role in ensuring a positive group process and that all voices within the group can be heard.

• Wide consultations should be done when selecting members of a group to develop WHO recommendations, for example by direct contact with stakeholder groups.

• Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing.

• Groups should include or have access to content experts.

• Many group members will not be familiar with the methods and processes that are used in developing recommendations, and should be offered training and support to help ensure understanding and facilitate active participation.

The process of developing recommendations, including the selection of group members, should be transparent.

The process should also include wide consultation that encourages feedback at subsequent steps in the process, which may include:

• Identifying and setting priorities for guidelines and recommendations • Commenting on the scope of the guidelines or recommendations • Commenting on the evidence that is used to inform guidelines or recommendations • Commenting on drafts of the guidelines or recommendations • Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations.

• Commenting on what research should be conducted based on the guidelines

Further work

We have not conducted an exhaustive systematic review, but have based much of this paper on a systematic review from 1998. We have not found subsequent studies that provide conflicting evidence. There is, however, limited research for the questions addressed in this report. We do not believe that a more exhaustive review would yield a great deal of additional evidence at this time. However, it would be valuable for WHO or others to undertake and keep up-to-date systematic methodology reviews that address specific issues of group composition, including the selection of a group leader, methods for effective consultations, and methods for effective consumer involvement.
 (page number not for citation purposes)
AcknowledgementsWe gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.Competing interestsAF and ADO work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.Authors' contributionsAF prepared the first draft of this review. HJS and ADO contributed to drafting and revising it.
Improving the use of research evidence in guideline development: 5. Group Processes. A Fretheim, H Schünemann, A D Oxman, Health Res Policy Syst. Fretheim A, Schünemann H, Oxman AD: Improving the use of research evidence in guideline development: 5. Group Proc- esses. Health Res Policy Syst 2006.

Improving the use of research evidence in guideline development: 10. Integrating Values and Consumer Involvement. Health Res Policy Syst 2006. 3. Regulations for Expert Advisory Panels and Committees. In Resolution WHA 35 10 WHO. H Schünemann, A Fretheim, A D Oxman, Schünemann H, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 10. Integrating Values and Consumer Involvement. Health Res Policy Syst 2006. 3. Regulations for Expert Advisory Panels and Committees. In Resolution WHA 35 10 WHO; 1982.

Guidelines for WHO guidelines. Geneva, Global Programme on Evidence for Health Policy, WHO. Guidelines for WHO guidelines. Geneva, Global Programme on Evidence for Health Policy, WHO; 2003.

A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. Oslo, Norwegian Knowledge Centre for the Health Services. R Moynihan, A D Oxman, J Lavis, E Paulsen, Moynihan R, Oxman AD, Lavis J, Paulsen E: A review of organiza- tions that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. Oslo, Norwegian Knowledge Centre for the Health Services; 2006.

The relation between methods and recommendations in clinical practice guidelines for hypertension and hyperlipidemia. A Fretheim, J W WilliamsJr, A D Oxman, J Herrin, J Fam Pract. 51Fretheim A, Williams JW Jr., Oxman AD, Herrin J: The relation between methods and recommendations in clinical practice guidelines for hypertension and hyperlipidemia. J Fam Pract 2002, 51:963-968.

Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. J S Burgers, R Grol, N S Klazinga, M Makela, J Zaat, SIGN Guideline Development Handbook. Edinburgh, Scottish Intercollegiate Guidelines Network. 15Int J Qual Health CareBurgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evi- dence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003, 15:31-45. 8. SIGN Guideline Development Handbook. Edinburgh, Scottish Intercollegiate Guidelines Network.; 2004.

Handbook for the Preparation of Explicit Evidence Based Clinical Practice Guidelines. Wellington, New Zealand Guidelines GroupHandbook for the Preparation of Explicit Evidence Based Clinical Practice Guidelines. Wellington, New Zealand Guide- lines Group.; 2001.

National Institute for Health and Clinical Excellence: The guidelines manual. London, National Institute for Health and Clinical Excellence. National Institute for Health and Clinical Excellence: The guidelines manual. London, National Institute for Health and Clinical Excel- lence; 2006.

National Health and Medical Research Council: A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra, National Health and Medical Research Council. National Health and Medical Research Council: A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra, National Health and Medical Research Council; 1998.

Improving the use of research evidence in guideline development: 1. Guidelines for Guidelines. : H Schünemann, A Fretheim, A D Oxman, Health Res Policy Syst. Schünemann: H, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 1. Guidelines for Guidelines. Health Res Policy Syst 2006.

Technology Appraisal Programme of the National Institute for Clinical Excellence. Copenhagen, WHO Regional Office for Europe. S Hill, S Garattini, J Van Loenhout, O 'brien, B J De Joncheere, K , Hill S, Garattini S, van Loenhout J, O'Brien BJ, de Joncheere K: Tech- nology Appraisal Programme of the National Institute for Clinical Excellence. Copenhagen, WHO Regional Office for Europe; 2003.

The NICE Patient Involvement Unit. Evidence-Based Healthcare & Public Health. M Kelsom, 9Kelsom M: The NICE Patient Involvement Unit. Evidence-Based Healthcare & Public Health 2005, 9:304-307.

Improving the use of research evidence in guideline development: introduction. A D Oxman, A Fretheim, H Schünemann, Health Res Policy Syst. Oxman AD, Fretheim A, Schünemann H: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst 2006.

. Cochrane Methodology, Register, Cochrane Methodology Register [http://www.cochrane.org/

Guidelines International Network (GIN) Guidelines International Network. Guidelines International Network (GIN) Guidelines International Network 2006 [http://www.g-i-n.net].

Consensus development methods, and their use in clinical guideline development. M K Murphy, N A Black, D L Lamping, C M Mckee, C F Sanderson, J Askham, T Marteau, Health Technol Assess. 288Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, Marteau T: Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998, 2:i-88.

Clinical guidelines: developing guidelines. P G Shekelle, S H Woolf, M Eccles, J Grimshaw, BMJ. 318Shekelle PG, Woolf SH, Eccles M, Grimshaw J: Clinical guidelines: developing guidelines. BMJ 1999, 318:593-596.

Directions for a New Program. M J Field, K N Lohr, Clinical Practice Guidelines. Washington DC, National Academy PressClinical Practice Guidelines. Directions for a New Program. Edited by: Field MJ and Lohr KN. Washington DC, National Academy Press; 1990.

A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. E M Antman, J Lau, B Kupelnick, F Mosteller, T C Chalmers, JAMA. 268Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC: A com- parison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992, 268:240-248.

The science of reviewing research. A D Oxman, G H Guyatt, Ann N Y Acad Sci. 703Oxman AD, Guyatt GH: The science of reviewing research. Ann N Y Acad Sci 1993, 703:125-133.

An experimental study of determinants of group judgments in clinical guideline development. R Raine, C Sanderson, A Hutchings, S Carter, K Larkin, N Black, Lancet. 364Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, Black N: An experimental study of determinants of group judgments in clinical guideline development. Lancet 2004, 364:429-437.

Effect of panel composition on physician ratings of appropriateness of abdominal aortic aneurysm surgery: elucidating differences between multispecialty panel results and specialty society recommendations. J Herrin, J A Etchason, J P Kahan, R H Brook, D J Ballard, Health Policy. 42Herrin J, Etchason JA, Kahan JP, Brook RH, Ballard DJ: Effect of panel composition on physician ratings of appropriateness of abdominal aortic aneurysm surgery: elucidating differences between multispecialty panel results and specialty society recommendations. Health Policy 1997, 42:67-81.

Appropriateness of surgery for sciatica: reliability of guidelines from expert panels. J P Vader, F Porchet, T Larequi-Lauber, R W Dubois, B Burnand, Spine. 25Vader JP, Porchet F, Larequi-Lauber T, Dubois RW, Burnand B: Appropriateness of surgery for sciatica: reliability of guide- lines from expert panels. Spine 2000, 25:1831-1836.

Effect of specialty and nationality on panel judgments of the appropriateness of coronary revascularization: a pilot study. S J Bernstein, P Lazaro, K Fitch, M D Aguilar, J P Kahan, Med Care. 39Bernstein SJ, Lazaro P, Fitch K, Aguilar MD, Kahan JP: Effect of spe- cialty and nationality on panel judgments of the appropriate- ness of coronary revascularization: a pilot study. Med Care 2001, 39:513-520.

Physician recommendations for coronary revascularization. Variations by clinical speciality. K Fitch, P Lazaro, M D Aguilar, Y Martin, S J Bernstein, Eur J Public Health. 9Fitch K, Lazaro P, Aguilar MD, Martin Y, Bernstein SJ: Physician rec- ommendations for coronary revascularization. Variations by clinical speciality. Eur J Public Health 1999, 9:181-187.

Rating the appropriateness of coronary angiography--do practicing physicians agree with an expert panel and with each other?. J Z Ayanian, M B Landrum, S L Normand, E Guadagnoli, B J Mcneil, N Engl J Med. 338Ayanian JZ, Landrum MB, Normand SL, Guadagnoli E, McNeil BJ: Rat- ing the appropriateness of coronary angiography--do prac- ticing physicians agree with an expert panel and with each other? N Engl J Med 1998, 338:1896-1904.

Trying to do more Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations. I Chalmers, The Annals of the American Academy of Political and Social Science. 589Chalmers I: Trying to do more Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations. The Annals of the American Academy of Political and Social Science 2003, 589:22-40.

Interventions for promoting consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. The Cochrane Database of Systematic Reviews. E S Nilsen, H T Myrhaug, M Johansen, S Oliver, A D Oxman, 3Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD: Interven- tions for promoting consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. The Cochrane Database of Sys- tematic Reviews, Issue 3 2006.

Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. S Oliver, Clarke-Jones , L Rees, R Milne, R Buchanan, P Gabbay, J Gyte, G Oakley, A Stein, K , Health Technol Assess. 81Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K: Involving consumers in research and development agenda setting for the NHS: developing an evi- dence-based approach. Health Technol Assess 2004, 8:1-IV.

Consumer involvement in health research: a review and research agenda. J Boote, R Telford, C Cooper, Health Policy. 61Boote J, Telford R, Cooper C: Consumer involvement in health research: a review and research agenda. Health Policy 2002, 61:213-236.

Systematic review of involving patients in the planning and development of health care. M J Crawford, D Rutter, C Manley, T Weaver, K Bhui, N Fulop, P Tyrer, BMJ. 3251263Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, Tyrer P: Systematic review of involving patients in the planning and development of health care. BMJ 2002, 325:1263.

Effective Consumer Voice and Participation for New Zealand. S Coney, New Zealand Guidelines Group. Coney S: Effective Consumer Voice and Participation for New Zealand. New Zealand Guidelines Group; 2004.