Check-off programs are designed to expand the market for a given agricultural commodity, such as eggs, pork, or highbush blueberries, through generic promotion, research, and consumer and industry information. A check-off program is meant to expand the demand for a commodity rather than for any particular brand or producer. Although some state, regional, and local check-off programs that have existed for over 70 years may be voluntary, federal programs are mandatory. Many commodity groups prefer mandatory programs to address the free rider problem—that is, producers, handlers, processors, importers, or others in the marketing chain who do not pay into a check-off program but benefit economically from voluntary programs that others have funded. After Congress authorized the Cotton Research and Promotion Act of 1966, the first federally mandated agricultural check-off program and board—for cotton—was created. Over the next three decades, Congress authorized the creation of an additional 11 commodity programs and their respective boards. The 12 programs and boards created under the authority of individual stand-alone legislation adhere to the specific requirements as set forth in their respective authorizing legislation. The passage of the Commodity Promotion, Research and Information Act of 1996 (generic legislation) gave USDA the authority to establish additional commodity check-off programs and boards. Since then, 10 additional boards were created based on this generic legislation. Those boards are subject to the requirements set forth in the generic legislation. (See table 1 for the year established and authorizing legislation for all 22 check-off programs.) To create a check-off program, industry groups first identify the need for such a program and then negotiate among themselves to agree on a basic program framework. The framework includes the rate of assessment and the various program activities to be undertaken, such as promotion, advertising, research, and providing information to consumers and industry. Additionally, each industry proposes regulations to USDA for the structure of the board that will carry out these activities. Because each industry has unique characteristics, a different board structure is appropriate for each check-off program. The boards vary in size, geographic representation, and types of individuals who are board members—that is, producers, processors, handlers, importers, public representatives, or others in the marketing chain. USDA, in consultation with the industry, then develops regulations to define how the program will be operated, how the funds will be collected, and how compliance with the authorizing legislation will be maintained, among other things. The check-off programs must be approved by a majority of producers— and in some cases processors, importers, and handler—subject to the assessments. To gain approval, a referendum must be held either before check-off program operations begin within some specified time after assessments are first collected, depending on the authorizing legislation. To fund a check-off program, producers, handlers, processors, importers, or others in the marketing chain are assessed for each unit of the commodity sold, produced, or imported. For example, for each 30-dozen cases of eggs sold, a producer is assessed $0.10. These funds go to the American Egg Board. The boards are to use assessments for the research, promotion, and consumer and industry information activities as well as for reimbursing AMS for its oversight costs. In 2016, total assessments collected for the 22 check-off programs ranged from $0.6 million for the popcorn check-off program to $332.1 million for the dairy check-off program (see table 2). To facilitate oversight, AMS breaks the 22 check-off programs into four of the agency’s commodity areas: (1) Cotton and Tobacco—the cotton check-off program; (2) Dairy—the dairy and fluid milk check-off programs; (3) Livestock, Poultry, and Seed—the beef, egg, lamb, pork, sorghum, and soybean check-off programs; and (4) Specialty Crops—the Christmas tree, Hass avocado, highbush blueberry, honey, mango, mushroom, paper and packaging, peanut, popcorn, potato, processed raspberries, softwood lumber, and watermelon programs. AMS has a functional committee for the check-off programs, which comprises a chair and the deputy administrators from the four AMS commodity areas and meets quarterly. The functional committee reports to the AMS Associate Administrator and was established to increase coordination and promote best practices and consistency across the 22 check-off programs. Additionally, the four commodity area directors and other senior agency officials meet weekly to discuss any issues that have arisen and to discuss any necessary policy changes. AMS marketing specialists are responsible for the day-to-day oversight of the check-off boards and for ensuring that board decisions and operations are carried out in accordance with applicable legislation and regulations. Each check-off program has a designated AMS marketing specialist serving as the primary overseer of all check-off program activities. (Fig. 1 shows AMS’s oversight structure for the check-off programs.) As part of their oversight duties, marketing specialists review and approve board budgets, contracts, promotional activities, board policies, and bylaws, among other activities. Every 3 years, marketing specialists also are to conduct management reviews that assess each of the 22 check-off boards’ internal controls intended to determine whether there is reasonable assurance that the boards are in compliance with statutes, regulations, and the board’s and AMS’s policies and procedures. AMS management reviews are to include reviews of check registers, contract and subcontract samples, assessments collected, and travel reimbursements, among other items. AMS’s Guidelines for AMS Oversight of Commodity Research and Promotion Programs, most recently updated in September 2015, is designed to facilitate the application of legislative and regulatory provisions of the check-off programs and promote consistency in AMS’s oversight of the 22 check-off programs. These guidelines, which pertain to AMS as well as board members and board staff, are not intended to cover the daily responsibilities of board operations or AMS’s oversight. Instead, the guidelines provide broad information on AMS’s expectations for how boards should operate and how AMS will oversee the programs in activities such as budget approval, contracts, financial accountability, referendum, and investments, among other items. In March 2012, USDA OIG released a report on AMS’s oversight of check-off programs. The work was initiated by the OIG after a 2010 investigative report, conducted at the request of the AMS Administrator, identified the possibility of weak oversight controls over the check-off boards. The 2012 report included two recommendations for AMS to develop and implement (1) standard operating procedures that provide detailed instructions for performing oversight activities to address all areas listed in the agency’s guidelines and (2) guidance for conducting periodic internal reviews of program area operations to ensure the enforcement of AMS’s guidelines. AMS agreed with the two recommendations and planned to implement them with a variety of actions, as discussed below. AMS has responded to recommendations for improving oversight made in the OIG’s 2012 report, particularly by developing and implementing standard operating procedures and conducting internal reviews of AMS check-off program oversight. However, AMS does not provide consistent oversight across check-off programs in some areas; specifically, it does not routinely review check-off program subcontracts during its management reviews, conduct follow-up on management review recommendations, ensure that financial assurances are included in annual audits, or ensure that check-off boards share information with assessment payers on program websites. In conducting their oversight of the check-off programs, senior agency officials and marketing specialists said they face challenges because of increased use of social media, the absence of an information system for tracking approvals, and complex Freedom of Information Act (FOIA) requests for some programs, which may delay the completion of some oversight priorities. The OIG’s 2012 report included two recommendations that AMS has since implemented: to develop and implement (1) standard operating procedures and (2) guidance for conducting periodic internal reviews of its oversight activities. In August 2013, AMS developed and implemented its standard operating procedures, which provide marketing specialists with more detailed guidance on the various oversight activities that are outlined in the agency’s program guidelines. The standard operating procedures cover a range of oversight activities, including budget review, contract review, advertising and promotional materials review, and financial and internal control oversight. Included in the more detailed guidance are various checklists that marketing specialists can use to itemize the requirements that boards must meet in a variety of areas. For example, the budget review checklist includes a list designed to ensure that budgets conform to law and contain, among other items, accurate sums and categories, as well as clearly listed administrative expenses. According to senior agency officials and marketing specialists, the standard operating procedures have assisted AMS in providing consistency across the 22 commodity check-off programs, have helped ensure that oversight responsibilities are carried out, and have provided documentation of specific duties for new marketing specialists. In response to the OIG’s second recommendation, AMS has developed and implemented guidance for conducting internal reviews of its oversight of check-off programs. Internal reviews are conducted by AMS’s Management and Analysis Program group to evaluate whether the AMS commodity areas that oversee check-off programs employ controls that provide reasonable assurance that the check-off programs are meeting legislative and regulatory requirements. According to an AMS directive, internal reviews of each of the four AMS commodity areas are to be conducted on a rotating basis. An AMS internal review of the Cotton commodity area was completed in November 2014, an internal review of the Specialty Crops commodity area was completed in September 2015, and an internal review of the Dairy commodity area was competed in May 2017. According to officials in the Management and Analysis Program, the Livestock, Poultry, and Seed commodity area internal review began in May 2017. The Cotton internal review found the program to provide reasonable assurance that the boards were complying with legislative requirements and that the oversight controls were adequate and functioning as intended. The Specialty Crop internal review found that the commodity area was fulfilling its oversight responsibilities but also found opportunities to strengthen control practices, including ensuring consistent and timely application in its use of checklists and tracking management reviews to ensure that they are completed and issued in a timely manner. As a result, the Specialty Crop commodity area implemented changes to its use of checklists and agreed to complete management reviews in a timely manner. The Dairy internal review found opportunities to strengthen oversight, primarily with regard to management reviews and recordkeeping. As a result, according to senior agency officials, the Dairy commodity area has implemented changes to its management review and recordkeeping processes. We identified four areas in which AMS does not provide consistency in its oversight across its check-off programs: (1) review of subcontracts, (2) follow-up on recommendations made to check-off boards, (3) ensuring that independent financial audits contain statements of assurance, and (4) ensuring that information is available on program websites for assessment payers (i.e., transparency). Subcontracts. The 2012 OIG report found that AMS did not recognize in its guidelines for check-off programs that its oversight role extended to monitoring subcontracts. Following the release of the OIG report, AMS updated the guidelines to respond to the OIG finding. Under the 2015 AMS guidelines, marketing specialists are to review a sample of subcontractor expenses during their management reviews. However, we found that AMS did not similarly update its standard operating procedures for the check-off programs and that these reviews are not being done consistently across programs. We found that the marketing specialist for one of the eight programs we reviewed chose a sample of subcontracts for the management review and documented this selection in the management review report. Marketing specialists for three of the programs said they reviewed subcontracts only if the sample of primary contracts that were part of the management review included subcontracts. Marketing specialists for the other four programs said they did not review subcontracts. Two marketing specialists we interviewed said they do not select a sample of subcontracts because check-off boards are responsible for overseeing and monitoring subcontracts. Senior agency officials and marketing specialists also noted that they review and approve all promotional materials regardless of whether any material is from a contract or subcontract. Senior agency officials also said that the contracting process differs among the various check-off boards and may cause confusion about what is considered a subcontract for purposes of a management review. For example, the cotton board contracts with Cotton Inc. to carry out the program’s research and promotion activities; Cotton Inc. may, in turn, contract with entities to carry out those research and promotion activities—considered a cotton board subcontract. This is in contrast to processes of other boards, such as the honey board, which can directly contract with entities to carry out research and promotion activities; those contractors may, in turn, subcontract duties. In addition, the potential exists for subcontract costs to total hundreds of thousands of dollars. A 2010 OIG investigative review found that a subcontractor of one check-off board used subcontracts to pay employees unauthorized bonuses of about $302,000. Without revising its standard operating procedures for check-off programs to recognize that each management review is to include a sample of subcontracts for review, AMS’s ability to prevent misuse of subcontract funds is impaired. Recommendation follow-up. Under AMS’s guidelines and standard operating procedures, marketing specialists are to ensure that corrective actions are taken by the boards in a timely manner if a matter is recommended in the management review, conducted every 3 years. For example, the standard operating procedures state that the board has 30 calendar days from the receipt of the management review report to respond to the findings by formal letter and that follow-up should include appropriate documentation of the corrective actions taken. The 2012 OIG report found that there was little consistency among AMS commodity areas regarding the reporting of management review results and follow- up procedures. Four of the check-off programs we reviewed obtained written confirmation from boards about how they intended to address issues identified during management reviews consistent with the standard operating procedures; three others did not obtain written confirmation, but said they obtain any check-off board plans for remediation via less formal means, such as via e-mails or during board meetings. The management review for the eighth program did not contain any recommendations. According to marketing specialists we interviewed, the follow-up process to ensure that boards have taken corrective actions is also informal—a specialist learns how management review recommendations have been implemented by attending board and committee meetings. Senior agency officials verified that AMS has no mechanism for tracking follow- up with check-off boards to ensure that they have taken corrective actions. Under federal internal control standards, management should remediate identified internal control deficiencies on a timely basis and, with oversight from the oversight body, monitor the status of remediation efforts so that they are completed on a timely basis. Without establishing a mechanism for documenting and tracking follow-up with check-off boards on the implementation of management review recommendations, AMS has no assurance that it is consistently monitoring the status of corrective actions. Senior agency officials said that having a formal method to track and follow up on management review recommendations would allow them to identify trends, best practices, and similar emerging issues among the check-off programs. Independent financial audits. Each year, each check-off board is required by law to hire an independent audit firm to conduct an audit of the board’s financial statements in accordance with generally accepted government auditing standards. This audit helps to ensure compliance with legislative, regulatory, and policy directives. AMS guidelines direct marketing specialists to review the annual financial audits to determine whether the auditor identified any misuse of board funds and if the audit adequately addressed whether (1) funds were discovered to be used for influencing government policy or action, (2) the board adhered to the AMS investment policy, (3) internal controls over funds met auditing standards, (4) funds were used only for projects and other expenses authorized in a budget approved by USDA, and (5) funds were used in accordance with AMS guidelines. The standard operating procedures state that AMS is to ensure that audits contain these five statements of assurance, and they state that the audit firm is to express an opinion on the financial statements of the board and include a report on internal controls and compliance with applicable laws and regulations. The 2012 OIG report found that none of the independent audit reports included the five statements of assurance for the 18 check-off boards reviewed. In our sample, audit reports for four of the eight programs included the five statements of assurance. For two of the programs in our sample, the engagement letters, which document the agreed-upon terms of the audit, contained all five assurances, but the audit reports did not contain the five assurances. For the remaining two programs in our sample, neither the engagement letters nor the audit reports contained all five assurances, but senior agency officials said that the AMS marketing specialists for those two programs ensured that these assurances were adequately addressed during pre- and post-audit meetings. According to marketing specialists we interviewed for those two programs, audits following government auditing standards incorporate the requirements and are fulfilled by a general statement that boards were in compliance with laws and regulations. However, the 2012 OIG report found that an independent auditor did not include the specific assurances because the auditor was not asked to perform such work and only minimal adjustments would be needed to provide for those assurances. Without ensuring that its annual independent financial audits include the five statements of assurance as outlined in the standard operating procedures, AMS will have less certainty that check-off funds are not subject to waste, fraud, or mismanagement. Transparency. According to the Business Roundtable and the Organisation for Economic Cooperation and Development’s principles of corporate governance, a strong disclosure regime that promotes transparency is central to stakeholders being able to access regular, reliable, and comparable information. As check-off programs use assessment money collected from stakeholders of the commodity being promoted, AMS’s guidelines state that both transparency and oversight of the check-off funds are critical. Moreover, AMS’s guidelines state that annual budget summaries should be posted on the check-off board’s website and that three additional documents are either to be on the website or otherwise made available: (1) the bylaws and policy statements, (2) annual reports, and (3) the independent economic evaluation of effectiveness. Four of the eight check-off programs in our sample posted all four documents on the programs’ websites. All eight check-off programs posted their annual reports online. Four of the check- off programs, however, did not post to their websites at least one of the remaining documents—the budget summary, bylaws, or independent economic evaluation. Marketing specialists we interviewed said that boards would supply information not included on the websites if an assessment payer requested such information, which is consistent with AMS guidelines. Board executives we interviewed from those programs that do not post all four documents on their websites also said that they would supply the information to assessment payers if contacted. Senior AMS officials also said that there are stakeholders who may not have computers or access to the Internet and may therefore request information via postal mail. Industry organization representatives we interviewed said that transparency of how funds are used and the effectiveness of the programs are important to their members. One industry organization representative we interviewed said that, although some stakeholders may not use the Internet, posting information on how assessments are being used, such as the information provided in annual reports, is useful for stakeholders and builds trust among check-off boards and stakeholders. Posting information on the boards’ websites could convey information to stakeholders who have access to the Internet at a low cost. Without including in the guidelines and standard operating procedures that all four key check-off board documents (i.e., bylaws and policy statements, annual reports, and independent evaluations of economic effectiveness) should be posted on a check-off program’s website, AMS may be missing an opportunity to ensure that some assessment payers have access to information on program operations and effectiveness. AMS officials identified ongoing challenges in check-off program oversight. In particular, AMS marketing specialists and senior agency officials identified three challenges: (1) the increase in some check-off boards’ use of social media, (2) the absence of an information system to track approvals, and (3) complex and time-consuming FOIA requests for some programs. Because of competing priorities, some oversight duties may be delayed as a result. Increase in boards’ social media efforts. According to marketing specialists, four of the eight check-off programs have seen a significant increase in the boards’ use of social media, which has been a challenge in terms of both workload and the need for additional AMS guidance because the specialists must approve the social media content. Marketing specialists for the other four programs said that the check-off programs they oversee have not yet increased their social media presence enough to make it a challenge for workload. Senior agency officials and marketing specialists agreed that oversight of the check-off programs requires a significant amount of time and effort that has been made more complicated since some check-off programs began using social media. For example, a marketing specialist for one check-off program approved over 3,000 items, including social media for promotional and research materials, in a 6-month period. According to this marketing specialist, depending on the complexity of the item needing approval, there could have been dozens of communications between the specialist and the check-off board staff. In addition, marketing specialists and senior agency officials said that because social media is constantly evolving, AMS has needed to reevaluate its guidance to boards for social media. The senior agency officials acknowledged that the duties of marketing specialists are demanding and that they are working to find ways to provide support to marketing specialists. Senior agency officials said that this is challenging because the boards must reimburse AMS for oversight costs, so any additional personnel would be paid for through check-off assessments. Also, AMS established a social media committee made up of marketing specialists who have drafted social media guidance for the boards to follow. Technology. Tracking the numerous promotional and research approvals can be a challenge for some AMS marketing specialists because of the absence of an information system to track approvals. According to two marketing specialists, during busy times, they may be handling more than 20 requests for approvals a day. While marketing specialists for two of the check-off programs we reviewed said that the use of approval tracking software, paid for by the respective check-off boards, has made their oversight function more efficient, other marketing specialists said that they must rely on e-mail messages to organize the status of approvals. Marketing specialists who have tracking software said that they can quickly see the status of any approval at any given time; further, check-off board staff can also use the software to prioritize approvals. One marketing specialist said that, although she had developed a system for organizing e-mails, a tracking system used by both AMS and the board would ensure that oversight activities would not be delayed and could expedite the approval process. Senior agency officials said that it would be helpful if each marketing specialist had this software but that the check-off boards would need to pay for this expense. FOIA requests. Responding to complex FOIA requests about check-off programs has been a challenge, according to senior agency officials, marketing specialists, and board executives of four of the eight programs we reviewed. Some requests do not take many resources to fulfill, but others take significant time and money. For example, to respond to a FOIA request, board staff and marketing specialists must identify pertinent documents; review them to ensure that there is no proprietary or sensitive information; and, as needed, involve the board’s legal counsel or third-party businesses. According to senior agency officials, in one case, a FOIA request resulted in the check-off board and AMS providing approximately 10,000 documents to the requester. AMS estimates that in fiscal year 2016, for the Livestock, Poultry, and Seed commodity area programs, it cost the agency about $182,000 and more than 2,700 hours to fulfill FOIA requests. For the same period for the Dairy commodity area programs, AMS estimates that it cost over $365,000 and about 6,600 hours to fulfill FOIA requests. Because AMS is reimbursed for its oversight costs, the funds to cover FOIA-related costs come directly from check-off assessments. These cost estimates do not include check-off board staff resources utilized to fulfill FOIA requests. Senior agency officials said that there are legal constraints on the types of individuals and organizations that they can request cover fees associated with document retrieval under FOIA. Independent economic evaluations of the effectiveness of check-off programs, conducted at least every 5 years, have generally shown a positive benefit to those who pay assessments. The evaluations we reviewed varied both in the methods used to conduct the analysis and how information was reported and revealed certain methodological limitations. According to senior agency officials as well as the economists who conducted the evaluations, the variations are in part due to the differences in check-off board resources. We found that AMS does not consistently document its review of independent economic evaluations and has no criteria established for determining what makes for a credible methodology and results. The Federal Agriculture Improvement and Reform Act of 1996 requires check-off boards to (1) fund independent economic evaluations of the effectiveness of their promotion activities every 5 years, (2) submit the evaluation to USDA, and (3) make the results available to the public. Check-off boards, through a request for proposals process, contract for independent economic evaluation to determine the effectiveness of promotion activities. The law does not specify how an independent economic evaluation should be completed, and AMS does not offer any guidance on the methodologies to use, the types of information to include, or how the results of such an evaluation are to be presented. AMS guidelines, which are available to the check-off boards, state that evaluations: (1) have a credible methodology, (2) articulate shareholder returns, and (3) present the results in a non-technical manner. The eight independent economic evaluations of check-off programs we reviewed focused on benefit-cost ratios (BCR) and returns on investment (ROI). While BCRs and ROIs are slightly different, they both measure the financial gain or loss generated from the costs of implementing a program. In both cases, economists use economic, industry-specific models to determine the benefits or economic gains from the check-off programs by isolating the impacts of program promotion dollars from other variables, such as competing products or changes in consumer income. For example, some models include the effects of changes in the prices of substitute food products, which may affect the demand for commodities. The model used in the evaluation for the beef check-off program, for instance, includes prices for both chicken and pork, as an increase in the price of chicken or pork could lead to an increase in the consumer demand for beef, regardless of check-off program activities. Other variables that may affect demand include changes in (1) consumer buying habits, (2) consumer income, and (3) government policy. These variables can either increase or decrease the demand for commodities despite the activities of check-off programs. Evaluation models may also include variables that affect the supply of a commodity, such as increased prices that send signals to farmers to increase production. Although it is difficult to capture, some commodity evaluation models also model increases in yields and acreage to determine how much the agricultural research portion of a check-off program affects the supply of the commodity. Increased supply as a result of agricultural research expenditures can also increase producer benefits and economic gains, but according to the sorghum and cotton evaluations, many of these gains cannot be immediately or directly measured. For the eight check-off programs we reviewed, the BCRs and ROIs ranged from 2.14 to 17.40. In other words, for every dollar invested in the check-off programs, the programs returned from $2.14 to $17.40 in revenue to assessment payers (see table 3). However, it is important to note that the results of the independent economic evaluations should not be compared across check-off programs because of differing methodologies, differing data, and differing demands for the products, according to economists we interviewed. Economists we interviewed and literature we reviewed suggested that although the results of an independent economic evaluation may appear large, the amount invested in promotion activities is small compared to the total value of industry sales. Therefore, the overall impact of promotion activities on the market may be small. Program referenda largely show that most assessment payers approve of check-off programs, but not all types of assessment payers may feel that they share equally in the benefits that are found through the independent economic evaluations, according to economists we interviewed. The studies we reviewed report either average or marginal measures of effectiveness, such as a BCR. Some economists we interviewed, both those who have conducted the evaluations we reviewed and those who did not, said that these types of studies do not address the level of ROIs across the distribution of check-off program payers or how much more larger-sized assessment payers receive in returns from their investment in the check-off program as compared to smaller-sized ones. This view was confirmed by representatives we interviewed from some of the industry organizations, who indicated that their members would prefer to better understand what they receive for their investment at the farm level. In addition, one economist we interviewed said that assessment payers may be skeptical of the results of independent economic evaluations of program effectiveness because while the costs are tangible, the benefits of the programs are not. That is, the producers cannot see what portion of their revenues is directly attributable to check-off program activities. The independent economic evaluations we reviewed were conducted using different methodologies and reported different information. According to senior agency officials, evaluations likely vary because legislation does not include any details on how evaluations should be completed and the amount of resources that each check-off board has available to devote to evaluations varies. Nearly all of the economists we interviewed said that it would be useful to have minimum standards for information that should be included in the evaluations. Some independent economic evaluations used different types of models and data to estimate the benefits and costs to assessment payers. For example, for the egg and honey check-off commodity models, the evaluation used two separate types of methodologies to estimate increases in demand because of the programs’ promotional activities. Some other independent economic evaluations in our sample, such as for cotton and fluid milk, used multi-market models that incorporated components for substitute products, the foreign sector, and the government sector. Some independent economic evaluations, such as the beef evaluation, measured a marginal BCR and others measured an average benefit-cost ratio. The cotton and sorghum evaluations performed an analysis of how increases in yields, acreage, and production because of the research portion of the check-off programs affected the supply of the commodities. The independent economic evaluations also examined different time periods in their analyses, depending on the available data (see table 3). For example, the egg evaluation covered the period of 2007 through 2010, and the fluid milk evaluation covered 1995 through 2012. In addition to having different methodologies to calculate benefits and costs, information and analyses included in the independent economic evaluation reports also varied among the eight programs we reviewed. For example, the beef check-off evaluation includes a section on the optimal allocation of funds to domestic activities of the program, which is not included in any other report. Seven of the eight evaluation reports had a conclusions section. One of the evaluation reports included a recommendations section, while others did not. Although the law does not specify information required to be included in the independent economic evaluations, representatives from one industry organization we interviewed said that having the information in a consistent format could help ensure that stakeholders could compare information from one evaluation to the next for a given check-off program. The independent economic evaluations provided useful information to key stakeholders and the general public, but we found that they also included a number of caveats and limitations. Some of these limitations resulted from the nature of a commodity or program itself and others from the modeling procedures used. According to economists we interviewed and senior agency officials, the law is not prescriptive about how evaluations are to be conducted, and the boards differ in the amount of resources available to devote to the evaluations. If, for example, a board has limited resources available for an evaluation, there may not be funds available to purchase a certain set of data. For the sample of eight evaluations we reviewed, these limitations included the following: Data limitations: A number of the independent economic evaluations had data limitations. For example, one independent economic evaluation (highbush blueberry check-off program) lacked either wholesale or retail price data for its demand model, and another (sorghum check-off program) lacked program data as it had only been in existence for 5 years when the evaluation was performed. All of the economists we interviewed who had completed the eight evaluations we reviewed said that data are a challenge when conducting the evaluations either because such data do not exist or the check-off boards do not have the resources to buy the data. Not discounting the BCR to present value: The cotton check-off evaluation was the only one in our sample with a methodology that discounted the BCR to present value to account for the time value of money. Discounting a program’s benefits and costs to present value transforms gains and losses occurring in different time periods to a common unit of measurement. Not accounting for spillover effects: Some independent economic evaluations did not include the spillover effects—the cross- commodity impact of promotion—on related markets, though some, such as the cotton evaluation, did account for spillover effects on competing commodities. If spillover effects pertain to a commodity, failure to account for these effects could overstate the benefits of a program and cause an upward bias in computing the BCR. Not adjusting models for structural changes: Some independent economic evaluations did not adjust models for structural changes in the industry over time. While some independent economic evaluations we reviewed, such as those for the honey and beef check-off programs, did use data or methods that accounted for changes in market structure over time, others did not. For example, for the pork check-off program, some hog farms have specialized in a single phase of production, and have encountered substantial gains in productivity because of technology over the past several decades, but the independent economic evaluation did not reflect this. Failure to correct for such structural change, if applicable to a commodity, can lead to incorrect modeling and misleading policy implications. AMS’s standard operating procedures acknowledge that each check-off program varies in size and scope; therefore, the amount of resources each program can devote to an independent economic evaluation varies. Smaller programs may have independent economic evaluations that reflect the realities of program scope, financial capability, and data availability. Our discussions with the economists who conducted the evaluations that we reviewed confirmed that this is the case. They said that the smaller programs are able to devote fewer resources to independent economic evaluations; therefore, the economist conducting an evaluation may not be able to complete all of the analysis that could be completed for a larger program that is able to pay for more complex analysis. According to senior agency officials, in some instances, a broader evaluation is not necessary because of the emphasis and goals of the program. In addition, the resources a board is able to devote may vary from evaluation to evaluation. For example, one economist said that he worked with a board that wanted a more comprehensive evaluation than was previously done. The new evaluation model included additional data over a longer period of time, which ultimately led to an increased ROI. In addition to ensuring that independent economic evaluations are conducted every 5 years and encouraging boards to make them available to assessment payers, AMS’s standard operating procedures state that marketing specialists should ensure that independent economic evaluations (1) have a credible methodology and results, (2) articulate shareholder benefits, and (3) present results in a non-technical manner. To verify that these three directives are met, the standard operating procedures state that marketing specialists may consult with agency economists. They are directed to document verification in writing. Outside of any agency review, there is no requirement that independent economic evaluations be peer reviewed. A National Academies report states that peer review is characterized, in part, as being a documented, critical review of assumptions, calculations, and methodology, performed by a person with technical expertise in the subject matter to be reviewed who is independent and external of the work being reviewed. The report further states that the peer, to the extent possible, should have sufficient freedom from funding considerations to ensure that the work is impartially reviewed. According to senior agency officials, AMS economists meet this definition; and their review of the independent economic evaluations can be considered peer review. Officials said that the economists on staff critically review the evaluations; they all have PhDs in economics and are independent as they do not work directly with the check-off programs except for reviewing the evaluations. Three of the four AMS commodity areas—Cotton and Tobacco, Dairy, and Specialty Crops—utilized an AMS economist to review the independent economic evaluations and document that review. Senior agency officials said that the Livestock, Poultry, and Seed commodity area has an AMS economist review the independent economic evaluations but does not document that review. According to senior agency officials, the Livestock, Poultry, and Seed commodity area has relied on informal reviews of the evaluations by an economist, which are orally presented to the director of the commodity area. Further, the economists who completed the eight independent economic evaluations we reviewed indicated that although their preference is to have the evaluations peer reviewed, this is not always possible because of time constraints and other priorities. One economist said that the board he worked with included a contractual requirement that the independent economic evaluation be peer reviewed. Because the Livestock, Poultry, and Seed commodity area does not document its reviews of independent economic evaluations, only four of the eight check-off programs in our sample had documented reviews of the evaluations. All four of the documented reviews ensured that the independent economic evaluations had a credible methodology and results and articulated shareholder benefits, as stated in the standard operating procedures. However, only two of these four check-off programs included in their documented review whether results were presented in a non-technical manner, as also stated in the standard operating procedures. Further, the internal reviews did not use standard criteria to determine whether the independent economic evaluations had a credible methodology or results, which is important because, as noted earlier, the evaluations we reviewed varied in their methodology and we found that they had certain limitations. Although check-off programs are not subject to the guidelines in the Office of Management and Budget’s Circular A-94, the circular provides general guidance for conducting analyses to help federal agencies efficiently allocate resources through well-informed decision making. For example, Office of Management and Budget Circular A-94 establishes key elements of an economic analysis, including (1) a statement of the objective and scope of the analysis, (2) an identification of alternatives, (3) an analysis of the economic effects, (4) a sensitivity analysis, and (5) adequate documentation and transparency. Conducting and documenting reviews of independent economic evaluations using criteria can be useful. For example, in 2014, a senior agency official found several inconsistencies in a check-off program independent economic evaluation. The senior agency official assigned an AMS economist and marketing specialist to work with the evaluator to revise econometric models to more accurately capture the activities of the check-off program. According to the official, if the independent economic evaluation had not been reviewed, benefits of the program would have been understated and would have misled those paying into the check-off program. Without developing criteria by which AMS can assess the methodology and results of independent evaluations and document those assessments to ensure that the standard operating procedures are met, the agency’s assessments of independent economic evaluations may be inconsistent across check-off programs and misleading to agency officials, check-off boards, and assessment payers. AMS oversees commodity check-off programs that conduct research and promotion activities to strengthen 22 commodities’ position in the marketplace. The agency has taken steps to improve oversight activities based on recommendations in USDA OIG’s 2012 report, but it continues to face challenges in other oversight activities. For example, AMS has not consistently reviewed subcontracts during its management reviews. Without revising its standard operating procedures for check-off programs to recognize that management reviews should include a sample of subcontracts for review, AMS’s ability to prevent misuse of subcontract funds is impaired. In addition, AMS has not consistently followed up on recommendations made to check-off boards, although its guidelines and standard operating procedures state that marketing specialists are to ensure that corrective actions are taken by the boards in a timely manner if a matter is recommended in a management review. Without establishing a mechanism for documenting and tracking follow-up with checkoff boards on the implementation of management review recommendations, AMS has no assurance that it is consistently monitoring the status of corrective actions. Moreover, AMS has not ensured that independent financial audits contain statements of assurance as called for in the agency’s program guidelines or standard operating procedures. Without ensuring that its annual independent financial audits include the five statements of assurance outlined in the standard operating procedures, AMS will have less certainty that check-off funds are not subject to waste, fraud, or mismanagement. Further, although principles of corporate governance state the importance of transparency for stakeholders, AMS has not ensured that certain information, such as budget summaries and program evaluations, are presented on check-off program websites and has not included in its guidelines or standard operating procedures that certain information should be included on program websites, although the agency’s program guidelines recognize that transparency of check-off funds is critical. Without including in the guidelines and standard operating procedures that key check-off board documents are to be posted on the check-off program’s website, AMS may miss the opportunity to ensure that some assessment payers have access to information on program operations and effectiveness. Finally, check-off boards are meeting legislative deadlines by completing independent economic evaluations of effectiveness every 5 years; however, the evaluations vary and have certain methodological limitations. Without developing criteria by which AMS can assess whether evaluations have a credible methodology and results and documenting those assessments, the assessments may be inconsistent across check- off programs and misleading to agency officials, check-off boards, and assessment payers. We are making the following five recommendations to the Administrator of the Agricultural Marketing Service: The Administrator of AMS should revise the standard operating procedures for AMS’s check-off programs to state that management reviews include a sample of subcontracts for review. (Recommendation 1) The Administrator of AMS should establish a mechanism for documenting and tracking follow-up with check-off boards on the implementation of management review recommendations. (Recommendation 2) The Administrator of AMS should ensure that annual independent audits include the five statements of assurance as outlined in the standard operating procedures. (Recommendation 3) The Administrator of AMS should include in the guidelines and standard operating procedures that key check-off board documents, such as bylaws and policy statements, annual reports, and independent evaluations of economic effectiveness are posted on the check-off programs’ websites. (Recommendation 4) The Administrator of AMS should develop criteria by which to assess the methodology and results of independent evaluations and document those reviews to ensure that the standard operating procedures are met. (Recommendation 5) We provided a draft of this report for review and comment to USDA. An auditor with AMS’s Management and Analysis Program responded via e- mail on October 24, 2017, that the agency generally agreed with our findings and recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of Agriculture, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the individual named above, key contributors to this report included Thomas M. Cook (Assistant Director), Rose Almoguera, Kevin S. Bray, Barbara El Osta, Cindy Gilbert, Holly Halifax, Khali Hampton, Dan Royer, Holly Sasso, Sheryl Stein, and Kiki Theodoropoulos.
“Got milk?” and “Pork: The Other White Meat” are examples of advertising campaigns undertaken by 2 of the 22 federal agricultural research and promotion programs, commonly known as commodity check-off programs. These programs, funded by a fraction of the sale of each unit of a commodity, are led by boards consisting of industry members appointed by the Secretary of Agriculture. The programs conduct research and promotion activities to strengthen a commodity's position in the market. In 2016, check-off funds totaled over $885 million. By law, funds cannot be used for lobbying or disparaging other commodities, among other things. AMS has primary responsibility for overseeing the check-off programs. GAO was asked to review AMS's oversight of the check-off programs. This report examines (1) the extent to which AMS has addressed previously identified weaknesses in its oversight and (2) how the effectiveness of the programs has been evaluated and what the results have indicated. GAO selected a sample of 8 such programs—selected, in part, based on total funds collected—and reviewed laws, regulations, and agency guidance. GAO interviewed agency officials, check-off board executives, and economists. The U.S. Department of Agriculture's (USDA) Agricultural Marketing Service (AMS) has improved its oversight of check-off programs since USDA's Office of Inspector General (OIG) made recommendations in a 2012 report. In response to two OIG recommendations, AMS developed and implemented standard operating procedures, which outline specific oversight responsibilities of AMS, and began to conduct internal reviews of its oversight functions. However, GAO found that AMS does not consistently review subcontracts—a legal agreement between a contractor and third party—or ensure that certain documents are shared with stakeholders on program websites. Subcontracts. Under AMS's 2015 guidelines for check-off programs, which cover broad oversight activities, staff are to review a sample of subcontracts during agency reviews of program operations. However, AMS did not revise its standard operating procedures to match its guidelines with this responsibility, and GAO found that AMS reviewed subcontracts for only one check-off program in its sample of eight. Without revising the standard operating procedures to include a review of subcontracts, AMS's ability to prevent misuse of funds is impaired. Transparency. According to leading business principles, transparency is central to stakeholders' access to regular, reliable, and comparable information. However, GAO found that four of the eight check-off programs reviewed posted all key documents, such as budget summaries and evaluations of effectiveness, to program websites. GAO found that AMS's guidelines state that budget summaries should be posted on program websites, while the other key documents are to be available on the website or otherwise made available to stakeholders. Agency officials said that boards would supply documentation if contacted by a stakeholder. Industry representatives GAO interviewed said that transparency of how funds are used and the effectiveness of programs are important to their members. Without including in its guidelines and standard operating procedures that all key documents should be posted on a check-off program's website, AMS may miss an opportunity to ensure that stakeholders have access to information on program operations and effectiveness. Independent economic evaluations of the effectiveness of check-off programs, required by law to be conducted every 5 years, have generally shown positive financial benefits. For the eight evaluations GAO reviewed, benefits ranged from an average of $2.14 to $17.40 for every dollar invested in the programs. However, the evaluations varied in the methods used and had certain methodological limitations. For example, some evaluations did not account for the effects of promotion from competing commodities, which could overstate the programs' benefits. AMS's standard operating procedures state that the agency should review the evaluations to ensure that there is a credible methodology, among other things; however, AMS did not consistently document reviews of the evaluations or have criteria by which to review the evaluations. Without developing criteria to assess the methodology and results of evaluations, the agency's assessments of independent economic evaluations may be inconsistent across check-off programs and misleading to stakeholders. GAO is making five recommendations, including that USDA revise its standard operating procedures to include the review of subcontracts, include key documents on check-off program websites, and develop criteria to assess evaluations. USDA generally agreed with GAO's recommendations.