How Did the Reports of OTA, the Congressional Research Service, and the National Academies Differ?

A 1995 Office of Technology Assessment report. Source: http://ota.fas.org

By Richard E. Rowberg

The Office of Technology Assessment (OTA) was de-funded in 1995. Whenever Congress considers whether OTA should be revived, a question inevitably arises: how did OTA’s reports differ from those issued by the Congressional Research Service (CRS) and the National Academies (NA)?

I was employed by all three of these agencies, and I believe the best way to answer this question is to compare how each of these organizations prepare a report. I will attempt this with a hypothetical example. Let’s assume each agency has been asked to prepare a report evaluating the technical, economic, environmental, and regulatory considerations for the development of the ‘smart’ electric transmission grid. This is clearly a major study that would take several months to complete.

(As an aside, I note that this comparison will be of limited value in comparing the numerous other types of products that the three organizations—particularly CRS—have offered and are offering, or—in the case of OTA—were offering. These latter projects usually focus on narrower questions and are designed to provide rapid responses.)

Office of Technology Assessment (OTA)

All OTA studies had to be requested by the chair and ranking member of a congressional committee or a Member of the Technology Advisory Board (TAB).  TAB consisted of 12 members—six from the House and Six from the Senate—equally divided by party.  Very few TAB requested studies occurred while I was there. Once the request arrived at OTA, it would be assigned to the appropriate program unit—Energy and Materials in this case. Before any work could start, the project had to be approved by the TAB. One of its functions was to ensure requests that should have gone to CRS, GAO, or CBO were redirected. In my recollection, such redirection was very rare. Once TAB approved the study, a project director, who had knowledge about the topic, would be selected from the regular OTA staff. Once selected, he or she and the Program director, in consultation with the requesting Committee staff, developed the project scope.  OTA staff would also develop a project budget. The funds came from the regular OTA appropriation, and the project budget included the salaries of all staff—permanent and temporary—working on the project. (This fact was not well known outside of OTA—certainly no one at CRS knew this as I discovered when I arrived there in 1986—and was the primary reason reported project budgets seemed so high.) Next, the project director would build a project staff of permanent OTA employees and, if necessary, on-site temporary—contract—employees covering the various disciplines needed for the study; economics, engineering, law, etc. The first step for this staff would be to select an advisory committee of 10 to 15 experts from outside the government to provide oversight for the study. This committee also contained experts in all of the disciplines covered by the study. The function of this committee was strictly advisory. The members did not write any of the study itself.

The study would cover all the important issues in detail. For example, what is the role of energy storage in developing a smart grid, what are the current options for addressing that question, and what are the research opportunities for advanced energy storage options. A similar focus existed for the other, non-technical areas such as costs, potential environmental impacts, safety considerations, and—in this case—vulnerability to attacks. The report would cover these issues so that the expert would find the coverage complete and correct, but in a way that was understandable for a broad, educated audience. The staff had no procedural restriction in obtaining information for the study, just budget limitations. Site visits, phone calls, (later e-mails) and meetings with utility operators, environmental groups, renewable energy companies, research engineers, and utility regulation experts among others were allowed and encouraged. On occasion, outside papers and/or quantitative model runs were commissioned to provide information for the study, but not for the actual writing of the report. (Given the very negative experience with large-scale—entire report—outside contracting in 1973, this method was quickly and permanently abandoned. This fact, too, was not well-known outside the agency.) Periodically, the staff would meet with the advisory committee to get its reading on study progress including a critique of its report to date and advice on other issues to consider and information sources to explore. One of the most important functions of the advisory committee was to make sure the study considered all relevant viewpoints on the issues and that these viewpoints were fully presented in the report.

When a draft of the report was complete and program management agreed, the report was sent to a large number of other experts for review. While we did not get responses from all of these reviewers, we made sure that reviews were received covering all of the key issues raised in the report. The names of all the reviewers to which the report was sent—whether they responded or not—were included in the finished document. Once the report had cleared peer review and was approved by program and division management, it was submitted to TAB for a final review and sign-off. While on occasion, the latter resulted in some pushback for ‘political’ reasons, it very rarely prevented a report from being released. Only when TAB approved the report could it be delivered to the requesting committee.

OTA reports provided conclusions, but no findings or recommendations. Rather, detailed discussions of the issues identified—be they technical, economic, environmental, regulatory and so forth—were presented.  These discussions included a look at potential unintended consequences of different ‘technical’ options to address the issues.  In our example, for instance, potential problems with power flow reliability may occur as the grid becomes more connected to renewable energy sources.  In addition policy options for addressing problems identified in the technical analysis were analyzed. Pros and cons of these policy options were presented but no findings or recommendations were presented.  As with the technical body of the report, the identification and analysis of these policy options was the responsibility of the report staff. This study would likely have taken over a year to complete. In addition to the report, a one- to two-page summary would be written, and the OTA press office would prepare a press release to be published after delivery to the requesting committee. The project director and perhaps other members of the staff would brief the Hill and perhaps testify if the committee called hearings on the topic. Once delivery was complete, however, the staff would disband, permanent employees would move onto other projects, and temporary staff would leave unless another internal opportunity came up. While no formal mechanism for follow-up activities existed unless it was the result of a follow-on request from the committee, informal assistance, however, was possible. OTA staff members working on the study often became a resource to Hill staff working on related legislation.   

Congressional Research Service

If the same request arrived at CRS, it would have been sent to the Resources, Sciences, and Industry division. It would have been assigned to the section working on electric utilities. The large scope of the request would likely involve the section’s research manager and perhaps the division’s head, who would ensure the project was adequately staffed. In this case, a staff member covering renewable energy, an energy economist, an environmental specialist, and possibly someone from CRS’s American Law Division would complete the staff. The study would be fairly comprehensive, and would address non-technical considerations such as economic, environmental, and legal issues. The level of technical detail, however, would likely be limited by the subject knowledge of the authors and the resources available to them. While CRS staff is not prohibited from contacting outside experts, resource limitations make that very difficult. The report could present conclusions, but findings and recommendations would be absent as in the OTA case. Like the OTA study, technical options for addressing the many such issues identified would be provided to the extent they could be identified by the project staff. The report could also present policy options with a pro and con analysis as in the OTA case. The nature of this request would probably demand such an analysis.

Such a study would probably take around six to nine months—longer than most CRS studies—but would not have been as comprehensive as done at OTA. Upon completion of the report, it could go for outside review if time permitted.  This did not happen very often, however, because of the short deadlines for most CRS products.  For the case considered here, however, an outside review by experts in the field—electric utility engineers and economists, regulators, environmental specialists, etc.—would be advised and could take place.  Before such a review proceeded, however, a  pre-review by CRS’s Review Office would be required. In my experience this was not a major barrier.  Once any outside assessment is completed, the Review Office would have another look at it. Its sign-off was required before the report could be released. Because the CRS Review Office is not staffed by subject matter experts, the review focused on whether the report met CRS standards of impartially—presented all sides of the issues—and did not inadvertently present recommendations. The finished product would include a one-page summary, and would be delivered to the requesting members or committee. CRS staff would be available for briefings, testify if asked to do so by the committee, and act as extended staff resource for development of legislation. Of course, as we are all aware, public release would not happen, at least not through CRS.

Finally, CRS does have the resources and or mandate for major updating its reports.  While a major update may be impractical, as new technical findings or other developments occur, it is very likely that the report will undergo a number of limited updates. In addition, CRS’s recent development of other means of publishing analysis—the blog-like, CRS Insights, CRS videos, etc.—means that important new developments on this topic can be transmitted rather quickly.

National Academies

The performance of National Academies’ studies are most often requires a contract from the requesting agency to pay for the work. Occasionally, however, they are funded by private foundations or by National Academy of Sciences’ endowment funds.  Even for those studies mandated by Congress—about 20 to 25% of its work—an agency must still come up with the funds.  Occasionally the funds are provided directly by a congressional appropriation. Once the request comes in, a proposal is prepared for the agency and also for internal review and approval. These preparations are done in consultation with the requesting agency and, if congressionally mandated, relevant congressional staff and Members.  The internal review is done by the National Academies Governing Board Executive Committee which consists of the presidents and selected other members of the three Academies and meets monthly.

Consensus studies, a category in which this request would fall, are the premier product of the National Academies. Each of these studies requires a study committee made up of outside experts representing the broad range of science and technology covered by the study. They also include experts from areas also important for the study—in this case economics, environmental and regulatory concerns, etc.—just as in the case of the OTA advisory committees. The National Academies’ study committees, however, are more heavily weighted towards the technical and scientific issues. The selection of a study committee is a long process and requires an elaborate approval process, leading up to the President of the National Academy of Sciences. Special care has to be taken to vet for bias, diversity, composition, and potential conflicts of interest. The approval process is more elaborate and formal than the OTA process was, but the latter was just as careful to ensure its advisory committees met high standards in these areas.

The study committee for a National Academies consensus study serves an advisory role and is responsible for the content of the study. In most cases, it writes the study although staff does contribute. Because of this role, staff, unlike either CRS or OTA, has more of a facilitating than analytical responsibility. At its first meeting, the study committee sets out a plan for carrying out the study and decides on the first round of information and data resources it needs. The staff will then arrange for experts—utility operators, academic researchers, regulatory officials, etc.—to brief the study committee at its subsequent meetings. Members of the study committee may also take site visits. For about two or three meetings of the study committee, it digests a great deal of information and begins to process it to form a set of findings and recommendations. Members of the study committee are assigned by its chair to write chapters and eventually—by its last meeting—a report emerges. The key is consensus—the study committee members are tasked with coming to consensus on the findings and recommendations. On rare occasions that consensus is not reached, a dissenting statement results. But consensus is almost always obtained, although the compromises required to come to consensus can sometimes undermine the differences among study committee members that reveal important strengths and weaknesses of alternative options. This is more of a problem for studies that consider economic, regulatory, and related issues for which there are a broader spectrum of differing views. Because achieving consensus is not an objective of CRS and OTA reports but rather it is the presentation and analysis of options, such “ideological” constraints faced by National Academy studies are usually not an issue for the other two groups.[i]

Once a draft is agreed to by they study committee, a formal peer review process is undertaken. Reviewers are chosen much as the study committee members were. Again a formal approval process is carried out. As it was with OTA, these reviewers focus on the subject matter content and whether sufficient and proper evidence is apparent to support the findings and recommendations.  The reviewers provide their comments to the staff, and in conjunction with the study committee, a response to that review is prepared. In addition to the reviewers, an office within the National Academies, the Report Review Office, selects an individual—usually a member of one of the three Academies—to monitor the review. He or she acts as a sort of referee to adjudicate the response. Once the monitor agrees the response is satisfactory, the report is approved for release. A typical report takes from 12 to 18 months, usually the longest of the three groups. It is also the most expensive, even though the members of the study committee are all volunteers.

The first customer of the release is the sponsoring agency. Almost simultaneously, copies are sent to the Congress—even if it was not among the requesters—along with offers to brief congressional staff and Members. In addition to the report copies, a two to four-page report highlights is prepared. Also, the National Academies press office prepares a press release or advisory for public release. The latter occurs after the sponsor and the Hill have received copies. Finally, members of the study committee—usually the chair—provide the briefings to the sponsor and the Hill if at all possible. In addition, often they end up testifying.

Once all this is done, however, further follow-up is limited. Updates, while not forbidden, do require a separate request and funding. Members of the study committee sometimes do get asked to brief Congress and/or agencies long after the report is out, but any updates at those times are only the opinion of the person doing the briefing.

To the extent National Academies studies can be compared to those of CRS and to OTA studies that are still relevant, I think it is clear that because of the Academies heavy reliance on outside expertise, they are the usually the most subject matter detailed of the three organizations.  They are less comprehensive than OTA studies were or that CRS studies have the potential to be, however, in consideration of non-technical aspects such as economic, environmental (except for those studies explicitly focused on environmental issues), or other societal impacts, and do not usually have the policy analysis coverage of either OTA or CRS studies. They usually take a long time, however, and they are not always written at a level to be of immediate use to lay audiences, although dissemination efforts have improved substantially over the last several years to address that problem. In addition, they are the only ones that include findings and recommendations. There has been much discussion at the National Academies about the extent it should get into policy recommendations but that debate has not been resolved.

Dr. Richard Rowberg is at the National Academies where he has been since 2002.  Prior to that, he worked for 16 years at the Congressional Research Service and before that for 10 years at the Congressional Office Technology Assessment.  He has a Ph.D. in physics from UCLA and spent five years on the engineering faculty at the University of Texas at Austin.

Notes  

[i] For more information on this topic see, Congress’s Own Think Tank: Learning from the Legacy of the Office of Technology Assessment (1972–1995), Peter Blair, Palgrave MacMillan, 2013, pp.76-80.