Developing a Portfolio Approach to Capital Investment: A Case Study in Re-Engineering Resource Allocation at the U.S. Department of Veteran's Affairs
Software and methodology helped the Veteran's Administration link its funding requests to the agency's strategic objectives, resulting in dramatic time savings and greater efficiency.
By O. John Wasyluk and Daniel Saaty
Prior to 1997, the Veteran's Administration (VA) largely developed its project priorities manually in table discussions with little structure. Decision makers considered strategic objectives but established no explicit link between strategic priorities and funding requests. The manual process resulted in a priority list that lacked an audit trail describing how projects were ranked.
In 1997, VA decision makers discovered an off-the-shelf software prioritization tool based on the Analytic Hierarchy Process (AHP) method of decision making. Through the use of AHP and decision support software, VA budget officers were able to instill discipline and consistency to their capital investment process. Over the next three years, the VA designed its organization to 1) review a variety of investment options from construction projects to information technology projects and 2) standardize the application for capital investments and develop a capital investment methodology comprised of formulation and execution reviews. Formulation review is comprised of three phases: functional development, technical review, and strategic review. The functional development phase is at the field level, where needs are realized, gap analysis completed, application proposals developed, and solutions ultimately applied. In the technical review phase, proposals receive early rounds of technical and financial scrutiny from department-wide councils or boards, as well as some initial prioritization within the owner organization. Finally, the strategic review phase includes a two step review process and prioritization by the VA Capital Investment Board (CIB). The Execution Review includes the confirmation phase and occurs once funds are appropriated and included in the president's budget. Planning assumptions for these funded proposals were made 18-24 months earlier. For example, prior to the obligation of funds, the VA Capital Investment Board (VACIB) will review the upcoming fiscal year's obligations to assure that the proposed investment is still needed and that the planning assumptions used are still valid VACIB prioritizes the projects into a single list according to how they contribute to strategic objectives.
Background
As with most other agencies in the federal government, the VA is a stovepiped organization with a variety of officers annually competing for scarce resources. The competing, executives include members of the Veterans Health Association (VHA) the Veterans Benefits Association (VBA), the National Cemetery Administration (NCA), and staff offices.
In 1997, executives at the U.S. Department of Veterans affairs set out to meet several legislative mandates to improve their decision making process for capital in vestment Legislation such as the Government Performance
Results Act (GPRA) of 1993, the Federal Acquisition Streamlining Act (FASA), OMB's Capital Planning Guide, and the Clinger-Cohen Act put pressure on government agencies to improve their ability to audit their program and project evaluations. The GPRA sets out general guidelines requiring agencies to develop strategic plans, evaluate programs based on internal and external goals, and set measurable performance improvement milestones as key components of every budget submission. Clinger-Cohen more specifically focuses on measuring performance and effectively selecting and controlling information technology investments. FASA and the OMB Capital Planning Guide provide specific guides on capital planning decision-making.
The VA responded to these mandates by developing a capital investment methodology and identifying ways to formalize project evaluation. VA executives identified the AHP decision method as the most effective approach to 1) prioritizing objectives and projects, 2) meeting their requirements to automate the decision process, and 3) providing ability to track the evaluation and scoring of projects. The AHP uses a mathematical algorithm to weight the hierarchy according to decision makers' priorities. The hierarchy of weighted goals is then used to determine each project's value. Projects are rated against goals that add value Then the ratings are multiplied by the weighted goals and summed to develop a "project priority." The process is powerful not only for ranking projects and investments, but for determining the relative value of investments against the organization's goals and risk factors. Although the process is simple, VA decision makers found that the mathematics used to derive weights for objectives is rather involved. They determined that the agency needed to automate the decision process using software. They began by hiring a contractor to build an AHP software tool and quickly found that software development was more than just coding. The VA professionals needed the AHP tool to group inputs from conflicting decision makers, then synthesize all votes to develop corporate priorities. No decision maker at the VA would accept a process that did not include stakeholder judgments and priorities.
Implementing TeamEC
In looking for a solution that would accomplish these tasks, the VA came across Expert Choice, a firm that offers a group-enabled software version of the AHP. Team Expert Choice, or TeamEC, is a software solution that is equipped with keypads for multiple voters. In the interim, VA established the CIB and a Capital Investment Panel to review all capital investments. These VA decision makers (representing the various stakeholder groups within the agency) were gathered and presented with a keypad.
Decision makers were walked through a set of pairwise comparisons where they were asked to compare goals for their relative importance to the VA. For example, executives were asked to compare "return on investment" (ROI) to "improving customer service" to determine which contributed more to the success of the organization. Voters were then asked to discuss their positions with others. They shared what they did and did not know and their stakeholder priorities. After discussing the priorities, voters were permitted to change their votes. Subsequently, they were asked to compare ROI to "Risk" and then Risk to customer service. The software calculated the votes' geometric average for each comparison, then the averages were used to compute the AHP priorities. The AHP algorithm places the ratios of each pair of factors' relative importance into a matrix. This matrix is then multiplied by itself and the eigenvector, or priority of the goals at a given level of hierarchy, is computed. The weighted goals were then used to rate investment options for their contributions to improving performance in the VA. In addition to providing consensus, the process also had a positive impact on the group's dynamics. For the first time, senior executives left a planning meeting in consensus about what was important to the organization and how the investments would enable the VA to achieve the strategic goals.
The decision model was delivered to the project evaluation team on a laptop with radio frequency keypads to enable the team to efficiently rate each project against the objectives to which it contributed. The software allowed decision makers to use both quantitative and qualitative information to rate competing investments. For intangible objectives like improving customer service, it provided the capability to build adjectival scales such as outstanding, very good, good, and so forth. The ratings are weighted and based on the value that decision makers perceive for each rating. If numerical performance data is available, the raw data is converted to a utility score between 0 and 1, with 1 being the highest possible data point. Evaluators can then build a utility curve to determine the value of any data point in the range.
VA evaluators entered ratings using thekeypads, then were given the opportunity to discuss their scores. One of the ground rules was that ratings had to be sufficiently close to move to the next judgment. Once the ratings had been applied to the projects the software produced a final priority list justifying the incremental strategic benefits of each project.
Continued Improvements
Although the VA continued to evaluate information technology (IT) and construction priorities separately throughout 1998, it still benefited considerably from automating the capital investment process. The VA reduced the time for project evaluation to a few days versus what used to be an endless process of submission and review. They also found that ill-defined projects tended to receive lower scores, so staff proposing new projects put more effort into developing project descriptions and providing primary source documentation to support risk analyses, alternatives development, and strategic evaluation. The software also enabled decision makers to use a graphical sensitivity feature to dynamically change the weights of performance goals. This allowed them to instantly see how changing performance goals' weights would impact project priorities.
In 1999, after implementing 20 of 28 recommendations compiled in a "best practices" survey, the VA finally integrated IT and construction into a single software model. While some projects did not address every performance goal in the hierarchy, they were scored based on which priorities they contributed to, how much they contributed, and how
important those areas were to the VA. Later that year, the OMB reviewed the VA's and other agencies' progress in implementing new management techniques to ensure acquisition programs achieved their intended results.
The report titled, "Annual Report to Congress on Implementing the Federal Acquisition Streamlining Act," covered 15 major civilian agencies and 185 individual acquisitions. In this report, the OMB ranked the VA as having made "Excellent Progress." The OMB went on to say that the VA "is the first civilian agency to develop an agency-wide capital planning process that allows for investment trade-offs among categories of assets, such as medical and non-medical equipment and infrastructure and IT."
The VA reviews and validates its process and decision model annually in a "Lessons Learned" conference to get feedback from the field and to ensure it is current with any legislative mandates or changes to the strategic goals. In 2000, the agency further streamlined the process by instituting a Web site, a technical library, and capital investment guidebook, and developing four electronic Web-based templates to make the process more user-friendly.
Myriad Benefits
The process of improving the VA's capital investment decision making took three years to fully evolve, but decision makers continue to realize additional benefits of using a structured framework. The process resulted in a dramatic reduction in decision making time, better proposals, and overall better selection and control of projects funded based on their contribution to strategic objectives. The success of the methodology was due
largely to the organization's commitment to improve the way it establishes the gap analysis and plans its projects, and the VA expects that the process will continue to improve over time due to the auditable framework they have now implemented for capital investment decisions.
TeamEC/EC 2000 2nd Edition for Groups Screen Shots
Questions for Analysis and Discussion
- What are the formulation review steps in the VA budget process? Please prepare a top-level flowchart of the process.
- What is the AHP decision method? Based upon the case do you feel you understand AHP and how it is implemented in TeamEC?
- How does Team Expert Choice, or TeamEC, collect stakeholder judgments and priorities?
- What were the benefits of using TeamEC for project evaluation and capital investment decision making?
- Would you categorize TeamEC as a communications-driven DSS? Is it also a GDSS? Explain and justify your conclusions.
- What is your assessment of the TeamEC software and its use at the U.S. Department of Veteran's Affairs ?
- Do you anticipate any problems with using TeamEC? If so, explain them.
| About the Authors |
| O. JOHN WASYLUK is a budget analyst in the Capital Budgeting and Oversight Service, Office of Budget, in the Office of the Assistant Secretary for Management. He is one of three principal architects who have reengineered the capital investment process in the Veteran's Administration. The other principals are Jim Sullivan, director of Capital Budgeting and Oversight Service, and Cynthia Krohmal, director of IRM Planning and Acquisition Service. DANIEL SAATY is a co-owner and principal decision consultant with Expert Choice. He has worked with business leaders from companies such as Ford, General Motors, Boeing, and IBM. Send comments on this article to cm@ncmahq.org. |
Jennifer Ugo
O. John Wasyluk and Daniel Saaty, "Developing a Portfolio Approach to Capital Investment: A Case Study in Re-Engineering Resource Allocation at the U.S. Department of Veteran's Affairs", Expert Choice, Inc., 2001, URL DSSResources.COM