Ch. 12
Evaluating Decision Support Systems Projects

Chapter Contents Previous Page Next Page

One can categorize DSS project evaluation issues into 4 general areas: Economic, Operational, Schedule, and Technical. In a DSS project evaluation, these general criteria need to be considered. Let’s examine the 4 criteria in more detail.

Based on Whitten, Bentley and Barlow (1994), we can define four evaluation "tests":

Economic Test – a measure of the cost-effectiveness of a project or solution. This is often called a cost-benefit analysis. This test was discussed extensively in the prior section.

Operational Test – a measure of how well the solution of problems or a specific solution will work in the organization. It is also a measure of how people feel about the Decision Support System proposal.

Schedule Test – a measure of the reasonableness of the project timetable.

Technical Test – a measure of the practicality of a specific technical solution and the availability of technical resources and expertise. In some DSS proposals technical issues are the major risk concern.

Which of the criteria should be the focus at various project evaluation stages? The initial evaluation should focus on the project need and the anticipated benefits. The focus should be on the operational test. As the project evaluation continues more feasibility issues need to be evaluated and the benefits need to be assessed more carefully to insure that project advocates are not inflating benefits and minimizing problems. The economic test may be revisited a number of times, but it should be a major part of a feasibility analysis.

As noted in Chapter 2, DSS projects have various levels of risk associated with them. When DSS projects have ambiguous objectives and low structure, the projects have higher levels of risk because the costs and scope of work of the project are hard to define. The schedule and technical tests are very important for high risk projects. Also, because the objectives of the project are ambiguous, it can be difficult to assess the return on the investment. When returns are hard to assess more qualitative economic analyses are used. DSS projects with a higher degree of structure and more clearly defined objectives generally are lower risk. More detailed planning is possible for projects with specific objectives. The size or scope of a DSS project in terms of the number of users served and the size of databases developed also impacts the risk of the assessed projects. Small DSS projects in terms of scope or dollar expenditures tend to be of lower risk than large projects. Finally, the sophistication of the technology and the experience of the developers using the technology influence the overall project risk. The ultimate decision to invest in a DSS project should not be based solely on project risk. As noted in the discussion of gaining competitive advantage with innovative DSS projects, the project that is most likely to result in a competitive advantage is sometimes the riskiest project.

In all evaluations, one needs to consider the down stream affects of short-term decisions that have been based solely upon short-horizon cost savings. DSS may reduce some costs, but that is not usually the motivating factor for a new system. No DSS project decision should be made in isolation. Even small projects can sometimes have million dollar impacts. It is important to broadly examine DSS project impacts. Once a DSS project is completed managers need to follow-up and periodically evaluate what is working well with the system and why and what problems are being encountered.