A computer program for administering and scoring confidence tests

In confidence-test procedures, subjects are asked to respond to all the options of multiple-choice items with weights. They may assign the entire weight to a single option or they may distribute their confidence. With Shuford's truncated logarithmic scoring function (Shuford, Albert, & Massengill, 1966), incentives are provided for honest reporting of confidence, and guessing is discouraged. By utilizing adjustments based on least squares fitting of the subject response (r) to the performance-based likelihood vector (jj), a score of realism can be computed. The realism score indicates whether or not the subject was appropriately certain or cautious (Brown & Shuford, 1973). Adjustment of the knowledge score for inappropriate realism improves reliability and validity (Rippey & Voytovich, 1983). In addition, feedback from the realism score can lead to improvements in realism or suggest deficits in basic forecasting skills and knowledge (Yates, 1982). The logarithmic function has identified occasional bizarre behavior of respondents, suggestive of a pathology of reasoning (Rippey & Voytovich, 1985). The logarithmic method of scoring is used rather widely in England in a course in risk analysis offered by the Open University (1980). Because of its amenability to the scoring of traditional multiple-choice test items, the logarithmic function has been applied to studies of cognitive achievement. Other scoring functions have also been used in other studies, especially in connection with forecasting (Blattenberger & Lad, 1985; Yates, 1982). The Brier score, for example, has been widely studied because of its partitionability using components of variance methods. Confidence testing initially suffered from problems of administration and scoring (Ebel, 1968). Many of the problems were solved by the use of mainframe computer scoring (Rippey & Donato, 1978). A PLATO version of confidence testing was developed by the Rand Corporation (Landa, 1976), expanded by Rippey and Smith (1979), and improved by Anderson (1982). However, access to the system is much more immediate with microcomputers; the microcomputer program described here is a complete system for preparing tests and scoring keys, and for administering and scoring the tests, either individually or in batch mode. The individual testing procedures are most appropriate for educational use,