THE PROCESS AND ACCURACY OF CLINICAL JUDGEMENT:Comparison Studies

<< THE BEHAVIORAL ASSESSMENT THROUGH INTERVIEWS, INVENTORIES AND CHECK LISTS

METHODS OF IMPROVING INTERPRETATION AND JUDGMENT >>

Clinical Psychology (PSY401)

LESSON 23

THE PROCESS AND ACCURACY OF CLINICAL JUDGEMENT

As scientific and objective as clinical psychology has tried to become, it is still virtually impossible to

evaluate its diagnostic and assessment techniques 'apart from the clinician involved. Clinical

Judgment," is enough to suggest that clinicians use inferential processes that are often far from

objective. The process, accuracy, and communication of clinical judgment are still very often extremely

personalized phenomena.

In the discussion ahead we will examine some of the means by which the clinicians put together

assessment data and arrive at particular conclusion. We will also discuss the accuracy of clinical

judgment and impressions.

PROCESS AND ACCURACY

The discussion of clinical judgment will begin with its basic element----Interpretation.

INTERPRETATION

It is hard to disagree with L. H. Levy's (1963) statement that "Interpretation is the most important single

activity engaged in by the clinician". Interpretation is an inferential process (Nisbett & Ross, 1980)

that takes where assessment leaves off. The interviews have been completed; the psychological tests have

been administered. Now, what does it all mean, and what decisions are to be made?

At the very least, clinical interpretation or judgment is a complex process. It involves

1. Stimuli----an MMPI-2 profile, an IQ score, a gesture, a sound, etc.

2. It also involves the clinician's response. "Is this patient psychotic?" "Is the patient's behavior

expressive of a low expectancy for success?" Or even "What is the patient like?"

3. It also involves the characteristics of clinicians their cognitive structures and theoretical orientations.

4. Finally, situational variables enter into the process. These can include everything from the type and

range of patients to the constraints that the demands of the setting place on prediction$. For

example, a clinician in a university mental health center may make a range of judgments from

hospitalization to psychotherapy to just dropping out of school---whereas a clinician in a prison

setting may be limited to many fewer options.

THE THEORETICAL FRAMEWORK

As we know that clinical psychologists strive to discover the etiology, or origins, of psychological

problems and to understand patients so that they can be helped. Clinical problems can be

conceptualized in a variety of ways (for example, psychodynamic, behavioral, and cognitive). The kinds

of interpretations made by a Freudian are vastly different from t h o s e m a d e by a behavioral clinician.

Two clinicians may each observe t h a t a chiid persistently attempts to sleep in his mother's bed.

For the Freudians, this becomes a sign of an unresolved Oedipus complex. For the behaviorist, the

interpretation may be in terms of reinforcement.

Indeed, one way in which clinicians can evaluate interpretations is by examining their consistency with

the theory from which they are derived. The number of interpretations that can be made from a set of

observations, interview responses, or test data is both awesome and bewildering. By adopting a

178

Clinical Psychology (PSY401)

particular theoretical perspective, clinicians can evaluate interpretations and inferences according to

their theoretical consistency and can also generate additional hypotheses.

SIGNS, SAMPLES AND CORRELATES

Patient data can be viewed in several ways.

1. Samples: First, one can view such data as samples. Observations, test scores, test responses, or

other data are seen as samples of a larger pool of information that could be obtained outside the

consulting room. For example, when a patient does poorly on the Wechsler Memory

Scale, this could be regarded as a sample of nontest behavior (memory problems).

2. Signs: A second way in which patient data can be interpreted is as signs of some underlying

state, condition, or determinant. Aside from radical behaviorists, many clinicians will seek to

infer from observations of the patient's behavior and test responses a variety of underlying

determinants. For some clinicians, the underlying determinant might be anxiety, for others, ego

strength; and for still others, expectancies. But in every case, the observation is seen as something that

signifies underlying determinants. For example, poor on a patient's Rorschach responses is often

interpreted as a sign of poor reality testing (psychosis).

3. Correlates: A third view of patient data emphasizes their status as correlates of other things. Once

the anxious behavior, the flat affect, or the inability to concentrate have been noted in a depressed

patient, the clinician might predict an associated decline in sexual activity, in social relationships,

in willingness to seek employment, and so on. In effect, then, assessment data can be inter-

preted to suggest behavioral, attitudinal or emotional correlates.

LEVELS OF INTERPRETATION

Whether clinicians view clinical data as samples, signs, or correlates, they are making inferences that will

enable them to go from those clinical data to recommendations, reports or predictions. Sundberg, Tyler,

and Taplin (1973) have described three levels of inferences or interpretations.

LEVEL 1 interpretation generally involves little in the way of inference and certainly nothing in the way

of a sign approach. From input to output, there are practically no intervening steps. For example, if it

is known on the basis of past experience that students who sit in the front row of a class almost always get A's

or B's, then clinicians can go directly from seat number to grade prediction without any necessity for

intervening attributions of intelligence scores, previous courses, and so on. This simple yet efficient ap-

proach can dispense with high-level clinicians who make exotic inferences prior to their predictions;

it can be handled by technicians, computers, or machines. Level I interpretations can often be used with

large populations if the prime purpose is screening and if predicting the outcome for a specific person is

relatively unimportant. A college entrance exam is a case in point. Here a single test score may predict

with considerable accuracy the academic performance of 1,000 freshmen. Although that single score may be

erroneous as a predictor for student X, a certain degree of error can easily be tolerated if one is interested-

primarily in the number who is likely to graduate.

LEVEL II interpretations involve two kinds' inferences. The clinician may observe a patient and then

conclude that the observe behavior generally characterizes the patient. Sundberg etal call this first kind of

inference descriptive generalization-----still at the descriptive level. Thus, for a patient who fidgets,

smokes cigarettes during the interview, and stammers the clinician may make a descriptive generalization---

-interview tension. If it turns out later that the patient has trouble relaxing at home, cannot sit through the

meeting at the office. And is very worried about paying off the mortage, the clinician may go to a broader

descriptive generalization. The second kind of inference is a hypothetical construct that suggests an inner state

and takes the clinician a bit beyond descriptive generalaization.When clinicians begin to make generalization

179

Clinical Psychology (PSY401)

and particularly, to impute inner determinants to the patient, they are moving directly to clinical interpretation

as it is often used.

LEVEL III interpretations take clinicians beyond level II primarily by being more inclusive and better

integrated. at this level, they attempt to achieve a consistent, broad understanding of the `individual in

situation', clinician will draw and integrated picture of the patient's developmental, social, and

psychological determinants that involves a highly articulated theoretical system of hypothesis and

deductions. For example, a preponderance of `blood' responses on the Rorschach might be interpreted as a

sign of underlying aggression that may lead to future impulsive outbursts or loss of control.

THEORY AND INTERPRETATION

Currently, clinicians may be assigned to three very broad interpretive classes.

BEHAVIORAL CLINICIANS

First, there are the behavioral clinicians. The strict behaviorist avoids making inferences about underlying

states and instead concentrates on the behavior of the patient. The behavioral clinician typically seeks

patient data based on personal observation or on direct reports from the patient or from the other observers.

These data are regarded as samples. Interpretation is Largely at Level I and II, although more recently some

behavioral clinicians have begun to show an interest in Level III interpretation.

PSYCHOMETRIC APPROACH

A second group of clinicians pride themselves on being empirical and objective. In particular, these

clinicians are likely to use objective tests to predict to relatively specific criteria. For example, will scores

from tests A, B, and C predict success in college, therapy outcome, or aggressive out bursts? This

psychometric approach to interpretation, as we shall see a bit later, is especially useful when the criteria

being predicted are crisp and well articulated. In general, this approach uses data as correlates of

something else-for example, a score at the 95th percentile on test X may be related to recidivism in

prisoners. The psychometrically oriented clinician is most concerned with standardized tests and their

norms, regression equations, or actuarial tables, and tends to employ_ Level I and II interpretation.

PSYCHODYNAMIC APPROACH

A third group of clinicians is more comfortable with a p s y c h o d y n a m i c a p p r o a c h . This has long been

a popular orientation in clinical psychology. Although current clinicians often seem to opt for a more

objective behavioral or psychometric approach, there is still more of the psychodynamicist in many of

them than they might like to admit. The psychodynamic approach strives to identify inner states or

determinants. Data from projective tests, unstructured clinical interviews and other sources are viewed as

signs of an underlying state. Interpretation tends to itched at Level III. A broad, often highly

impressionistic picture of the patient is drawn, although in many instances subtle normative assertions are

made.

QUANTITATIVE VERSUS SUBJECTIVE APPROACHES

Quietly embedded in the preceding discussion are two distinct approaches to clinical judgment and

interpretation. First is the quantitative or statistical approach, which emphasizes objectivity and is

presumably free from fuzzy thinking. Second is the subjective or clinical approach, which adherents

claim is the only method to offer truly useful interpretations and predictions.

THE QUANTITATIVE STATISTICAL APPROACH

Perhaps the simplest form of quantitative prediction that clinicians can use involves_ assigning scores

180

Clinical Psychology (PSY401)

to the various characteristics of their patients. This enables clinicians to determine the correlation

between any two characteristics. For example, suppose that after several years of practice, a clinician

begins to suspect a direct relationship between early termination of therapy and patients' needs for

independence. The clinician might attempt to verify this hypothesis by correlating "need for

independence" scores from a self-report inventory with the length of time that patients remain in

therapy. Should the correlation turn out to be substantially above .50, the clinician could use need

for independence scores to make interpretations and predictions regarding the duration of therapy.

Of course, more often than not, one cannot base important predictions on a single score or attribute. The

conclusion of therapy is more often a complex event that has a number of determinants.

Consequently, the clinician might want to obtain scores on several other variables, such as ego

strength, the experience of the therapist, marital satisfaction, and interpersonal trust. A multivariate

prediction model could then be constructed and tested. A particular caution to bear in mind,

however, is that even though a mu ltiple correlation from such an analy sis may turn out to be

quite high, it may well be much lower when applied to a new sample. This is especially true if the

original sample is small and the number of predictors is large. Further, the sample on which the

initial study is carried out may not be representative of therapy patients in general. What is true in

Kansas may not be true in California; what is true for psychoanalytic therapy may not be true for

behavior therapy. Therefore, clinicians have to be sure that they have correctly weighted various predictor

scores before they can generalize very far. They must cross-validate their prediction models using other

samples.

These statistical techniques permit a mechanical application that does not involve clinical decision

making at all once the formulas have been established. The feature that distinguishes these statistical

approaches from clinical approaches is that the former (no matter what their complex mathematical

development), once established, can be routinely applied by a clerk or a computer.

The quantitative, statistical approach, then, requires that the clinician keep careful records of the data,

observation and related material so that clinical interpretations and judgments can be quantified. Such

careful record keeping will permit the clinician to go beyond informal impressions based on previous

experience. With adequate records on large enough samples, the relations among a host of variables can

be assessed. Whether clinicians are evaluating their own performance or the performance of an entire

clinic, or are relating certain patient characteristics to various diagnostic or therapeutic outcomes,

quantified data can play a facilitating role. Such data enable clinicians to evaluate their judgment,

interpretations, and performance.

THE SUBJECTIVE CLINICAL APPROACH

The clinical approach is much more subjective, experiential, and intuitive. Here, subjective weights based

on experience suffice. The emphasis is on the application of judgment to the individual case. The classical

notation is that "clinical intuition" is not readily amenable to analysis and quantification It is a private

process in which clinicians themselves are sometimes unable to identify the cues in a patient's test responses

or verbalizations that led them to a given conclusion or judgment.

Once, for example, in the course of a Rorschach administration, a patient said, "This looks like a

Christmas tree." What did this mean? Perhaps nothing. Or perhaps it indicated a career in forestry. Or

perhaps it suggested an underlying sadness or depression in a person with few friends or family with

whom to enjoy the approaching holiday season. In this case, the last interpretation was later supported by

the patient during a discussion of his family background. The clinical student who had made the

correct interpretation in a training exercise explained her reasoning as follows: "It was near the Christmas

season; there were several references in the TAT to remote family figures; I remembered how I always

seem to become a little sad during Christmas; it suddenly popped into my head, and I just knew with

complete certainty that it was true-it simply felt right!"

181

Clinical Psychology (PSY401)

This example illustrates several things about clinical interpretation. First, such interpretation involves a

sensitive capacity to integrate material. The astute clinical psychologist pays attention to the wide range

of events that characterize the patient's behavior, history, other test responses, and so on. A clinician

must function a bit like the detective who takes in everything at the scene of the crime and then

makes a series of inductive or deductive generalizations that link these observations together. In

addition, there is often a willingness in the clinician to see a bit of him or herself in the patient-----a

kind of assumed similarity that enables the clinician to utilize his or her own experience in

interpreting the behavior and feelings of another.

Unfortunately, the presentation of this example has been one-sided. Little has been made of the

clinical student who believed that the Christmas tree suggested an interest in forestry .

Therefore, we may make two additional observations. First, there are individual differences in clinical

sensitivity. Second, for every instance of brilliant and sensitive clinical inference, there probably lurks

in the unrecalled recesses of memory an equally impressive misinterpretation.

Clinical interpretation, then, involves the sensitive integration of many sources of data into a coherent

picture of the patient. It also fulfills a hypothesis-generating function that is best of personality. But it

behooves responsible clinicians to make every effort to articulate the cues involved in their judgments

and to explicate the manner in which they make the leap from cues to conclusions. It is not enough to

be good clinicians. There is also a responsibility to pass on these skills to others.

COMPARING CLINICAL AND ACTUARIAL APPROACHES

Over the years, many studies have compared the relative accuracy of clinical and actuarial methods. Let

us now examine some of that work.

Comparison Studies

Sarbin (1943) contrasted the prediction of academic success of college freshmen made by a clerk

employing a regression equation with the predictions made by several counselors. The regression equation

predictors were aptitude test scores and high school rank. The counselors had available to them

the two preceding sources of data (but without their mathematical weighting), vocational interest

scores, interview data, and biographical data. Sarbin (1943) found that the counselors were no better than

the regression equation in their predictions even though they had the benefit of much mere

information.

Meehl (1954) surveyed a number of the studies available on clinical versus statistical, prediction and

concluded that in "all but one ... the predictions made actuarially [statistically] were either

approximately equal or superior to those made by a clinician" .In a later survey of additional research,

Meehl (1965) reaffirmed his earlier conclusions. However, Meehl (1954) also observed that, in several

studies, statistical predictions were made on the same data from which the regression equations

were developed. In short, the formulas were not cross-validated. Such formulas frequently show a

marked reduction in efficiency when they are applied to samples different from those used in their

derivation.

Sawyer (1966) regarded data collected by interview or observation as clinical data. He viewed

inventory, biographical, or clerically obtained data as statistical or mechanical. Having considered the

methodological problems and the equivocal results of the studies he examined, Sawyer concluded that

in combining data the mechanical mode is superior to the clinical mode. However, he also

concluded that the clinical method is useful in the data collection process. The clinical method can

provide an assessment of characteristics that would not normally be assessed by more mechanical

techniques of data collection. But once the data (from whatever source) are collected, they can best be

combined by statistical approaches.

182

Clinical Psychology (PSY401)

An example of an individual study comparing clinical and statistical prediction may help further

illustrate the nature of this controversy.

One of the most frequently cited studies of clinical versus statistical prediction was reported by

Goldberg(1965) .in this study, 13 PhD level staff-members and 16 predoctoral trainees were asked to

make judgments regarding the diagnostic status of more than 800 patients, based on these patients'

MMPI scores. These judgments were made without any contact with the patient or any additional

information on the patient. Each judge simply examined the MMPI profile (scores) for each patient

and then predicted whether the patient was "psychotic" or "neurotic." These judgments constituted

clinical predictions because it was left up to each judge as to how she or he used the MMPI information

to formulate a diagnosis.

In contrast, statistical predictions involved the application of a variety of algorithms, in which MMPI

scale scores were combined (added or subtracted), in some manner and previously established cutoff

scores for psychosis versus neurosis were used. In addition, some statistical predictions involved the

application of specified decision rules based on MMMPI high point codes or other psychometric signs.

A total of 65 different quantitatively based rules were considered.

What were these clinical and statistical predictions compared to in order to assess their accuracy? In

this study, the criterion diagnosis was the psychotic versus neurotic diagnosis provided by each

patient's hospital or clinic. Thus, the accuracy of each clinician's and each statistical algorithm's

prediction was determined by assessing the agreement between predictions and the actual criterion

diagnoses across all cases.

A variety of additional, updated reviews of the studies pitting clinical versus statistical prediction have

uniformly-demonstrated the superiority of s t a t i s t i c a l procedures (for example, Dawes, 1979, 1994;

Dawes, Faust, & Meehl. 1989; Garb, 1998; Goldberg, 1991; Kleinmuntz, 1990; Meehl, 1986; Wiggins,

1973). As stated by Meehl (1986):

Objections to These Findings

Dawes (1994) has outlined several of the major objections to large body- of evidence supporting the

superiority of statistical prediction, along with response, such objection.

First, critics argue that several of the individual studies reviewed contained research design flaws that may

have affected the findings. Dawes (1994) refers to this an "argument from a vacuum"-a possibility is

raised, but there is no empirical demonstration supporting the possibility. Although every study has its

limitations, it is difficult to imagine that the opposite conclusion (clinical prediction is superior) is

warranted when practically all of the studies support statistical prediction.

The second objection concerns the expertise of the judges/clinicians in these studies. Perhaps they were

not "true" experts, and a study employing expert clinicians would demonstrate the superiority of

clinical judgment. Although a wide variety of judges/clinicians were used in these studies, a number

of studies employed recognized "experts"-clinicians with many years of experience performing the

predictive task in question. There were a few instances in which an individual clinician performed as

well as the statistical formula, but this was more the exception than the rule. Thus, there is no

compelling empirical evidence that "expert" clinicians are superior.

A third objection is that the predictive tasks were not representative of prediction situations facing

clinicians (that is, not ecologically valid). A clinician's diagnosis may not be based only on the MMPI-2,

for example, but also on an interview with the patient. Dawes (1994) argues, however, that the

predictive tasks are components of what may go on in clinical practice clinicians purportedly use

the MMPI-2 information to make predictions. Further, several of the studies demonstrate that additional

information (such as interview material) obtained and used in the judge's clinical prediction may actually

183

Clinical Psychology (PSY401)

result in less accurate predictions than would be the case if the clinician had simply "stuck with

the statistical formula that was available.

Dawes (1994) goes on to suggest that much of the negative reaction to the findings is a function of our

human need to believe in a high degree of predictability in the world. This appears to be both a

cognitive and an emotional need. People have a built-in tendency to both seek and see order in the

world, and a lack of predictability in the world is likely to result in some degree of discomfort or

emotional distress. However, the need for predictability does not prove its existence.

BIAS IN CLINICAL JUDGMENT

Clinical judgment suffers when bias of any kind intrudes into the decision-making process. Bias exists

when accuracy of clinical judgment or prediction varies as a function of some client or patient

characteristic, not simply when judgments differ according to client characteristics (Garb, 1997, 1998).

For example, finding that a higher percentage of women than men are judged to suffer from major

depression would not indicate a bias against women. However, finding that a higher percentage of women

than men are given this diagnosis when the same symptoms are presented would indicate bias.

Garb (1997) recently reviewed the empirical evidence for race bias, social class bias, and gender bias in

clinical judgment. Interestingly, he found that many conventionally held beliefs about these types of

bias were not supported. For example, there was little support for the beliefs t h at

( 1 ) l o w e r - so c i o e c o n o m i c - c l a s s p a t i e n t s a r e judged to be more seriously disturbed than those from

higher socioeconomic classes or

(2) Women patients are judged to be more disturbed or dysfunctional than men patients. However, there

was strong evidence to support the existence of several other types of biases:

1. Black and Hispanic patients who have psychotic mood disorders are more likely to be

misdiagnosed with schizophrenia than are similar White patients.

2. Even when presenting the same constellation of symptoms, men are more likely to be diagnosed

as antisocial and women are more likely to be diagnosed as histrionic.

3. Middle-class patients are more likely to be referred for psychotherapy than lower-class patients.

4. Black patients are more likely to be prescribed antipsychotic medications than members of other

racial groups, even when the Black patients are not more psychotic.

Garb (1997) made the following recommendations to help clinicians overcome these and other bias

(1) Be aware of and sensitive to the biases that have been documented in the literature.

(2) Attend to the diagnostic criteria in diagnostic manuals.

(3) Whenever possible, use statistical prediction rules instead of clinical judgment or prediction

EXPERIENCE AND TRAINING

To date, empirical evidence does not support the position that increased clinical experience results in

increased accuracy in prediction (Dawes, 1994; Garb, 1989, 1998). This seems to fly in the face of

conventional wisdom. Why is it that we do not see evidence for the effect of clinical experience in

clinical psychology and other mental health fields? There are several possibilities (Dawes, 1994).

First, the accuracy of predictions is limited by the available measures and methods that are used as aids

184

Clinical Psychology (PSY401)

in the prediction process. If scores from psychological tests, for example, are not strongly correlated with

the criterion of interest (that is, highly valid), then it is unlikely one could ever observe an effect for

clinical experience. The accuracy of predictions will remain modest at best and will not depend on how

"clinically experienced" the clinician is.

Second, we often cannot define precisely what we are trying to predict (for example, "abusive

personality"), and no gold standards for our criteria exist to enable us to assess objectively the accuracy of

our predictions. As a result, true feedback is impossible, and diagnosticians are not able to profit from

experience.

Third, we tend to remember our accurate predictions and to forget our inaccurate ones. Therefore,

more experience in the prediction process does not necessarily lead to increased accuracy because the

feedback that is incorporated is incomplete.

As for the virtue of receiving specific types of professional training, there is not much evidence to

suggest that one profession is superior to another in making accurate diagnostic judgments. For

example, even in differentiating psychological symptoms that are masking medical disorders from those

without underlying medical disorders, medical and non medical practitioners did not differ in their

accuracy (Sanchez & Kahn, 1991).

All of this research is somewhat sobering for the field of clinical psychology. However, it is our

professional responsibility to be aware of the limits of our predictive ability and not to promote the "myth

of experience." One thing is sure. Clinicians will continue to make decisions-they have no choice.

The important thing is to ensure that clinical psychologists are as well prepared as' they can be, as well

as to train clinical psychologists to use the-best available measures and techniques for a given prediction

situation.

CONCLUSION

Given the current state of affairs, the following conclusions regarding the relative strengths of clinical and

actuarial methods seem warranted.

The clinical approach is especially valuable when:

1. Information is needed about areas or events for which no adequate tests are available. In this case,

the research fails to offer any evidence that the data-gathering function of the clinician can be

replaced by a machine.

2. Rare, unusual events of a highly individualized nature are to be predicted' or judged. Regression

equations or other formulas cannot be developed to handle such events, and clinical judgment is the

only recourse.

3. The clinical judgments involve instances for which no statistical equations have been developed.

The vast majority of instances, in effect, fall into this category. The day-to-day decisions of the

clinician are such that the availability of a useful equation would itself be a one and unusual event.

4 . T h e role of unforeseen circumstances could negate the efficiency of a formula. For example, a

formula might very easily outstrip the performance of a clinician in predicting suitability for hospital

discharge. In the role of data gatherer, however, the clinician might unearth important data from a

patient that would negate an otherwise perfectly logical statistical prediction.

The statistical approach is especially valuable when:

185

Clinical Psychology (PSY401)

1. The outcome to be predicted is objective and specific. For example, the statistical approach

w i l l b e e sp e c i a l l y e f f ec t i v e i n p r ed i c t i n g grades, successful discharge, vocational success,

and similar objective outcomes.

2. The outcomes for large, heterogeneous samples are involved, and interest in the individual case is

minimal. Having a statistical formul a to predict how many of 50,000 men will receive

dishonorable discharges from the Army will be highly useful to the Army, though less so

for the clinician who is dealing with Private Smith.

3. There is reason to be particularly concerned about human judgmental error or bias. Fatigue,

boredom, bias, and a h o st o f o t her h u man failings can be responsible for clinical error.

Oft en , su ch effect s are ran do m 'a n d u np redictable. Formulas, equations, and computers

never become tired, bored, or biased.

186

Table of Contents: