By BLR Founder and CEO Bob Brady
Readers Don’t Agree About New Haven Firefighters
Tests?
Two weeks ago we ran my column about the U.S. Supreme Court’s decision in the New Haven firefighters’ case. (The justices overruled lower court decisions, holding that the city was wrong when it refused to use the results of a promotion test because of “adverse impact” on minorities.)
I took issue with the decision, and we included a poll asking you for your views. Of the more than 400 who responded, 68% felt that the justices were right and I was wrong. (So much for my powers of persuasion!) Seventy-six percent of respondents said that they would have certified the results of the exam. Despite that overwhelming sentiment, it was very interesting that only 22% felt that a test is a good way to determine who should be promoted to firefighter officer rank. So, you think that the ruling was correct, but the promotion process was wrong. Is that a contradiction?
Poll Results
Here is a capsule summary of the poll:
| Do you think the Supreme Court decided the New Haven firefighters case correctly? |
| Number | Percent | |
| Yes | 284 | 68% |
| No | 98 | 23% |
| Undecided | 40 | 9% |
| Is a test a reasonable way to decide firefighter promotions? |
| Number | Percent | |
| Yes | 91 | 22% |
| No | 294 | 69% |
| Undecided | 36 | 9% |
| Would you have approved the test results? |
| Number | Percent | |
| Yes | 272 | 76% |
| No | 58 | 16% |
| Undecided | 27 | 8% |
Why the difference?
Judging from these results, and from the comments that were made, it is clear that readers agreed with the justices’ central argument: Once it created the test, the city had an obligation to use it, even though there was evidence of disparate impact against minorities. However, the low number of people who thought that a test was a good way to decide promotions shows concern about the process and explains why there may be room for honest disagreement.
Comments
Many of you sent your thoughts on this controversial case, and it is clear that it touches a nerve. Most of them—even the ones I disagreed with—were thought provoking. Arguments from people who agreed with the decision took a variety of forms. One group said that the development and use of the test created an “expectation” on the part of employees that the city had to honor. Another group said that successful test takers had worked and studied hard, and their success qualified them for promotion. Still another segment saw the decision in terms of “strict construction”—judges interpret law; they shouldn’t make it.
The first argument (expectations) makes sense to me. The second (the deserving test taker) doesn’t. The third (strict construction) is in the middle, but I think that civil rights are highly protected by the law and the Constitution. The intent of the law is clear enough to encompass a different decision, in my (ever-so-unhumble) opinion.
One reader asked if I would want to be operated on by a surgeon from the top of the class, or the bottom. My response: A surgeon’s class standing is not the result of a single test. Nor is it what would be used to evaluate competence once he or she is out of class.
Changed my mind?
Has my mind changed as a result of the poll? Having 3 out of 4 of you disagree with me does give me pause. Bitter experience has shown that, once I get emotionally committed to a cause, I don’t like to let the facts get in the way. However, I’m not persuaded that candidates “deserve” a promotion just because they study for an exam. There are plenty of people who can do well on tests, but lack the common sense and other qualities that make the difference on the job. But I will admit to the real possibility that I’m wrong and will eventually see the error of my ways. I thank you for your input, but I’m not there—at least not yet.
Correction
One correction that I should make to the original column relates to my use of the term “paper and pencil test.” In fact, the test was written and oral, with 60% based on the written portion, 40% on the oral section. However, from every description that I have heard, the oral portion was pretty much the same as the written section. I’ve been told that it was very short, consisting of two or three questions taking 10 minutes—hardly enough to assess a candidate’s leadership ability.
Anyway, that’s my e-pinion. I’d love to hear yours. Rbrady@blr.com
Thanks for the story, the survey and the results. My votes were in the group that believed because the test had been developed it should be used however, I do not believe a paper and pencil test is a true indicator. I hope the city has learned from it’s racist approach. I hope an objective body will be on hand to develop a conclusive all encompassing means to test going forward.
It seems to me that all the arguments over this topic have actually missed the central point. If a test is properly constructed to be a valid predictor of future job performance, and professionally developed evidence can be presented to demonstrate this fact, then it makes little sense to throw the results of the test out simply because you did not like the outcome. This confuses two distinctly different concepts — test validity (which is a scientific concept), and test fairness (which is a social / political concept). Test validity is all about whether the inferences that are going to be made from the test scores (i.e., those who score higher on the test are more likely to perform better on the job) are in fact true, whereas test fairness is more about whether the test results, if implemented, are really meeting social policy needs. The most important fact here is that to my knowledge, there was never any assertion made by the City or anyone else that the validity of the test was in question. And if there were such assertions, then they could haven been easily answered by any one of a number of well established processes that psychologists used to quantify the validity of a test for a particular use (which I understand the city declined to do.)
As the the most current Standards for Educational and Psychological Testing published by the American Psychological Association et. al. states, “It is important to distinguish between evidence that is directly relevant to validity and evidence that may inform social policy but falls outside of the realm of validity.” (p.16)
This distinction has been completely lost here. The City tossed the results because they wanted to avoid a Title VII lawsuit, not because they believed that the test was invalid. Tests are used to predict future performance in a wide range of critical professions such as medicine and law just to name a couple. If we throw out the results of professionally developed, valid tests every time we are not satisfied with the outcome, we might as well eliminate testing for any purpose altogether. This makes no sense to me whatever.