The problem of the validity of judgments of the various traits was considered in a more detailed way by Norsworthy, from whose account of her inquiry we have already quoted. She chose the traits enumerated by Cattell, and performed several experiments to determine the reliability of judgments of this sort. Thus she had five intimate acquaintances independently grade a sixth person for her possession of these twenty-four traits, on two different occasions several weeks apart.
Two things were clearly shown. In the first place the individual judges, in their second trials, did not diverge far from their first ratings. In the second place the double judgments of the five different judges did not diverge far from each other. These two facts "prove that the ratings do stand for some actual quantitative value and are not subject to mere chance. The validity of the judgments, in the sense of their correspondence with the actual character of X is then only a matter of the impartiality of the group of judges."
Similar results were found in the judgments of nine members of a college society by five of their comrades, and in the judgments of a teacher by two hundred college students. It was apparent also that judges differ from one another in the general accuracy of their gradings. Some of them agree closely with the consensus of opinion, while others depart, in varying degrees, from the average or correct estimate. It was also seen that, in estimating certain individuals, judges with presumably equal acquaintance with those being judged agreed closely with one another. Other persons had produced quite different impressions on the different judges and this was revealed in the greater divergence of the grades assigned to such persons.
As in the case of Cattell's results, figures are presented showing the degree of divergence among the judges in estimating the different traits. In the table on page 139 these figures are given, as shown in the records of five judges in one of Norsworthy's experiments, and the records of the twelve judges in Cattell's investigation. The average variability or degree of divergence for all the twenty-four traits is taken as the standard and each trait compared with this standard. A variability of one hundred thus indicates the average amount of disagreement. Figures smaller than one hundred indicate that the agreement was closer than average, and figures larger than one hundred indicate that here the judges disagreed by more than the average amount.
Naturally, there is not perfect agreement in these two cases, since the one set of data is from a group of girls judging one another on the basis of their acquaintance as social comrades and fellow students, while the other set is from scientific men judging one another on the basis of less constant association and largely on acquaintance in lecturing, research, teaching and the writing of articles and books. Moreover, results from groups of only five judges in the one case and only twelve in the other are subject to considerable variable error. In spite of these facts, interesting suggestions are afforded by the comparison.
TABLE 1
Variability in Judging Different Traits
| Trait | Relative Divergence of Different Judges | ||
| Cattell, 12 Judges | Norsworthy, 5 Judges | Average of Both Experiments | |
| Efficiency | 75.0 | 92.4 | 83.7 (Close |
| Originality | 95.2 | 77.2 | 86.2 Agreement) |
| Quickness | 90.0 | 88.0 | 89.0 |
| Intellect | 95.2 | 92.0 | 93.6 |
| Perseverance | 75.0 | 101.0 | 88.1 |
| Judgment | 100.0 | 78.7 | 89.4 (Fair |
| Will | 85.1 | 98.1 | 91.8 Agreement) |
| Breadth | 100.0 | 92.4 | 96.2 |
| Leadership | 90.0 | 102.9 | 96.5 |
| Clearness | 104.9 | 75.7 | 90.3 |
| Mental Balance | 110.2 | 81.8 | 96.0 |
| Intensity | 85.1 | 113.7 | 99.4 |
| Reasonableness | 115.0 | 86.4 | 100.7 (Slight |
| Independence | 104.9 | 98.5 | 101.7 Agreement) |
| Refinement | 90.0 | 116.5 | 103.5 |
| Physical Health | 115.0 | 92.4 | 103.7 |
| Emotions | 120.0 | 91.0 | 105.5 |
| Energy | 75.0 | 109.0 | 91.0 |
| Courage | 100.0 | 119.5 | 109.8 |
| Unselfishness | 115.0 | 106.0 | 110.5 (Little |
| Integrity | 104.9 | 130.1 | 117.5 Agreement) |
| Coöperativeness | 125.0 | 113.5 | 119.3 |
| Cheerfulness | 130.0 | 112.0 | 121.0 |
| Kindliness | 120.0 | 125.7 | 122.9 |
It is to be noted that certain traits show small divergence in both cases. Thus intellect, quickness, originality and efficiency have low measures of variability, both for the sorority members and for the men of science. The average percentages of these four traits are, in the order named, 93.6, 89.0, 86.2, and 83.7. These, it is to be observed, are the traits which are likely to yield objective products. The more personal, social and moral traits, however, such as coöperativeness, unselfishness, kindliness, cheerfulness, and integrity, show large divergence of the individual judgment with both groups. The average measures of variability for these traits, in the order named, are 119.3, 110.5, 122.9, 121.0, and 117.5. There is another group of traits which, while showing only about average variability with one group, show close agreement in the other: such as will, judgment, perseverance, leadership and breadth. These, it is clear, are more nearly like the objective than they are like the personal traits. Then there are several traits which, while showing only average variability with one group, show large divergences in the other, such as courage and independence. These would seem to be more nearly like the more strictly personal traits.