Evaluation & gold standard

Text Classification with Open Source Models

Welcome!

  • Use the arrow keys to navigate
  • Press M to open the slide menu
  • Press ? to see all navigation shortcuts
  • This deck and all other materials are at the course website

Test Time!
Evaluating LLM Codings

Without Validation, We’re Groping in the Dark

“All quantitative models of language are wrong — but some are useful”

(Grimmer & Stewart, 2013, p. 269)


“The use of computers in content analysis invokes a shift in methodological emphasis, from solving the human problem of achieving reliable coding for large volumes of text […] to solving the computational problem of preserving relevant readings of the texts.”

(Krippendorff, 2018, p. 215)

Validity and Reliability

Low reliability
high validity
High reliability
high validity
Low reliability
low validity
High reliability
low validity

In manual content analysis, we care about reliability:

  • Inter-coder reliability: do coders equally apply the coding rules at t1?
  • Intra-coder reliability: does coder A apply the coding rules equally at t1 and t2?
  • Coder–researcher reliability: do coders apply coding rules like the researcher?

In automated content analysis, we care mostly about validity:

  • Semantic validity (model–researcher reliability): does the model apply coding rules like the researcher / preserve meaning? → tested by comparing model outcomes with a gold standard
  • Criterion validity: does the model outcome correlate with external measures? → tested by correlating model outcomes with self-reports or self-assigned labels

Coding a Gold Standard

  • Manually coding a smaller-ish amount of text with high reliability
  • Testing semantic validity by comparing model outcomes against this “ground truth”
  • Assumption: “if trained correctly, humans will make the most correct and valid classifications of texts” (Song et al., 2020, p. 551)
  • Because humans interpret rules differently, establishing a ground truth typically requires:
    • a detailed, crystal-clear codebook (e.g., https://osf.io/2z3dk/)
    • many rounds of coder training and reliability testing
    • multiple independent codings of the same material, with the final code determined by majority vote

Suggestions by Song et al. (2020) for a gold standard

  • Improve inter-coder reliability as much as possible (ideally Krippendorff’s α ≥ .70, or even ≥ .90)
  • Use large validation samples (min. 1,300 cases if inter-coder reliability is α ≥ .70)
  • Proper training is required to avoid coder drift (i.e., low intra-coder reliability)

Want to know more about improving reliability in human codings?

Krippendorff (2018); Neuendorf (2017); Rössler (2017, German)

Typical Key Metrics

  • When comparing categories measured on numeric scales (e.g., a sentiment scale from −3 to +3):
    Pearson or rank correlation
  • When comparing categories measured as dummy variables (e.g., present vs. not present):
    Recall, Precision, F1, and Accuracy (or more advanced reliability coefficients, like Kappa or Krippendorff’s α)



Confusion Matrix:

Model Classification
1 0
Gold-
Standard 		1 True Positive
(TP) 		False Negative
(FN)
0 False Positive
(FP) 		True Negative
(TN)
\[Precision = \dfrac{TP}{TP + FP}\]
\[Recall = \dfrac{TP}{TP + FN}\]
\[Accuracy = \dfrac{TP + TN}{TP + TN + FP + FN}\]
\[F1 = \dfrac{2 \times Precision \times Recall}{Precision + Recall}\]

Typical Key Metrics

Comparing the fear coding of two models against a gold standard, for four example statements:

Gold
Standard 		Model A Model B
Statements Fear Fear Fear
“The current heat wave really scares me.” 1 1 1
“It’s getting worse every year. COME ON MERZ, DO SOMETHING!!!” 0 1 0
“It drives me nuts that we’re still running coal-fired power plants.” 0 1 0
“I often lose sleep thinking about global warming.” 1 1 0
TP 2 1
TN 0 2
FP 2 0
FN 0 1
Accuracy 50% 75%
Recall 100% 50%
Precision 50% 100%

\[Precision = \dfrac{TP}{TP + FP}\] \[Recall = \dfrac{TP}{TP + FN}\] \[Accuracy = \dfrac{TP + TN}{TP + TN + FP + FN}\]

References

Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267–297. https://doi.org/10.1093/pan/mps028
Krippendorff, K. (2018). Content analysis: An introduction to its methodology (Fourth Edition). SAGE.
Neuendorf, K. A. (2017). The Content Analysis Guidebook. SAGE Publications, Inc. https://doi.org/10.4135/9781071802878
Rössler, P. (2017). Inhaltsanalyse (3. überarbeitete Auflage, Online-Ausgabe). UVK Verlagsgesellschaft mbH. https://doi.org/10.36198/9783838547060
Song, H., Tolochko, P., Eberl, J.-M., Eisele, O., Greussing, E., Heidenreich, T., Lind, F., Galyga, S., & Boomgaarden, H. G. (2020). In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis. Political Communication, 37(4), 550–572. https://doi.org/10/ggnh69