# simply run this cell to install the packages
install.packages(c("tidyverse", "yardstick"))This notebook belongs to the DGPs 2026 workshop Text Classification with Open Source Models.
Ideally, you’ve now got predictions from three different classifiers, for three emotions each (anger, fear, sadness).
In this notebook, we’ll evaluate those predictions two ways: first against RW3D’s self-reported emotion intensity ratings, second against a manual gold standard.
We will run this exercise entirely in R (but you could use Python or another language if you prefer), using the tidyverse and yardstick packages. If you haven’t already, please install them.
How to use this notebook: run each cell from top to bottom with Shift+Enter or by clicking the run button. Most cells are given and fully explained; a few are marked Your turn — those are where you edit or write something yourself. Every exercise cell has a safe default, so nothing breaks if you leave it as-is, but you’ll get more out of the session if you actually change it.
Important note for Colab users: If you are running this notebook in Google Colab, please follow these steps: 1. Make sure you have a Google account and are logged in. 2. To create a copy of this notebook in your own Google Drive (advised), select “Copy to Drive” at the top of the Colab page. 3. In this notebook, you do not need to change the runtime to GPU because we are not using any GPU-intensive operations. 4. You can run the notebook cells as instructed.
Setup
In [3]:
In [1]:
library(tidyverse)
library(yardstick)── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ── ✔ dplyr 1.2.1 ✔ readr 2.1.5 ✔ forcats 1.0.0 ✔ stringr 1.6.0 ✔ ggplot2 4.0.1 ✔ tibble 3.3.1 ✔ lubridate 1.9.4 ✔ tidyr 1.3.1 ✔ purrr 1.0.4 ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── ✖ dplyr::filter() masks stats::filter() ✖ dplyr::lag() masks stats::lag() ℹ Use the conflicted package to force all conflicts to become errors Attaching package: ‘yardstick’ The following object is masked from ‘package:readr’: spec
Step 1: Get your results
Load a reference results file that we have created (using the same methods that you have used).
(If you are a pro 🧐 and bored: You can save your own results from the previous notebooks to CSV files, read in those files here and combine them into one data frame. This way, you get to see how your choices (e.g., your prompts) actually scored. Make sure to combine the outputs into a data frame with the same structure as the reference results file so that the rest of the notebook works without changes.)
In [2]:
RESULTS_URL <- "https://raw.githubusercontent.com/felixdidi/llm-content-analysis/main/sections/workshop/data/classification_results_reference.csv"
GOLD_URL <- "https://raw.githubusercontent.com/felixdidi/llm-content-analysis/main/sections/workshop/data/gold_standard.csv"
# read in the reference results
results <- read_csv(RESULTS_URL, show_col_types = FALSE)
# read in the gold standard and rename the columns to avoid name clashes
gold <- read_csv(GOLD_URL, show_col_types = FALSE) |>
select(id, gold_anger = anger, gold_fear = fear, gold_sadness = sadness)
# join the two data frames on the "id" column
results <- results |> left_join(gold, by = "id")
# look at the first few rows of the results data frame
head(results)# A tibble: 6 × 20 id text main_emotion self_report_anger self_report_fear <dbl> <chr> <chr> <dbl> <dbl> 1 1 Stressed and uninformed… anger 9 1 2 2 I feel worried about th… fear 8 9 3 3 Disgusted how unprotect… sadness 2 6 4 4 I am not too worried ab… sadness 1 3 5 5 I'm finding the Corona … sadness 5 2 6 6 At the moment I feel li… sadness 1 2 # ℹ 15 more variables: self_report_sadness <dbl>, encoder_anger <dbl>, # encoder_fear <dbl>, encoder_sadness <dbl>, nli_anger <dbl>, nli_fear <dbl>, # nli_sadness <dbl>, decoder_anger <lgl>, decoder_fear <lgl>, # decoder_sadness <lgl>, decoder_reasoning <chr>, decoder_raw <chr>, # gold_anger <lgl>, gold_fear <lgl>, gold_sadness <lgl>
Step 2: Do the probability scores track intensity?
The category-specific encoder and the NLI encoder each returned a score between 0 and 1 per emotion.
A natural first instinct for interpreting these scores is to read a higher score as “the model thinks this text expresses more of that emotion,” the same way a higher self-report rating means more intensity. But all a classifier’s score technically tells you is how confident the model is in its yes/no call; it seems likely that confidence scales with the underlying emotion’s intensity, but we do not know that for sure.
Let’s check this: how closely do encoder_anger/encoder_fear/encoder_sadness and nli_anger/nli_fear/nli_sadness correlate with the matching self_report_* rating?
If you know how to do this in R, you can write your own code.
In [3]:
# write your own code, e.g., for anger
results_corr_anger <- results |>
select(self_report_anger, encoder_anger, nli_anger)
cor(results_corr_anger) self_report_anger encoder_anger nli_anger
self_report_anger 1.0000000 0.3031520 0.4396954
encoder_anger 0.3031520 1.0000000 0.4968341
nli_anger 0.4396954 0.4968341 1.0000000If your brain is alredy fried from the previous exercises, you can simply run the following convenience code to calculate the correlations:
In [4]:
LABELS <- c("anger", "fear", "sadness")
# A helper function to compute correlations between self-report, encoder, and NLI scores for a given emotion label
# you do not need to edit this function, just call it with the label you want to analyze
relevant_correlations <- function(label) {
cols <- c(
self_report = paste0("self_report_", label),
encoder = paste0("encoder_", label),
nli = paste0("nli_", label)
)
combn(names(cols), 2, simplify = FALSE) |>
map_dfr(function(pair) {
test <- cor.test(results[[cols[[pair[1]]]]], results[[cols[[pair[2]]]]])
tibble(
pair = paste(pair[1], "vs", pair[2]),
correlation = round(unname(test$estimate), 2),
p_value = signif(test$p.value, 3)
)
})
}
# Compute correlations for each emotion label and store them in a named list
correlations <- set_names(map(LABELS, relevant_correlations), LABELS)
correlations$anger# A tibble: 3 × 3 pair correlation p_value <chr> <dbl> <dbl> 1 self_report vs encoder 0.3 0.000163 2 self_report vs nli 0.44 0.0000000181 3 encoder vs nli 0.5 0.0000000001
In [5]:
correlations$fear# A tibble: 3 × 3 pair correlation p_value <chr> <dbl> <dbl> 1 self_report vs encoder 0.49 2.38e-10 2 self_report vs nli 0.38 1.3 e- 6 3 encoder vs nli 0.42 1.18e- 7
In [6]:
correlations$sadness# A tibble: 3 × 3 pair correlation p_value <chr> <dbl> <dbl> 1 self_report vs encoder 0.18 2.75e- 2 2 self_report vs nli 0.04 5.91e- 1 3 encoder vs nli 0.6 2.43e-16
Your turn
Would you call these correlations strong enough to treat the probability scores as a helpful indicator for self-reported intensity, e.g., to rank texts by “how much anger” they express?
What do you think: Are self-reported emotions a good benchmark for model evaluation? Why or why not?
Step 3: Comparison with a manual gold standard
Although the self-report ratings are interesting to compare to, they are not a perfect match for what the classifiers are actually predicting. Whereas they reflect the participant’s judgment of how much of that emotion they felt in their current situation, the classifiers are making a yes/no call about whether each emotion is expressed in the text at all (sometimes returned as a probability score, sometimes as a boolean):
- the category-specific encoder and the NLI encoder each returned a score between 0 and 1 per emotion (how confident the model is);
- the decoder returned a boolean per emotion directly (its JSON answer already was
true/false).
We should therefore also compare the classifiers’ predictions to a manual gold standard (i.e., manual annotations of whether each emotion is expressed in the text).
(please consider this gold standard as an example for how to evaluate your own models, not as a definitive truth about the RW3D dataset)
To demonstrate, we compare the gold standard to the decoder’s predictions, which are already in a yes/no format just like the gold standard. The other two classifiers’ scores can be converted to yes/no calls by thresholding (e.g., score > 0.5; if you know how to do data transformations in R and have some extra time, you can do this as a bonus exercise. Please note: Defining thresholds is a mess and we could do a workshop only on this question).
Take a look at a few rows before diving into metrics:
In [7]:
results |>
select(text, gold_anger, decoder_anger) |>
head(3)# A tibble: 3 × 3 text gold_anger decoder_anger <chr> <lgl> <lgl> 1 Stressed and uninformed. Don't feel enough is being … TRUE FALSE 2 I feel worried about the virus, burnout about me get… FALSE FALSE 3 Disgusted how unprotected NHS staff are and how late… TRUE TRUE
Step 3a: Accuracy
Accuracy is simply the share of texts the decoder got right, for one emotion at a time. It’s the easiest metric to compute, but it hides a lot: the gold standard isn’t evenly split between TRUE and FALSE for every emotion (see the baseline accuracy printed below, i.e., the share you’d get by always guessing whichever class is more common). A model can reach a high accuracy just by leaning toward the majority class, without actually discriminating anything.
You can do this with some very simple R code:
In [8]:
# Transform the values to factors
results$gold_anger <- as.factor(results$gold_anger)
results$decoder_anger <- as.factor(results$decoder_anger)
# Accuracy function is as simple as that
accuracy(results, gold_anger, decoder_anger)# A tibble: 1 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 accuracy binary 0.787
In [9]:
# Let's take a look at baseline
prop.table(
table(results$gold_anger)
)
FALSE TRUE
0.4933333 0.5066667 You can also use our convenience function below:
In [10]:
# you can simply run this cell to compute accuracy for each emotion label, or edit it to compute accuracy for your own predictions
for (label in LABELS) {
cat(sprintf("=== %s ===\n", label))
gold_col <- results[[paste0("gold_", label)]]
pred <- results[[paste0("decoder_", label)]]
acc <- mean(pred == gold_col, na.rm = TRUE)
cat(sprintf(" decoder : %.1f%% accuracy\n", acc * 100))
baseline <- max(prop.table(table(gold_col)))
cat(sprintf(" baseline: %.1f%% accuracy (always predict the majority class)\n\n", baseline * 100))
}=== anger ===
decoder : 78.7% accuracy
baseline: 50.7% accuracy (always predict the majority class)
=== fear ===
decoder : 83.3% accuracy
baseline: 77.3% accuracy (always predict the majority class)
=== sadness ===
decoder : 76.0% accuracy
baseline: 52.7% accuracy (always predict the majority class)
Step 3b: Precision, recall, and per-class performance
To get a more complete picture of how well the decoder is doing, we can compute three additional metrics:
- Precision: of the texts the decoder called
TRUEfor an emotion, how many actually were? - Recall: of the texts that actually expressed that emotion, how many did the decoder find?
- F1-score: the harmonic mean of precision and recall.
In R, you can use the yardstick package to calculate these metrics (similar output can be obtained in other programming languages). To help you extract all relevant metrics in one call, we provide a helper function below. A few things about its output that aren’t obvious on first read:
- It reports a row for each class the target can take (here
FALSEandTRUE) with its own precision/recall/F1. TheTRUErow is usually the one you care about (did the model correctly catch the emotion?), but theFALSErow matters too: a model with greatTRUE-recall but terribleFALSE-recall is just saying “yes” to almost everything. supportis the number of texts that belong to that row’s class (the count of goldTRUEs or goldFALSEs), not the number of predictions. It’s what lets you judge whether a score is based on 5 cases or 100.macro avgtakes the unweighted mean of theFALSEandTRUErows, so a class with only 30 members counts exactly as much as one with 120.weighted avginstead weights each class by itssupport. Comparing the two is a quick imbalance check: if they’re far apart, the model is doing much better on one class than the other.
You can also calculate the metrics manually with some very simple R code:
In [11]:
# FALSE
precision(results, gold_anger, decoder_anger)
recall(results, gold_anger, decoder_anger)
f_meas(results, gold_anger, decoder_anger)# A tibble: 1 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 f_meas binary 0.822
In [12]:
# TRUE
precision(results, gold_anger, decoder_anger, event_level = "second")
recall(results, gold_anger, decoder_anger, event_level = "second")
f_meas(results, gold_anger, decoder_anger, event_level = "second")# A tibble: 1 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 f_meas binary 0.733
A convenience function to get everything at once:
In [13]:
# this is the helper function, you do not need to edit it
classification_report <- function(data, truth_col, pred_col) {
d <- data |>
transmute(
truth = factor(.data[[truth_col]], levels = c(FALSE, TRUE)),
estimate = factor(.data[[pred_col]], levels = c(FALSE, TRUE))
) |>
drop_na()
cm <- conf_mat(d, truth, estimate)
per_class <- bind_rows(
summary(cm, event_level = "first") |> mutate(class = "FALSE"),
summary(cm, event_level = "second") |> mutate(class = "TRUE")
) |>
filter(.metric %in% c("precision", "recall", "f_meas")) |>
select(class, .metric, .estimate) |>
pivot_wider(names_from = .metric, values_from = .estimate) |>
left_join(
count(d, truth, name = "n") |> mutate(class = as.character(truth)) |> select(class, n),
by = "class"
)
macro <- per_class |>
summarise(class = "macro avg", precision = mean(precision), recall = mean(recall),
f_meas = mean(f_meas), n = sum(n))
weighted <- per_class |>
summarise(class = "weighted avg",
precision = weighted.mean(precision, n), recall = weighted.mean(recall, n),
f_meas = weighted.mean(f_meas, n), n = sum(n))
bind_rows(per_class, macro, weighted) |>
rename(f1 = f_meas, support = n) |>
mutate(across(c(precision, recall, f1), \(x) round(x, 2)))
}
# here, we compute the classification report for each emotion label and print it
for (label in LABELS) {
cat(sprintf("=== %s ===\n", label))
print(classification_report(results, paste0("gold_", label), paste0("decoder_", label)))
}=== anger === # A tibble: 4 × 5 class precision recall f1 support <chr> <dbl> <dbl> <dbl> <int> 1 FALSE 0.7 1 0.82 74 2 TRUE 1 0.58 0.73 76 3 macro avg 0.85 0.79 0.78 150 4 weighted avg 0.85 0.79 0.78 150 === fear === # A tibble: 4 × 5 class precision recall f1 support <chr> <dbl> <dbl> <dbl> <int> 1 FALSE 0.68 0.5 0.58 34 2 TRUE 0.86 0.93 0.9 116 3 macro avg 0.77 0.72 0.74 150 4 weighted avg 0.82 0.83 0.82 150 === sadness === # A tibble: 4 × 5 class precision recall f1 support <chr> <dbl> <dbl> <dbl> <int> 1 FALSE 0.81 0.65 0.72 71 2 TRUE 0.73 0.86 0.79 79 3 macro avg 0.77 0.75 0.75 150 4 weighted avg 0.77 0.76 0.76 150
Your turn
Where does the decoder actually get it wrong?
A confusion matrix (as a simple table) shows how the decoder’s TRUE/FALSE calls line up against the gold standard. Rows are the true (gold) label, columns are the decoder’s prediction.
For which of the three emotions does the decoder have the most false positives (calling an emotion TRUE that the gold standard says is FALSE)? Set chosen_label below to check yourself.
In [14]:
chosen_label <- "fear" # TODO: try "anger", "fear", "sadness"
table(
gold = results[[paste0("gold_", chosen_label)]],
predicted = results[[paste0("decoder_", chosen_label)]]
) predicted
gold FALSE TRUE
FALSE 17 17
TRUE 8 108Your turn
Aggregate metrics only get you so far – sometimes you want to read the actual texts behind a specific kind of mistake. Say you want to see texts where the decoder called fear TRUE, but the gold standard says it wasn’t: set chosen_label <- "fear" and direction <- "false_positive" below.
You can change chosen_label and direction to look at other combinations, e.g. sadness false negatives (texts the decoder called FALSE on, even though the gold standard says sadness was present).
In [15]:
chosen_label <- "fear" # TODO: "anger", "fear", "sadness"
direction <- "false_positive" # TODO: "false_positive" (decoder said TRUE, gold says FALSE)
# or "false_negative" (decoder said FALSE, gold says TRUE)
# you do not need to edit the code below, just change the two variables above
pred <- results[[paste0("decoder_", chosen_label)]]
gold_col <- results[[paste0("gold_", chosen_label)]]
mask <- if (direction == "false_positive") {
pred & !gold_col
} else if (direction == "false_negative") {
!pred & gold_col
} else {
stop('direction must be "false_positive" or "false_negative"')
}
results |>
filter(mask) |>
select(text, gold_anger, gold_fear, gold_sadness) |>
head(3)# A tibble: 3 × 4 text gold_anger gold_fear gold_sadness <chr> <fct> <lgl> <lgl> 1 "I feel trapped in my own life. I am stuck … FALSE FALSE TRUE 2 "Fed up of the situation, which could have … TRUE FALSE TRUE 3 "I'm sad for everything I see on the tv - a… FALSE FALSE TRUE