Krippendorff's alpha for nominal data.
Parameters:
| Name |
Type |
Description |
Default |
reliability_data
|
2D array-like
|
Shape (n_raters, n_units); entry [r, u] is rater r's label for
unit u, or missing where that rater did not label that unit.
|
required
|
missing
|
value
|
Sentinel for an absent rating.
|
``nan``
|
Returns:
| Type |
Description |
float
|
1 = perfect agreement, 0 = chance, < 0 = systematic
disagreement. nan if fewer than one pairable unit.
|
See Also
cohen_kappa : Two-rater chance-corrected agreement.
wilson_interval : A proportion confidence interval.
Examples:
Two raters, three units, identical labels (rows are raters, columns units):
>>> krippendorff_alpha([[1, 2, 1], [1, 2, 1]])
1.0
Source code in src/gbfs_toolkit/analytics/agreement.py
| def krippendorff_alpha(reliability_data, *, missing=np.nan) -> float:
"""Krippendorff's alpha for nominal data.
Parameters
----------
reliability_data : 2D array-like
Shape ``(n_raters, n_units)``; entry ``[r, u]`` is rater ``r``'s label for
unit ``u``, or ``missing`` where that rater did not label that unit.
missing : value, default ``nan``
Sentinel for an absent rating.
Returns
-------
float
``1`` = perfect agreement, ``0`` = chance, ``< 0`` = systematic
disagreement. ``nan`` if fewer than one pairable unit.
See Also
--------
[`cohen_kappa`][gbfs_toolkit.cohen_kappa] : Two-rater chance-corrected agreement.
[`wilson_interval`][gbfs_toolkit.wilson_interval] : A proportion confidence interval.
Examples
--------
Two raters, three units, identical labels (rows are raters, columns units):
>>> krippendorff_alpha([[1, 2, 1], [1, 2, 1]])
1.0
"""
data = np.asarray(reliability_data, dtype=object)
def _is_missing(v) -> bool:
if v is missing:
return True
try:
return isinstance(v, float) and math.isnan(v)
except TypeError:
return False
# Coincidence matrix over labels that share a unit with >= 2 ratings.
coincidence: dict[tuple, float] = {}
n_pairable = 0.0
for u in range(data.shape[1]):
vals = [v for v in data[:, u] if not _is_missing(v)]
m = len(vals)
if m < 2:
continue
n_pairable += m
w = 1.0 / (m - 1)
for i in range(m):
for j in range(m):
if i == j:
continue
key = (vals[i], vals[j])
coincidence[key] = coincidence.get(key, 0.0) + w
if n_pairable < 2:
return float("nan")
labels = sorted({c for pair in coincidence for c in pair}, key=repr)
n_c = dict.fromkeys(labels, 0.0)
for (c, _k), o in coincidence.items():
n_c[c] += o
n = sum(n_c.values())
diag = sum(coincidence.get((c, c), 0.0) for c in labels)
sum_nc2 = sum(v * v for v in n_c.values())
expected = n * n - sum_nc2
if expected == 0:
return 1.0
return 1.0 - (n - 1) * (n - diag) / expected
|