Skip to content

krippendorff_alpha

krippendorff_alpha

krippendorff_alpha(reliability_data, *, missing=nan) -> float

Krippendorff's alpha for nominal data.

Parameters:

Name Type Description Default
reliability_data 2D array-like

Shape (n_raters, n_units); entry [r, u] is rater r's label for unit u, or missing where that rater did not label that unit.

required
missing value

Sentinel for an absent rating.

``nan``

Returns:

Type Description
float

1 = perfect agreement, 0 = chance, < 0 = systematic disagreement. nan if fewer than one pairable unit.

See Also

cohen_kappa : Two-rater chance-corrected agreement. wilson_interval : A proportion confidence interval.

Examples:

Two raters, three units, identical labels (rows are raters, columns units):

>>> krippendorff_alpha([[1, 2, 1], [1, 2, 1]])
1.0
Source code in src/gbfs_toolkit/analytics/agreement.py
def krippendorff_alpha(reliability_data, *, missing=np.nan) -> float:
    """Krippendorff's alpha for nominal data.

    Parameters
    ----------
    reliability_data : 2D array-like
        Shape ``(n_raters, n_units)``; entry ``[r, u]`` is rater ``r``'s label for
        unit ``u``, or ``missing`` where that rater did not label that unit.
    missing : value, default ``nan``
        Sentinel for an absent rating.

    Returns
    -------
    float
        ``1`` = perfect agreement, ``0`` = chance, ``< 0`` = systematic
        disagreement. ``nan`` if fewer than one pairable unit.

    See Also
    --------
    [`cohen_kappa`][gbfs_toolkit.cohen_kappa] : Two-rater chance-corrected agreement.
    [`wilson_interval`][gbfs_toolkit.wilson_interval] : A proportion confidence interval.

    Examples
    --------
    Two raters, three units, identical labels (rows are raters, columns units):

    >>> krippendorff_alpha([[1, 2, 1], [1, 2, 1]])
    1.0
    """
    data = np.asarray(reliability_data, dtype=object)

    def _is_missing(v) -> bool:
        if v is missing:
            return True
        try:
            return isinstance(v, float) and math.isnan(v)
        except TypeError:
            return False

    # Coincidence matrix over labels that share a unit with >= 2 ratings.
    coincidence: dict[tuple, float] = {}
    n_pairable = 0.0
    for u in range(data.shape[1]):
        vals = [v for v in data[:, u] if not _is_missing(v)]
        m = len(vals)
        if m < 2:
            continue
        n_pairable += m
        w = 1.0 / (m - 1)
        for i in range(m):
            for j in range(m):
                if i == j:
                    continue
                key = (vals[i], vals[j])
                coincidence[key] = coincidence.get(key, 0.0) + w
    if n_pairable < 2:
        return float("nan")

    labels = sorted({c for pair in coincidence for c in pair}, key=repr)
    n_c = dict.fromkeys(labels, 0.0)
    for (c, _k), o in coincidence.items():
        n_c[c] += o
    n = sum(n_c.values())
    diag = sum(coincidence.get((c, c), 0.0) for c in labels)
    sum_nc2 = sum(v * v for v in n_c.values())
    expected = n * n - sum_nc2
    if expected == 0:
        return 1.0
    return 1.0 - (n - 1) * (n - diag) / expected