Skip to content

insert_explicit_gaps

insert_explicit_gaps

insert_explicit_gaps(panel: DataFrame, *, expected_freq: str = '5min', tolerance: str = '15min', time_col: str = 'fetched_at', by: tuple[str, ...] = ('system_id', 'station_id')) -> DataFrame

Insert NaN rows where collection stalled, so plots show the outage honestly.

When a scraper dies on Friday and resumes Monday, a line plot draws a misleading straight segment across the gap. Inserting an explicit NaN row in each gap longer than tolerance forces a visual break (and stops rolling statistics bridging the hole). Purely descriptive: it makes missingness explicit, it does not fill it.

Parameters:

Name Type Description Default
panel DataFrame

Panel/frame with the grouping columns in by and a time_col.

required
expected_freq str

The intended cadence (documented for intent; the test is against tolerance).

"5min"
tolerance str

Gaps strictly larger than this get a NaN marker row at their midpoint.

"15min"
time_col see [`resample_panel`][gbfs_toolkit.resample_panel].
'fetched_at'
by see [`resample_panel`][gbfs_toolkit.resample_panel].
'fetched_at'

Returns:

Type Description
DataFrame

A flat copy of panel with one NaN-valued row inserted per detected gap.

See Also

coverage_report : Quantify the same gaps numerically. resample_panel : Carry state across gaps instead of marking them.

Examples:

>>> import pandas as pd
>>> panel = pd.DataFrame({
...     "station_id": "a",
...     "fetched_at": pd.to_datetime(
...         ["2026-01-01T00:00Z", "2026-01-01T00:05Z", "2026-01-01T02:00Z"]),
...     "num_bikes_available": [5, 4, 6],
... })
>>> out = insert_explicit_gaps(panel)
>>> len(out), int(out["num_bikes_available"].isna().sum())
(4, 1)
Source code in src/gbfs_toolkit/io/timeseries.py
def insert_explicit_gaps(
    panel: pd.DataFrame,
    *,
    expected_freq: str = "5min",
    tolerance: str = "15min",
    time_col: str = "fetched_at",
    by: tuple[str, ...] = ("system_id", "station_id"),
) -> pd.DataFrame:
    """Insert ``NaN`` rows where collection stalled, so plots show the outage honestly.

    When a scraper dies on Friday and resumes Monday, a line plot draws a misleading
    straight segment across the gap. Inserting an explicit ``NaN`` row in each gap longer
    than ``tolerance`` forces a visual break (and stops rolling statistics bridging the
    hole). Purely descriptive: it makes missingness explicit, it does not fill it.

    Parameters
    ----------
    panel : pandas.DataFrame
        Panel/frame with the grouping columns in ``by`` and a ``time_col``.
    expected_freq : str, default "5min"
        The intended cadence (documented for intent; the test is against ``tolerance``).
    tolerance : str, default "15min"
        Gaps strictly larger than this get a ``NaN`` marker row at their midpoint.
    time_col, by : see [`resample_panel`][gbfs_toolkit.resample_panel].

    Returns
    -------
    pandas.DataFrame
        A flat copy of ``panel`` with one ``NaN``-valued row inserted per detected gap.

    See Also
    --------
    [`coverage_report`][gbfs_toolkit.coverage_report] : Quantify the same gaps numerically.
    [`resample_panel`][gbfs_toolkit.resample_panel] : Carry state across gaps instead of marking them.

    Examples
    --------
    >>> import pandas as pd
    >>> panel = pd.DataFrame({
    ...     "station_id": "a",
    ...     "fetched_at": pd.to_datetime(
    ...         ["2026-01-01T00:00Z", "2026-01-01T00:05Z", "2026-01-01T02:00Z"]),
    ...     "num_bikes_available": [5, 4, 6],
    ... })
    >>> out = insert_explicit_gaps(panel)
    >>> len(out), int(out["num_bikes_available"].isna().sum())
    (4, 1)
    """
    df = panel_frame(panel).copy()
    keys = [k for k in by if k in df.columns]
    require_columns(df, [*keys, time_col], what="insert_explicit_gaps")
    df[time_col] = pd.to_datetime(df[time_col], utc=True)
    df = df.sort_values([*keys, time_col]) if keys else df.sort_values(time_col)
    tol = pd.Timedelta(tolerance)
    value_cols = [c for c in df.columns if c not in keys and c != time_col]

    gap_rows = []
    grouped = df.groupby(keys, sort=False) if keys else [((), df)]
    for _, g in grouped:
        g = g.reset_index(drop=True)
        dt = g[time_col].diff()
        for i in np.where((dt > tol).to_numpy())[0]:
            prev, nxt = g.iloc[i - 1], g.iloc[i]
            row = {k: prev[k] for k in keys}
            row[time_col] = prev[time_col] + (nxt[time_col] - prev[time_col]) / 2
            for c in value_cols:
                row[c] = np.nan
            gap_rows.append(row)
    if gap_rows:
        df = pd.concat([df, pd.DataFrame(gap_rows)], ignore_index=True)
        df = df.sort_values([*keys, time_col]) if keys else df.sort_values(time_col)
    return df.reset_index(drop=True)