Timeliness¶
Is the data available soon enough after the event to act on it?
Current value (data_quality case data): 97.4% - ✓ On target. Open the case.
| Field | Value |
|---|---|
| Definition | Records loaded within the allowed delay after the business event divided by records. |
| Formula | count(loaded_at - event_at <= max_delay_hours) / count(rows) |
| Unit | % |
| Granularity | Dataset x load; event and load columns defined in kpis/dq_rules.yaml. |
| Direction | Higher is better ↑ |
| Target and thresholds | ≥ 95.0% (amber ≥ 90.0%) |
| Parameters | – |
| Owner | Data Engineer |
| Source | Datasets that carry a load timestamp (rules in kpis/dq_rules.yaml) |
| Input | table any, columns per kpis/dq_rules.yaml |
| Recommended chart | heatmap - Same grid as the other dimensions. |
| Related KPIs | Completeness |
How to read it¶
Late data makes leading indicators lag. Datasets without a load timestamp cannot be measured - that absence is itself a finding.
Common misreadings¶
- Freshness of the latest record is not timeliness of all records; late back-dated entries hide behind a fresh max date.
Worked example (from the test suite)¶
| event_at | loaded_at |
|---|---|
| 2024-01-01 08:00 | 2024-01-01 09:00 |
| 2024-01-01 08:00 | 2024-01-02 08:00 |
| 2024-01-01 08:00 | 2024-01-02 08:01 |
| 2024-01-01 08:00 | 2024-01-05 |
Parameters: event_col = event_at, loaded_col = loaded_at, max_delay_hours = 24
Expected result: 50.00%. Max delay 24 h. Delays 1 h, 24 h (inclusive, on time), 24 h 1 min, 88 h. 2 / 4 = 0.5.
This example is asserted in tests/test_formulas.py against the Python function.
Implementations¶
Python function (the single implementation used by the site):
@kpi("dq_timeliness")
def dq_timeliness(
df: pd.DataFrame, event_col: str, loaded_col: str, max_delay_hours: float
) -> float:
return safe_ratio(*timeliness_counts(df, event_col, loaded_col, max_delay_hours))