Completeness¶
Are the fields we need actually filled in?
Current value (data_quality case data): 96.3% - ! Watch. Open the case.
| Field | Value |
|---|---|
| Definition | Non-null cells in the required columns divided by all required cells. |
| Formula | count(non-null required cells) / (rows * required columns) |
| Unit | % |
| Granularity | Dataset x load; required columns defined in kpis/dq_rules.yaml. |
| Direction | Higher is better ↑ |
| Target and thresholds | ≥ 99.0% (amber ≥ 95.0%) |
| Parameters | – |
| Owner | Data Steward |
| Source | All datasets of this repository (rules in kpis/dq_rules.yaml) |
| Input | table any, columns per kpis/dq_rules.yaml |
| Recommended chart | heatmap - Dataset x dimension grid with thresholds - one glance for the data owner. |
| Related KPIs | Validity, Uniqueness |
How to read it¶
Measured only on the columns a KPI depends on. A missing customer id is a business problem the moment retention is computed without it.
Common misreadings¶
- Measuring completeness over every column dilutes the critical ones; scope it to required fields.
- Placeholder values ("N/A", 0, "unknown") count as filled; pair completeness with validity.
Worked example (from the test suite)¶
| id | customer | amount |
|---|---|---|
| 1 | x | 1.0 |
| 2 | – | 2.0 |
| 3 | – | |
| 4 | y | 4.0 |
Parameters: columns = ['customer', 'amount']
Expected result: 62.50%. Required columns customer, amount: 4 rows x 2 = 8 cells. customer is missing twice (None and a blank string), amount once. 5 filled of 8 = 0.625.
This example is asserted in tests/test_formulas.py against the Python function.
Implementations¶
Python function (the single implementation used by the site):
@kpi("dq_completeness")
def dq_completeness(df: pd.DataFrame, columns: list[str]) -> float:
return safe_ratio(*completeness_counts(df, columns))