Validity¶
Do the values respect the business rules?
Current value (data_quality case data): 98.9% - ✓ On target. Open the case.
| Field | Value |
|---|---|
| Definition | Rows passing every validity rule of the dataset divided by rows. |
| Formula | count(rows passing all rules) / count(rows) |
| Unit | % |
| Granularity | Dataset x load; rules defined in kpis/dq_rules.yaml. |
| Direction | Higher is better ↑ |
| Target and thresholds | ≥ 98.0% (amber ≥ 95.0%) |
| Parameters | – |
| Owner | Data Steward |
| Source | All datasets of this repository (rules in kpis/dq_rules.yaml) |
| Input | table any, columns per kpis/dq_rules.yaml |
| Recommended chart | bar - Failures by rule, sorted - the fix list. |
| Related KPIs | Completeness |
How to read it¶
Rules encode business meaning (quantity > 0, resolved after created). Each failing rule is a ticket for the source system owner.
Common misreadings¶
- A row can fail several rules; report rows, not rule failures, or the percentage can go negative.
Worked example (from the test suite)¶
| quantity | price |
|---|---|
| 5 | 2.0 |
| -1 | 2.0 |
| 3 | -1.0 |
| 0 | 1.0 |
Parameters: rules = ['quantity > 0', 'price >= 0']
Expected result: 25.00%. Rules: quantity > 0 and price >= 0. Row 1 passes; row 2 fails quantity; row 3 fails price; row 4 fails quantity. 1 / 4 = 0.25 (rows, not rule failures).
This example is asserted in tests/test_formulas.py against the Python function.
Implementations¶
Python function (the single implementation used by the site):
@kpi("dq_validity")
def dq_validity(df: pd.DataFrame, rules: list[str]) -> float:
return safe_ratio(validity_mask(df, rules).sum(), len(df))