Separating Decision Logic

The Problem

When data retention rules are hardcoded into pipelines, regulatory changes require redeploying code. A pipeline filtering user data might enforce retention like this:

CREATE VIEW retained_user_events AS
SELECT * FROM user_events
WHERE created_at > CURRENT_DATE - 365

GDPR requires 90-day retention for EU users. CCPA allows 2 years for California users. The hardcoded 365-day window violates GDPR. Fixing it requires changing SQL and redeploying. This is the core problem with business logic in SQL.

Separated Approach

Instead of embedding retention windows in SQL, store them as queryable metadata organized by data type and jurisdiction. Pipelines retrieve and apply current rules at runtime:

rule: data_retention
user_events:
  EU:
    retention_days: 90
    regulation: "GDPR"
  US_CA:
    retention_days: 730
    regulation: "CCPA"
  US_OTHER:
    retention_days: 1095
    regulation: "internal_policy"

profile_data:
  EU:
    retention_days: 180
    regulation: "GDPR"
  US_CA:
    retention_days: 1095
    regulation: "CCPA"

The pipeline code queries retention rules based on data type and user jurisdiction instead of hardcoding windows:

def get_retained_user_events():
    rules = context.get_rule("data_retention")

    # Build jurisdiction-specific retention conditions
    conditions = []
    for jurisdiction, policy in rules.user_events.items():
        days = policy['retention_days']
        conditions.append(f"""
            (user_jurisdiction = '{jurisdiction}' AND
             created_at > CURRENT_DATE - {days})
        """)

    return db.query(f"""
        SELECT * FROM user_events
        WHERE {' OR '.join(conditions)}
    """)

When GDPR updates retention requirements from 90 to 60 days, update the rule metadata. The pipeline adapts without code changes. This transforms static workflows into dynamic ones.

What This Enables