# Privacy lifecycle Managing personal data from collection to deletion: keeping it only as long as it is needed, archiving or deleting it on schedule, and erasing one person's data everywhere when they ask. > [!NOTE] > > **Before reading** > > [PII and data classification]() ยท [Iceberg maintenance]() ## Why it matters Privacy laws require personal data to be kept no longer than necessary and, in many cases, erased on request. Under the GDPR, a request must be answered without undue delay and within one month, extendable by two more in complex cases. A data platform copies data into many layers, tables, snapshots, topics and backups, so deleting it everywhere is a design problem, not a single DELETE statement. ## How it works Retention rules say how long each kind of data is kept and why: order records for the period tax law requires, clickstream at full detail for 13 months, support notes for two years. The owner sets them per dataset and records them with its classification; scheduled jobs delete, or aggregate and then delete, data past its period. Archiving moves data that must be kept but is rarely used to cheaper storage, under the same rules. An erasure request asks to delete everything about one person. Finding everything is the first problem. Classification tags and lineage show which tables hold personal data and which columns identify the person; a register of erasure-relevant tables, kept with the contracts, makes the search repeatable. Deleting everywhere is the second problem, because each place behaves differently: - Lake tables: a DELETE or MERGE writes a new snapshot without the rows, but the old data files still exist and are readable through time travel. The rows are physically gone only after snapshots older than the deletion expire and their files are removed, as described in [Iceberg maintenance](). Merge-on-read deletes are physical only after compaction rewrites the files. - The raw layer is append-only by design, but erasure is the exception: raw rows for the person must be deleted too, or every rebuild brings the data back. - Kafka topics: a compacted topic removes a key's older values after a tombstone, but only when the log cleaner compacts that segment; set max.compaction.lag.ms so this happens within the deadline. Other topics age out, so their retention must be shorter than the deadline. - Derived data: aggregates that no longer identify anyone can stay; tables, extracts, caches and features that do identify the person must be rebuilt or cleaned. Machine learning models trained on the data raise separate questions that the policy must address. - Backups are usually not edited. The common approach is to let them expire within a defined period and to reapply erasures if a backup is ever restored. ``` flowchart TB R[Erasure request
for customer c-91] --> L[Log request,
add to suppression list] L --> F[Find: tagged tables
holding c-91] F --> D1[Delete in raw
and clean tables] F --> D2[Rebuild or clean
derived tables and extracts] D1 --> X[Expire snapshots and
remove files within deadline] D2 --> X X --> P[Record proof:
tables, row counts,
snapshot IDs, dates] ``` Crypto-shredding is an alternative for data that is hard to delete in place. Each person's identifying fields are encrypted with a key specific to that person; erasing the key makes every copy unreadable, including old snapshots and backups. See [Encryption and key management](). A suppression list keeps erased people from coming back. The pipeline checks incoming records against a list of erased IDs, stored as keyed hashes, and drops matches, so a late event or a source that has not yet deleted cannot reintroduce the person. Proof of deletion records the request, the tables searched, the rows deleted, the snapshot expiry and file removal that made the deletion physical, and the dates. Regulators ask for this; the erasure job should write it as it runs. ## Key terms - Retention period How long a kind of data is kept, and why. - Storage limitation The principle of keeping personal data no longer than needed. - Erasure request A request to delete all personal data about one person. - Physical deletion The data no longer exists in any file, as opposed to being hidden from queries. - Crypto-shredding Making data unreadable by destroying its encryption key. - Suppression list IDs of erased people, used to block reintroduction. ## Example Customer c-91 asks for erasure on 2 October. The shop's erasure job: ``` DELETE FROM shop.raw_customers WHERE customer_id = 'c-91'; DELETE FROM shop.clean_customers WHERE customer_id = 'c-91'; -- orders are kept for tax law under the Article 17(3)(b) exemption, -- with fields the law does not require removed, in raw as well as -- clean so a rebuild does not restore them UPDATE shop.raw_orders SET shipping_address = NULL, customer_note = NULL WHERE customer_id = 'c-91'; UPDATE shop.clean_orders SET shipping_address = NULL, customer_note = NULL WHERE customer_id = 'c-91'; -- plus clean_customer_changes, feat_customer_daily and ml_churn_scores; -- the CRM contact is deleted through the reverse ETL tool's delete sync INSERT INTO shop.erasure_log VALUES ('req-2026-1002', current_timestamp, 'deleted: raw_customers, clean_customers, clean_customer_changes, feat_customer_daily, ml_churn_scores, CRM contact; nulled: raw_orders, clean_orders'); ``` dim\_customer and the fact tables are rebuilt by the next dbt run. The engine writes the deletes as position delete files, so the job then compacts the affected partitions; the old files are referenced only by older snapshots, and daily expiry with seven-day retention removes them by 10 October. Noncurrent object versions in the versioned lake bucket last another 30 days, until about 9 November, past the deadline; the log records this residual copy and its expiry date. Kafka topics keep 7 days and backups 30. The log entry, with the expiry dates, is the proof, and c-91 is on the suppression list. ## Pitfalls - Deleting from clean tables only. The next rebuild from raw restores the person. - Forgetting snapshots. Deleted rows stay readable by time travel until snapshots expire and files are removed. - Retention longer than the deadline. Topics and snapshots that outlive one month make erasure impossible to complete on time. - No suppression list. A late event brings the person back. - No record. Without proof, a correct deletion cannot be demonstrated. ## Learn more - [GDPR, Article 5(1)(e) storage limitation, Article 12(3) time limits and Article 17 right to erasure]() - [European Data Protection Board, right to erasure]() ## Related [Residency and regions]() [Kafka and Kafka Connect]() [Disaster recovery]() [The ML lifecycle and MLOps]() [Encryption and key management]() [Audit logging]()