Typorithm learning path
ETL and Data Engineering: reliable pipelines
Build reliable data pipelines with Python, SQLite and real Parquet files. Preserve raw lineage, parse CSV, normalize schema versions, quarantine invalid rows, round decimal amounts, batch and deduplicate. Persist versioned UPSERTs, atomic checkpoints, composite watermarks, bounded late-arrival replay, CDC tombstones and SCD Type 2 history. Add quality gates, staged partition publication, dependency states and bounded retries. Finish with a disk-backed recovery lab checked through abrupt process exit and safe replay. 18 topics with six sections, complete examples, measured prepared results, explained questions and independent practice with two hints. Download the whole course explicitly for offline app reading; computer lab execution is separate.
Study with a practical destination
Use the linked lessons in order, complete the lab and explain the decisions without the reference. Select the path and depth that match your role.
Your learning path
0 of 18 lessons completed
Saved on this device
Module 1 · 1 lessons
Immutable raw input and lineage
Keep original source bytes, derive a content identity and separate replayable raw data from transformations.
Module 2 · 1 lessons
CSV records, quoting and structural errors
Parse logical records with the CSV library instead of splitting physical lines or commas.
Module 3 · 1 lessons
Schema versions and quarantine
Normalize known schema versions and retain precise rejection reasons for unknown or invalid records.
Module 4 · 1 lessons
Money, rounding and UTC normalization
Use decimal source values and an explicit rounding policy; reject timestamps without a timezone.
Module 5 · 1 lessons
Bounded batches and the final partial chunk
Consume an iterator in bounded groups without dropping its last records.
Module 6 · 1 lessons
Deterministic deduplication and conflicting versions
Choose the highest source version per identity and reject ambiguous equal-version payloads.
Module 7 · 1 lessons
Parquet schema, nulls and predicate boundaries
Write and read an actual columnar file with DuckDB, checking schema as well as returned rows.
Module 8 · 1 lessons
Versioned SQLite UPSERT and idempotent replay
Persist source ordering across batches so replay cannot duplicate or roll back current state.
Module 9 · 1 lessons
Atomic facts and source checkpoints
Commit data and progress together, then reopen the database to verify rollback after failure.
Module 10 · 1 lessons
Composite cursors and tied source timestamps
Page a fixed ordered source with a strict tuple cursor so equal timestamps are not skipped.
Module 11 · 1 lessons
Bounded lookback and late-arriving updates
Freeze the extraction upper bound, replay a limited overlap and state which late records remain outside it.
Module 12 · 1 lessons
CDC tombstones and stale resurrection
Retain delete versions so an older replay cannot bring a deleted entity back to life.
Module 13 · 1 lessons
SCD Type 2 and half-open validity intervals
Retain dimension history with unambiguous as-of boundaries and an explicit ordered-change assumption.
Module 14 · 1 lessons
Quality gates and quarantine budgets
Combine independent publication requirements, reconcile rejected rows and define an empty-dataset policy.
Module 15 · 1 lessons
Partition generations and atomic manifest publication
Keep partial writes invisible by publishing a complete immutable generation through one pointer.
Module 16 · 1 lessons
Dependency states, failure propagation and cycles
Use a real topological dependency graph while distinguishing terminal processing from successful execution.
Module 17 · 1 lessons
Bounded retries and stable logical batch identity
Retry transient failures within a finite budget and keep logical batch identity independent of attempts.
Module 18 · 1 lessons
Recovery lab: facts, CDC, checkpoint and batch audit
Build a real disk-backed pipeline that survives process interruption and safe replay inside one SQLite boundary.