Dependency states, failure propagation and cycles

A pipeline graph describes which tasks depend on which other tasks. A valid dependency order does not imply every predecessor succeeded. The fixture uses TopologicalSorter for dependency readiness and a separate state map to decide whether a task runs, fails or becomes blocked by an unsuccessful dependency.

The example is an in-process state machine over owned task names. It does not start Airflow, schedule remote workers or provide distributed retries. That limited scope makes the core rule testable: a failed branch blocks its descendants while an independent branch can still finish, and a cycle is rejected before execution begins.

The graph maps each node to its predecessors. prepare checks for cycles, and each get_ready group is processed in sorted order for deterministic results. A node with any nonsuccessful dependency is blocked; the designated fault node fails; otherwise the node succeeds. Every terminally processed node is then marked done.

Calling done means the sorter can release structural successors; it does not certify a successful task. That is why the explicit state check remains necessary after readiness. The returned report sorts node states, letting the acceptance cases inspect a failed extract branch, its blocked load and an unrelated successful branch independently.

The mutation checks for successful predecessors when deciding to block. It reverses the semantic condition, allowing the wrong downstream behavior after failure and preventing valid continuation elsewhere. The graph library still supplies a valid topological order, which demonstrates that order alone cannot enforce business success dependencies.

Ignoring cycles can leave a run waiting forever for an impossible prerequisite. The program converts CycleError into a deliberate ValueError before it starts assigning task states. In a larger scheduler, distinguish failed, skipped, blocked, cancelled and retried according to documented trigger rules rather than overloading one done marker with all meanings.

Add successful, a sorted list of task names whose final state equals success. Preserve the full state report, cycle rejection and independent-branch behavior. The added list should be derived from measured task states, not from the original list of nodes or a guess about which tasks usually run.

Build the list after the topological processing loop. Use the failure graph, all-success graph, empty graph and cyclic graph. The independent cases ensure the list excludes both the designated failing task and its blocked descendants while retaining unrelated successes. Do not change done calls or dependency conditions to make the report look complete.

Step 1: Report successful nodes without treating failed or blocked predecessors as permission to publish. Step 2: Edit the owned exercise file and run verify.py --practice etl-dag. Preserve caller inputs and the independently declared failure cases.

The solution adds a useful run summary without changing execution policy. Exact state assertions still protect failure propagation, while the success list can support a compact human-facing report. The cycle case remains an exception because no valid execution order exists under the declared dependencies.

A production scheduler needs persisted run identity, task attempt identity, retries, timeouts and resource limits. Backfill dependencies may differ from ordinary daily runs. Those policies should be explicit extensions, not assumptions smuggled into the small graph model. A task's successful process exit may also be weaker than verified durable output publication.

Explain the difference between topological readiness and permission to execute after predecessor results. Then trace a failure through two levels of descendants and one independent branch. A good answer accounts for all terminal states while keeping the graph's structural progress separate from task success.

For a stretch, add a second failure and return a deterministic event log with ready, blocked and completed transitions. Keep the scheduler single-process and bounded. If you later port the graph to an orchestration product, map each state and trigger rule deliberately instead of assuming similarly named states have identical semantics.

Optional stretch — after the core task works: Add a deterministic transition log and map the local policy explicitly before using a scheduler product.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-dag.py and run --practice etl-dag. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.