Typorithm · medium

cache & persist

cache-persistsparkcachepersistreuse

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn cache & persist

Transformations & Actions

When a DataFrame is reused across multiple actions, cache() keeps it in memory so it is not recomputed each time. persist(StorageLevel) chooses memory/disk tiers explicitly.

By default each action recomputes a DataFrame's whole lineage from the source. If you run several actions on the same df, that work repeats every time.

cache() marks the DataFrame so the first action materializes it and stores the result in memory; later actions read the cached data instead of recomputing.

persist(StorageLevel) is the general form — it lets you pick a tier (memory only, memory-and-disk, disk only) when the data is too big to fit purely in RAM. Caching pays off only when a DataFrame is used by more than one action.

cache() keeps a DataFrame in memory after first use.

Later actions reuse the cache instead of recomputing.

persist(StorageLevel) picks memory/disk tiers.

df.cache();
df.count();
df.groupBy("country").count().show();

Read the implementation

Connect each part of the code to the concept before you type.

1. Mark for caching

cache() marks the DataFrame to be kept in memory the first time it is materialized.

df.cache();

2. Materialize once

The first action (count) computes the DataFrame and populates the cache.

df.count();

3. Reuse cached data

This second action reads from the cache instead of recomputing df from scratch.

df.groupBy("country").count().show()

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.