simd-anim
All strategy pages

02 Strategy · compute vs memory

Roofline

A core is only as fast as its data supply. Slide the two roofs, then find — and work around — the wall you are actually hitting.

The roofline model charts a machine’s two limits — peak compute and memory supply — against arithmetic intensity: the FLOPs a kernel performs for every byte it moves. Illustrative values — set peak compute and memory supply to your own hardware. The model is generic: performance = min(peak, supply × intensity).

Roofline · simd-anim Roofline chart: a rising memory roof and a flat compute roof meet at the ridge point; shapes of work are plotted by their arithmetic intensity. FMLA FMOPA @128 FMOPA @512 Arithmetic intensity — FLOP per byte Performance — FLOP per cycle

Memory roof: supply × intensity Compute roof: peak
Ridge: peak ÷ supply Dots: accent = memory-bound, dark = compute-bound

Work Intensity (FLOP/B) Reachable (FLOP/cycle) Limited by
FMLA (element pair) 0.25 — —
FMOPA @ SVL 128 1 — —
FMOPA @ SVL 512 4 — —

Find the wall first

Halve the clock. If the work takes the same time, it never depended on compute — that is the signature of a bandwidth wall.

Compute time —
Memory time —
Total —

Illustrative normalized units — compute time scales with the clock, memory time does not.

Wall → signature → lever

Wall Signature Levers
Bandwidth Halving the clock changes nothing Block for reuse, pack to contiguous layouts, non-temporal stores, narrower dtypes
Latency Utilisation far below peak even when everything fits in cache Prefetch ahead, unroll to keep more misses in flight, large pages
Capacity Throughput falls off a cliff as the working set grows Shrink tiles to fit L1/L2, keep accumulators in registers, stream in segments
Scaling Extra cores stop helping before bandwidth saturates Cut per-core traffic, keep data local, plan for the per-core share

Reading the picture

  1. Intensity is the currency

    Count the bytes a piece of work reads for every FLOP it produces. Element-wise work reads four bytes per FLOP (0.25 FLOP/byte); an outer product reuses each element across a whole row or column, lifting FMOPA to 1 FLOP/byte at SVL 128 and 4 at SVL 512. Loads, stores and shuffles spend bytes and produce nothing — they are pure traffic.

  2. Crossing the ridge

    The ridge sits at peak ÷ supply. Work to its left is bandwidth-bound: the only way over is more FLOPs per byte — blocking the K loop so caches keep serving reused data, and keeping accumulators in registers (a wide ZA tile is architectural space for exactly that). This is the general reason ARM v9 keeps pushing wide vectors and outer products.

  3. Latency is bandwidth’s shadow

    A bandwidth number only becomes real when latency is hidden: prefetching ahead, enough outstanding misses in flight, large pages so TLB walks do not eat the pipeline, and non-temporal stores for write-once results. Same math on any machine — the techniques are generic.

  4. More probes

    Grow the working set until throughput drops — that cliff is a cache capacity boundary (L1, then L2, then memory). Add cores until scaling flattens — that is a shared resource, not a core limit. And read the counters: cycles, misses per level, bytes per second. Measure, then trust the roofline.

  5. Tuning: move bytes smartly

    Packing a strided matrix into contiguous panels turns many small reads into few wide ones; prefetching with the right distance starts fills early; non-temporal stores skip the read-for-ownership a write-once result never needs. Same algorithm, better supply.

  6. Tuning: overlap and unroll

    Software pipelining lets packing and computing run at the same time on different data. Unrolling keeps enough independent loads in flight: the bandwidth you can use equals concurrency ÷ latency, so hiding latency is the price of the number on the memory roof.

  7. Tuning: shrink the data

    Half precision halves the bytes; 8-bit integers quarter them — the same kernel shape with two to four times the intensity, for free. Let predicates own the tails too, so no padding or scalar cleanup is fetched and never used.