Auto-Vectorization and store-to-load forwarding

I have seen huge slowdowns from auto-vectorization for the bubble sort benchmark of John Hennesseys small integer benchmarks, because there is a partial overlap between the double-int store of one iteration and the double-int load of the next, and the usual store-to-load forwarding optimizations do not work; instead, CPUs take a slow path.

This kind of glass chin may not show up in the usual benchmarking (because it may not occur in the few hot loops of the benchmarks), but it may slow down other codes significantly (as demonstrated by the bubble-sort case).

Apparently the GCC maintainers noticed this themselves, and reduced the aggressiveness of auto-vectorization between gcc-12 and gcc-14, but there still is some. So I wanted to see if there still are cases where the auto-vectorization leads to slowdowns. So I wrote a program where the central loop is:

  for (i=0; i<n; i+=4) {
    long wl0, wl1;
    ns[i] = nl[i]+1;
    wl0 = wl[i];
    wl1 = wl[i+1];
    ws[i] = wl0;
    ws[i+1] = wl1;
  }

Here "n" stands for narrow, "w" for wide, "l" for load, and "s" for store (e.g., "ns" stands for "narrow store". gcc-14 compiles the body of the loop as follows:

gcc-14 -O                   gcc-14 -O3
loophead1:                  loophead2:
  mov (%rcx,%rax,8),%rdi      mov    (%rcx,%rax,8),%r10
  add $0x1,%rdi               lea    0x1(%r10),%r9
  mov %rdi,(%r10,%rax,8)      mov    %r9,(%rdi,%rax,8)
  mov (%rsi,%rax,8),%r9       movdqu (%rsi,%rax,8),%xmm0
  mov 0x8(%rsi,%rax,8),%rdi
  mov %r9,(%rdx,%rax,8)       movups %xmm0,(%rdx,%rax,8)
  mov %rdi,0x8(%rdx,%rax,8)
  add $0x4,%rax               add    $0x4,%rax
  cmp %r8,%rax		      cmp    %r8,%rax
  jb  loophead2		      jb     loophead2

In the -O3 version the two longs wl[i] and wl[i+1] are loaded with one movdqu instruction, and then are stored to ws[i] and ws[i+1] with one movups instruction, whereas the -O code loads and stores each long separately, resulting in a total of four mov instructions for these accesses. The remainder of the code is similar.

The surrounding code sets up nl, ns, wl, ws such that various memory dependencies are possible between the loads and the stores. Advancing i by 4 in each iteration allows to also set up cases where no memory dependences happen.

The loop shown above is exercised by the binaries stwfl-*, which take four parameters: offsets (in longs) of ns wl ws nl from a common base address (the ordering of the parameters puts nl last for historical reasons). E.g.,

  ./stwlf-x86_64-gcc-14-O3 2 2 4 0

means that ns and wl point to the same place, ws points 2 longs later, and nl points 2 longs earlier (4 longs before w2). Since i advances by 4, this means that ws[i] stores to the same place in one iteration that nl[i] loads from in the next iteration. In this way there is a dependence chain of stores and loads across iterations (two loads and two stores per iteration).

You can benchmark a whole group of parameter sets on stwlf-x86_64-gcc-14-O and stwlf-x86_64-gcc-14-O3 with:

  make

or benchmark the same group of parameter sets on a specific version with

  make bench CC=gcc-14 OPT=-O3

which will also build the needed binary if it does not exist yet. So for downloading and running the existing binaries on a specific machine you do:

  wget -O - https://www.complang.tuwien.ac.at/anton/stwlf/stwlf.zip | tar xfJ -
  cd stwlf
  make

The output contains numbers which are the number of cycles of one iteration of the loop. It also contains a description of the data flow of each iteration. The dependencies nl>ns and wl>ws through registers are built into the binary and cannot be changed by parameter settings, but there can be memory dependencies between any store and any load due to the way the parameters are set. These dependencies are shown as:

  =>   
  =_=> mixed narrow/wide access where the low-address long of the wide access is accessed by the narrow access.
  =^=> mixed narrow/wide access where the high-address long of the wide access is accessed by the narrow access.
  _=^> partially overlapping wide store to wide load dependency, with the low long of the store becoming the high long of the load (i.e., the store address is one long higher than the load address).

A "recurrence" means that there is a dependence chain within one iteration that continues with a loop-carried dependency to the start of the chain in the next iteration, resulting in a dependence chain throughout all iterations of the loop.

The dependencies are my guess at what causes these numbers. There may also be other microarchitectural effects at work that depend on the actual parameter set and the hardware, and you may want to look at the actual parameter sets for deeper insight.

The numbers come from one run (100_000 repetitions of the loop with 1000 iteratins), and at least on Zen 3 I have seen clustered variations for certain results (e.g., sometimes a given parameter set is measured as costing 3.00 cycles/iteration, and sometimes 3.15-3.17 cycles/iteration), but in the big scheme of things these variations are minor.

Here are some numbers for a few recent microarchitectures:

   Zen 4
  -O   -O3
 2.70  1.42 nl>ns wl>ws (maximally independent)
 3.09 21.00 nl>ns=_=>wl>ws
 3.09 21.00 nl>ns=^=>wl>ws
 3.00 29.00 nl>n2=_=>wl>ws=>nl (recurrence)
 3.00  8.80 wl>ws=>wl (recurrence), nl>ns (no recurrence)
 3.09  2.00 nl>ns=>nl (recurrence), wl>ws (no recurrence)
 3.07 21.00 nl>ns=>nl,wl>ws
 3.06 29.00 ws=_=>nl>ns=_=>wl
 3.09  2.00 wl>ws=_=>nl>ns
 3.09 22.33 wl>ws_=^>wl (recurrence) nl>ns (no recurrence)
 3.00 29.00 ws=_=>nl>ns=^=>wl>ws (recurrence), also ws_=^>wl