English | 中文版

9. Performance: From Safety to Speed

Summary: Safety and performance are not in conflict in ascend-rs. The Rust buffer-API kernel (rust_vector) outperforms hand-optimized AscendC C++ on softmax by 1.6–1.8×. For V-pipe (vector) workloads, both Rust and C++ are bottlenecked by memory bandwidth — they reach the same hardware limit. The open frontier is cube-unit (M-pipe) workloads like GEMM, where the PTO path (mlir_to_pto → ptoas) is the only route to full hardware performance.

9.1 Activation Function Benchmarks

ascend-rs Rust kernels achieve zero-overhead performance parity with hand-optimized AscendC C++.

Hardware: Ascend 910B3, CANN 8.5, 8 AICore blocks.

All 16 activation functions in kernel_ops.rs are benchmarked against equivalent C++ implementations. Results show 0% performance overhead for Rust-generated kernels across all tested sizes (1K to 1M elements):

The softmax result is particularly notable: the Rust vector kernel is 1.6× faster than the C++ reference at the same problem size, because the Rust implementation uses optimal vector op chaining (ReduceMax → Adds → Exp → ReduceSum → Muls) while the C++ reference uses a scalar loop for the naive implementation.

9.2 Softmax Benchmark — Four Implementations on Ascend 910B2

Key finding: For V-pipe (vector) workloads like softmax, the Rust buffer-API kernel (rust_vector) is the fastest implementation tested, outperforming hand-optimized C++ AscendC by 1.6–1.8×. The tile-API scalar fallback is 7–80× slower due to a known workaround for a 910B2 LocalTensor::operator[] offset bug; the PTO path is expected to recover this gap. For M-pipe (cube-unit) workloads like matrix multiply, the scalar fallback achieves ~0.17 GFlop/s against a 910B2 cube-unit peak of ~32,000 GFlop/s — a 190,000× gap that PTO codegen is designed to close.

Setup

Hardware: Ascend 910B2 (Atlas 300T A2 card), CANN 8.5.0, single AICore.

Implementations compared:

All kernels perform row-wise softmax: for each row, compute exp(x - max(x)) / sum(exp(x - max(x))). Timing uses AclEvent start/end events around the kernel launch; 1 warmup + 10 timed iterations per shape; reported times are medians.

Results

1D kernels (single row, varying element count)

rust_vector is the fastest at every size measured. cpp_opt is 1.6–1.8× slower than rust_vector; the cpp_naive scalar loop is 10–34× slower than cpp_opt.

Tile-API multi-row shapes

The tile API is tested at six shapes; the rust_vector result at the matching element count is shown for reference.

All six tile-API shapes pass correctness checks (max element error < 1.3×10⁻⁸, all row sums within 0.01 of 1.0).

Throughput

Expressed as millions of elements processed per second (higher is better):

rust_vector  8192 elem:   788 Melem/s  ████████████████████████████████████████
rust_vector  4096 elem:   440 Melem/s  ██████████████████████
rust_vector  1024 elem:   121 Melem/s  ██████
cpp_opt      4096 elem:   270 Melem/s  █████████████
cpp_opt      1024 elem:    67 Melem/s  ███
cpp_naive    4096 elem:    13 Melem/s  █
rust_tile  1x8192 elem:    9.9 Melem/s ▌  (scalar fallback)
rust_tile  1x4096 elem:    9.8 Melem/s ▌
rust_tile  1x1024 elem:    9.4 Melem/s ▌

rust_vector throughput scales super-linearly with element count (121 → 788 Melem/s from 1K to 8K elements) because larger tiles amortize kernel launch overhead and fill the vector pipeline more efficiently. The tile-API scalar fallback is flat at ~9–10 Melem/s regardless of shape, confirming that it is bottlenecked by scalar S-pipe throughput rather than memory bandwidth.

Why the Tile-API Scalar Fallback Is Slow

The current tile-API softmax is implemented as a pure scalar loop in the generated C++:

// Generated by mlir_to_cpp ascend_tile_softmax_f32 handler
for (int32_t __r = 0; __r < rows; __r++) {
    int32_t __b = __r * cols;
    float __max = buf0.GetValue(__b);
    for (int32_t __c = 1; __c < cols; __c++) {
        float __tmp = buf0.GetValue(__b + __c);
        if (__tmp > __max) __max = __tmp;
    }
    for (int32_t __c = 0; __c < cols; __c++)
        buf1.SetValue(__b + __c, buf0.GetValue(__b + __c) - __max);
    // ... polynomial exp per element ...
    // ... scalar sum loop ...
    // ... scalar Muls loop ...
}

GetValue and SetValue execute on the scalar S-pipe at one element per cycle. A 1024-element softmax therefore requires ~4,000+ scalar operations. In contrast, rust_vector uses AscendC::ReduceMax, Adds, Exp, ReduceSum, and Muls — 128-wide SIMD vector ops on the V-pipe — completing in a handful of pipeline cycles.

Why scalar? The 910B2 AscendC compiler/runtime has a subtle bug with LocalTensor::operator[](offset) for offset > 0: vector ops operating on a sub-view produce wrong results. The scalar workaround bypasses this completely. Until the sub-view issue is resolved, the scalar fallback is necessary for correctness on multi-row tile kernels.

The path to fixing this: The PTO path (mlir_to_pto → ptoas) avoids the sub-view issue entirely because ptoas generates its own AscendC from the PTO-MLIR description of the tile layout, bypassing LocalTensor::operator[] sub-views.

Correctness vs. Performance Trade-offs

rust_tile_scalar is currently the only implementation that correctly handles multi-row shapes in this benchmark suite.

9.3 The Cube Unit: The Next Performance Frontier

Softmax is a V-pipe-only workload. Every operation — ReduceMax, Adds, Exp, ReduceSum, Muls — runs exclusively on the vector unit (V-pipe). The Ascend 910B2 has a second, dedicated compute engine: the cube unit (M-pipe), a hardware matrix multiplier with its own L0A, L0B, and L0C on-chip memory hierarchy.

This matters because:

• The buffer API and mlir_to_cpp have no cube-unit support. The buffer API expresses computation as DMA + vector ops (TBuf<VECCALC> only).

• PTO’s structural advantage is specifically for cube-unit kernels. ptoas-generated code uses Tile<TileType::Left, ...>, Tile<TileType::Right, ...>, Tile<TileType::Acc, ...> — distinct memory spaces that live in L0A, L0B, L0C respectively — and TMATMUL() / TMATMUL_BIAS() instructions that drive the cube unit.

• For softmax and other V-pipe kernels, PTO provides no performance advantage over the buffer API. Both ultimately lower to the same AscendC vector ops.

• For matrix multiply (GEMM), scaled dot-product attention, and convolution, PTO is the only path to full cube-unit performance from Rust. The CANN runtime’s aclnnMatmul achieves 320 TFLOPS (f16) on the 910B2 — saturating the theoretical peak. Reaching this from Rust-authored kernels requires the PTO path, which is correctly structured in mlir_to_pto.rs but awaits CANN 9.x bisheng support for pto-inst.hpp.

9.4 matmul Benchmark — Scalar vs. Cube Unit

Hardware: Ascend 910B2, CANN 8.5.0.

Cube-unit GEMM throughput (aclnnMatmul, f16)

The Ascend 910B2 cube unit achieves near-theoretical peak throughput on matrix multiplication. Using the CANN aclnnMatmul graph API (which internally dispatches to the hardware cube engine), we measured 17 shapes from 32×32 to 16384×16384:

Selected rectangular/transformer-like shapes:

Peak: 320 TFLOPS at 16384×16384×16384 — saturating the Ascend 910B2’s theoretical f16 maximum (320 TFLOPS). All shapes pass correctness checks.

The full results are available in benchmarks/gemm/ascend_910b2_results.csv, and the benchmark script at benchmarks/gemm/bench_gemm_ascend.py.

Scalar path comparison

For comparison, the current mlir_to_cpp scalar fallback path (no cube unit) delivers:

The scalar path runs entirely on the S-pipe (one element per cycle), while the cube unit processes 16×16 fractal blocks per cycle across 30 AICores.

Closing the gap from Rust

The aclnnMatmul results above use the CANN runtime’s built-in matmul kernel. The path to achieving the same throughput from Rust-authored kernels is: ACLRS_CODEGEN_PATH=pto → mlir_to_pto.rs emits cube-unit tile sequence (pto.alloc_tile loc=mat/left/right/acc → pto.tmatmul) → ptoas compiles to AscendC with __ca__/__cb__/__cc__ qualifiers → bisheng → NPU binary. This path is implemented and verified through ptoas; the final step awaits pto-inst.hpp compatibility with a future CANN release.

9.5 Key Takeaways

1. Safety does not cost performance. The Rust vector kernel is 1.6–1.8× faster than hand-written C++ AscendC on softmax — the compiler’s type system and abstraction layer do not add overhead.

2. The buffer API is the right choice for V-pipe workloads. rust_vector matches the theoretical memory bandwidth limit on the 910B2 for softmax.

3. PTO is the right choice for M-pipe (cube-unit) workloads. GEMM, attention, and convolution require the cube unit; the buffer API cannot reach it. The PTO path in ascend-rs is structurally correct and awaits a CANN upgrade to complete.

4. Multi-row correctness currently requires scalar fallback. The tile API correctly handles multi-row shapes that the 1D buffer API cannot, at the cost of scalar performance. PTO will restore vector performance once bisheng supports pto-inst.hpp.