Getting 50 GB/S Back from the Apple Neural Engine
Source Entity
Hacker News
An RTL performance erratum in the Apple M3 Neural Engine has been identified, causing significant throughput throttling for specific weight sizes. By bypassing this problematic DMA path, researchers have achieved a more than two-fold increase in token generation speeds for LLMs.
Uncovering the Apple M3 Neural Engine Throughput Bottleneck
Recent technical profiling of the Apple M3 Neural Engine (ANE) has uncovered a significant performance erratum within its Register Transfer Level (RTL) design. This hardware-level quirk effectively throttles the DRAM weight streaming throughput, reducing performance from a nominal 45–60 GB/s down to a sluggish 17–19 GB/s. The issue is triggered specifically when the total weight size of a neural network model is an integer multiple of 1 MiB. Given that many modern machine learning models align their parameters in such blocks, this discovery impacts nearly half of the models currently utilized in the ANEMLL framework.
The Mechanics of the Erratum
The root cause of this performance degradation lies within the kernel DMA engine's speculative prefetch ring. When the weight size criteria are met, the engine inadvertently triggers a suboptimal path, resulting in the observed throughput collapse. Profiling data revealed stark discrepancies in performance; for instance, at a hidden dimension (N) of 4096, a dimension (D) of 1536 performed nearly three times faster than a dimension of 2048. This discrepancy highlights how seemingly minor architectural alignment choices can lead to massive variances in real-world AI inference capabilities.
Impact on LLM Inference
The implications of this discovery are profound for developers deploying Large Language Models (LLMs) on Apple Silicon. By identifying and bypassing the problematic DMA path, researchers observed dramatic gains in token throughput. For the Llama 3.2 1B model, token generation speed jumped from 10.0 to 24.3 tokens per second, with DRAM usage efficiency climbing from 24.7 to 60.0 GB/s. Similarly, the Qwen3-8B model saw its throughput more than double, rising from 1.36 to 2.97 tokens per second as DRAM utilization improved from 22.4 to 48.7 GB/s.
Broader Implications for Apple Silicon
This finding serves as a critical case study in the complexities of proprietary hardware optimization. Apple’s M-series chips are highly integrated, and while their unified memory architecture provides significant speed advantages, it also relies on precise coordination between the Neural Engine and the memory controller. When these hardware-level optimizations fail to trigger correctly, the performance penalty is substantial, effectively negating the power of the silicon. It underscores the necessity for low-level profiling tools in the AI development ecosystem.
Conclusion and Future Trends
As AI models continue to grow in size and complexity, the ability to maximize raw hardware bandwidth becomes a defining competitive advantage. This erratum suggests that while Apple Silicon is highly capable of running local inference, software-side optimizations—specifically those that account for hardware-level alignment—are vital. Moving forward, developers will likely need to adopt more sophisticated memory alignment strategies to ensure their models avoid such "problematic paths" in the ANE's DMA engine, ensuring that they consistently extract the full potential from Apple's specialized neural hardware.