Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Source Entity
Hacker News
Swiftlet is a new Swift and Metal-based runtime allowing large Qwen MoE models to run on Apple hardware with minimal RAM. By streaming weights, it enables 35B and 80B models to function on devices as small as an iPhone 17.
Breakthrough in Edge Computing: Swiftlet and Local LLMs
The landscape of on-device artificial intelligence has shifted dramatically with the introduction of Swiftlet, a runtime specifically engineered to run complex Mixture-of-Experts (MoE) models on Apple hardware. By leveraging Swift and Metal, the framework allows for the execution of high-parameter models like the Qwen3-Next 80B on consumer-grade hardware. This development marks a transition from cloud-dependent AI to true edge-native capabilities, challenging the conventional necessity for massive GPU clusters to handle sophisticated language models.
The Mechanics of Memory Efficiency
At the heart of Swiftlet’s performance is its innovative memory management strategy. Traditional LLM inference requires loading the entire model into RAM, which typically restricts large models to high-end workstations. Swiftlet circumvents this by keeping only the dense core of the model resident in memory while streaming the routed MoE weights from storage on demand. This architectural decision enables the Qwen3.6-35B-A3B model to function within a peak RAM footprint of just 2.6 GB, a fraction of what would be required under standard loading procedures.
Scaling to Mobile Architecture
Perhaps the most significant milestone is the successful deployment of a 35B parameter model on an iPhone 17. Operating within approximately 2.5 GB of RAM, this achievement demonstrates that the barriers to running large-scale intelligence on mobile devices are structural rather than purely computational. While current decode speeds are approximately 1 token per second, the ability to run a model of this class natively on a smartphone suggests that personal devices are rapidly evolving into powerful, private inference engines.
Implications for Privacy and Latency
By shifting the weight of inference from the cloud to the device, Swiftlet addresses critical concerns regarding data privacy and network latency. Users no longer need to transmit sensitive prompts to external servers, as the entire processing cycle occurs locally. Furthermore, this eliminates the bottleneck of internet connectivity, allowing for robust AI performance in offline or bandwidth-constrained environments, which is essential for the future of ubiquitous computing.
Future Trends and Technical Hurdles
While Swiftlet is currently functional and producing validated outputs, the project is entering an optimization phase. The current focus is on enhancing kernel speed, as the decode loop is currently dispatch-bound. As Apple continues to refine its Unified Memory Architecture and Neural Engine capabilities, we can expect the performance gap between cloud and edge inference to narrow further. Future iterations of Swiftlet will likely focus on optimizing the streaming throughput to make these models more responsive for real-time applications.
Conclusion
The emergence of Swiftlet represents a pivotal moment in the democratization of large language models. By enabling 80B-class models to run on Macs and 35B models on iPhones, the developers have effectively lowered the barrier to entry for high-performance AI. As optimization continues, the integration of these models into daily mobile workflows appears increasingly inevitable, signaling a new era for local, private, and powerful generative AI.