GPU Offload in Rust: Portable, Safe, and Fast
Rust's compiler can now write GPU code itself, without giving up memory safety
GPU programming has long forced a choice between fast-but-unsafe languages like CUDA/C++ and safe-but-GPU-incapable Rust. This paper builds GPU offloading directly into the Rust compiler (rustc) and LLVM, keeping Rust's memory-safety guarantees while running on NVIDIA and AMD GPUs. Tested on the RAJAPerf benchmark suite, the resulting GPU kernels run competitively against hand-written CUDA and HIP code.
METAL MEDIA explanatory visual
Rust's compiler can now write GPU code itself, without giving up memory safety
- 01Three interfaces are offered: fully automatic data transfer for casual use, integration with vendor libraries like cuBLAS/rocBLAS, and manual memory control for experts who need predictable performance
- 02A safety abstraction called 'Region' lets multiple GPU threads write to disjoint parts of the same array without unsafe code, solving Rust's usual 'one mutable reference at a time' restriction for parallel kernels
- 03A two-pass compilation pipeline separates host (CPU) and device (GPU) compilation to avoid CPU-only instructions leaking into GPU code, integrated directly into rustc and LLVM's Offload infrastructure
- 04On AMD MI250X and NVIDIA H100/RTX GPUs, individual kernel execution times were competitive with native CUDA/HIP, though whole-program runtime ranged from 32% faster to 46% slower depending on the benchmark, due to differences in host-device data transfer handling
- 05A naive, unoptimized implementation was measured to be over 400x slower on MI250X, showing how essential the paper's data-transfer optimizations are
What they did
- Three interfaces are offered: fully automatic data transfer for casual use, integration with vendor libraries like cuBLAS/rocBLAS, and manual memory control for experts who need predictable performance
- A safety abstraction called 'Region' lets multiple GPU threads write to disjoint parts of the same array without unsafe code, solving Rust's usual 'one mutable reference at a time' restriction for parallel kernels
- A two-pass compilation pipeline separates host (CPU) and device (GPU) compilation to avoid CPU-only instructions leaking into GPU code, integrated directly into rustc and LLVM's Offload infrastructure
- On AMD MI250X and NVIDIA H100/RTX GPUs, individual kernel execution times were competitive with native CUDA/HIP, though whole-program runtime ranged from 32% faster to 46% slower depending on the benchmark, due to differences in host-device data transfer handling
- A naive, unoptimized implementation was measured to be over 400x slower on MI250X, showing how essential the paper's data-transfer optimizations are
| host.o |
|---|
| .text / .rodata |
| .llvm.offloading |
| llvm_offload_entries |
Why it matters
High-performance and scientific computing has long depended on memory-unsafe languages like C/C++/CUDA because Rust couldn't reach the GPU efficiently; this work removes that trade-off. It gives developers a vendor-neutral, safety-checked path to GPU acceleration instead of being locked into a single hardware vendor's unsafe API.
Terms in this paper
- noalias · a compiler hint meaning a pointer doesn't overlap with others in memory, enabling optimizations
- MIR · an internal intermediate code representation used inside the Rust compiler
- LLVM Offload · LLVM's underlying infrastructure for moving computation from CPU to GPU
- RAJAPerf · a benchmark suite used to compare CPU/GPU performance
- unsafe block · a code region in Rust where the compiler's automatic safety checks are turned off and the programmer takes responsibility
Original abstract (English)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears