Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

William Moses, Massachusetts Institute of Technology

Photo of William Moses

Applying differentiable programming techniques and machine-learning algorithms to foreign programs requires developers to either rewrite their code in a machine-learning framework or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme can synthesize gradients for programs written in any language whose compiler targets LLVM IR, including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine learning-focused benchmark suite, including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5 times over AD on IR before optimization, allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine-learning workflows.

Abstract Author(s): William S. Moses, Valentin Churavy