IA
Global Inference Library Engineer
Accepting applicationsInfinity Artificial Intelligence Institute · San Francisco Bay Area
Full-Time Entry AIC++Pythonanalog
Posted
1d ago
Category
Design
Experience
Entry
Country
United States
Global Inference Library - Member of Technical Staff
Company: Infinity · Team: Systems / AI Infrastructure · Location: San Francisco (on-site) · Type: Full-time
The Mission
Every accelerator that comes online needs an inference stack, and today that stack is hand-built per chip, per model, per optimization - a permanent, growing backlog of engineering work that scales linearly with the number of chips and models in the world.
We're building Infy: a global inference library, generated and continuously maintained by AI, that targets every major chip - NVIDIA, AMD, Trainium, TPU, Maia, Cerebras, Groq, Tenstorrent, and dozens more - across every model category, from LLMs to vision, audio, and robotics. Think vLLM, but for every accelerator on the market, kept automatically up to date as chips, models, and optimization techniques change.
What you'll work on
Depending on your strengths, you'll own one or more layers of the system:
Optimization agents - the strategy → analyze → generate → curate loop that turns a chip + model pair into an optimized kernel or serving path, and the orchestrator that runs this across many chips and models in parallel.
Kernel optimization loop - the hierarchical agent system that iterates on individual kernels (matmul, attention variants, normalization, RoPE, MoE, collectives) against reference implementations and measured-peak performance gates, drawing on and growing a registry of thousands of known kernels.
Library generator - the system that takes a validated set of optimized components and assembles them into a real, installable inference library per chip, with a standards-compatible serving interface.
Hardware probe - agents that build a structured representation of a chip's memory hierarchy, compute regions, and inter-component communication characteristics, so optimization strategy can branch correctly on the chip's actual architecture.
Coverage tracking - the enablement and optimization tables that track, per chip and per model category, what's implemented and how fast it runs, and that drive prioritization of what to build next.
SOTA paper replication - agents that read inference-optimization papers and automatically implement and validate the techniques they describe, so the library's optimizations keep pace with published research rather than lagging behind it.
Continuous tracking - the pipeline that watches for new papers, new hardware, and new models, and feeds that into what Infy builds and rebuilds next.
Benchmark harness - the metrics layer (TTFT, TPOT, ITL, E2EL, goodput, and model-category-specific equivalents like image-generation latency) that every optimization is measured against.
Inference service - the gateway, router, and billing layers that turn the library into a running, AI-optimized inference service people can actually call.
What we're looking for
We care more about depth and range than a specific checklist, but strong candidates will have most of:
Real experience with ML inference internals - vLLM or similar serving stacks, attention kernel variants, KV cache management, quantization, continuous batching.
Comfort working across heterogeneous hardware - GPU kernels (CUDA, ROCm/HIP, Triton), and ideally exposure to non-GPU execution models (systolic arrays, dataflow, wafer-scale, in-memory/analog compute).
Hands-on experience building with coding agents / LLMs - prompting, tool-use loops, evaluating and constraining model output, and designing systems where the model writes and optimizes code while tests and benchmarks catch regressions.
Fluency in Python and at least one systems language (Rust, C, or C++).
Comfort reading and implementing ideas directly from research papers, not just from existing open-source code.
Nice to have
Contributed to vLLM, TVM, LLVM, MLIR, or a hardware vendor's compiler/runtime stack.
Experience with distributed inference or training internals (NCCL, Megatron-LM, DeepSpeed) and collective-communication algorithms.
Familiarity with model categories beyond LLMs - vision, audio, multimodal, or robotics inference.
Experience building or maintaining a benchmark suite or performance regression system at scale.
Background reproducing results from ML systems papers (KernelBench-style benchmarks, evolutionary/search-based kernel optimization).
Who you are
You want to build the thing that makes every chip's inference performance a solved problem instead of a standing engineering project. You think in terms of systems that scale across hardware and models rather than one-off implementations, you're energized by AI agents doing the generation and optimization work under a tight benchmark harness, and you want to help set the global standard for how inference libraries get built and maintained.
Infinity is an early-stage AI infrastructure research company building the software layer that makes non-NVIDIA chips competitive for AI inference. Rather than relying on scarce human kernel engineers, we use AI to automatically generate, test, and optimize the low-level code that determines how efficiently a chip runs AI models. We've signed or are negotiating design partnerships with d-Matrix, AMD, AWS Trainium, Microsoft (Maia and Nexus), Qualcomm, and others. Founded by Jeremy Nixon (former Google Brain; co-founder of AGI House with Andrej Karpathy), Infinity has raised $15M from investors including the founder of Intercom, the VP of AI at AMD, and the founder of MLCommons. We're headquartered in San Francisco.
Show more Show less
Company: Infinity · Team: Systems / AI Infrastructure · Location: San Francisco (on-site) · Type: Full-time
The Mission
Every accelerator that comes online needs an inference stack, and today that stack is hand-built per chip, per model, per optimization - a permanent, growing backlog of engineering work that scales linearly with the number of chips and models in the world.
We're building Infy: a global inference library, generated and continuously maintained by AI, that targets every major chip - NVIDIA, AMD, Trainium, TPU, Maia, Cerebras, Groq, Tenstorrent, and dozens more - across every model category, from LLMs to vision, audio, and robotics. Think vLLM, but for every accelerator on the market, kept automatically up to date as chips, models, and optimization techniques change.
What you'll work on
Depending on your strengths, you'll own one or more layers of the system:
Optimization agents - the strategy → analyze → generate → curate loop that turns a chip + model pair into an optimized kernel or serving path, and the orchestrator that runs this across many chips and models in parallel.
Kernel optimization loop - the hierarchical agent system that iterates on individual kernels (matmul, attention variants, normalization, RoPE, MoE, collectives) against reference implementations and measured-peak performance gates, drawing on and growing a registry of thousands of known kernels.
Library generator - the system that takes a validated set of optimized components and assembles them into a real, installable inference library per chip, with a standards-compatible serving interface.
Hardware probe - agents that build a structured representation of a chip's memory hierarchy, compute regions, and inter-component communication characteristics, so optimization strategy can branch correctly on the chip's actual architecture.
Coverage tracking - the enablement and optimization tables that track, per chip and per model category, what's implemented and how fast it runs, and that drive prioritization of what to build next.
SOTA paper replication - agents that read inference-optimization papers and automatically implement and validate the techniques they describe, so the library's optimizations keep pace with published research rather than lagging behind it.
Continuous tracking - the pipeline that watches for new papers, new hardware, and new models, and feeds that into what Infy builds and rebuilds next.
Benchmark harness - the metrics layer (TTFT, TPOT, ITL, E2EL, goodput, and model-category-specific equivalents like image-generation latency) that every optimization is measured against.
Inference service - the gateway, router, and billing layers that turn the library into a running, AI-optimized inference service people can actually call.
What we're looking for
We care more about depth and range than a specific checklist, but strong candidates will have most of:
Real experience with ML inference internals - vLLM or similar serving stacks, attention kernel variants, KV cache management, quantization, continuous batching.
Comfort working across heterogeneous hardware - GPU kernels (CUDA, ROCm/HIP, Triton), and ideally exposure to non-GPU execution models (systolic arrays, dataflow, wafer-scale, in-memory/analog compute).
Hands-on experience building with coding agents / LLMs - prompting, tool-use loops, evaluating and constraining model output, and designing systems where the model writes and optimizes code while tests and benchmarks catch regressions.
Fluency in Python and at least one systems language (Rust, C, or C++).
Comfort reading and implementing ideas directly from research papers, not just from existing open-source code.
Nice to have
Contributed to vLLM, TVM, LLVM, MLIR, or a hardware vendor's compiler/runtime stack.
Experience with distributed inference or training internals (NCCL, Megatron-LM, DeepSpeed) and collective-communication algorithms.
Familiarity with model categories beyond LLMs - vision, audio, multimodal, or robotics inference.
Experience building or maintaining a benchmark suite or performance regression system at scale.
Background reproducing results from ML systems papers (KernelBench-style benchmarks, evolutionary/search-based kernel optimization).
Who you are
You want to build the thing that makes every chip's inference performance a solved problem instead of a standing engineering project. You think in terms of systems that scale across hardware and models rather than one-off implementations, you're energized by AI agents doing the generation and optimization work under a tight benchmark harness, and you want to help set the global standard for how inference libraries get built and maintained.
Infinity is an early-stage AI infrastructure research company building the software layer that makes non-NVIDIA chips competitive for AI inference. Rather than relying on scarce human kernel engineers, we use AI to automatically generate, test, and optimize the low-level code that determines how efficiently a chip runs AI models. We've signed or are negotiating design partnerships with d-Matrix, AMD, AWS Trainium, Microsoft (Maia and Nexus), Qualcomm, and others. Founded by Jeremy Nixon (former Google Brain; co-founder of AGI House with Andrej Karpathy), Infinity has raised $15M from investors including the founder of Intercom, the VP of AI at AMD, and the founder of MLCommons. We're headquartered in San Francisco.
Show more Show less