GA
Founding AI Infra Engineer
Accepting applicationsGoaly AI · Sunnyvale, CA
Full-Time Entry AIC++MentorPythonai
Posted
5d ago
Category
Manufacturing
Experience
Entry
Country
United States
About Goaly At Goaly, our mission is to make custom AI affordable for every business. Our founding team comes from the front lines of top AI labs and tech giants (Meta MSL, TikTok AI, Google DeepMind, xAI, Microsoft Research, etc.), where we built large-scale training infrastructure powering trillion-parameter models and scaled GenAI models to a global user base. Now, we are building something we wish we had before: a platform that makes training and adapting custom AI affordable for all modern companies, not just Big Tech. Our north star is ambitious: for a domain-specific task, reach 90% of SOTA performance at less than 10% of the cost. To get a taste of what we are doing, see our first tech blog.
Role Description You will sit at the intersection of systems engineering and applied AI, building specialized AI infrastructure that keeps frontier models fast, reliable, and cost-effective. Key responsibilities:
AI Performance Efficiency: Improve LLM training and inference efficiency through better memory utilization, optimized parallelism, and kernel-level innovations to serve frontier models in both GPU-poor and GPU-rich scenarios.
Training stability & RL robustness: Build scalable, stable training and RL pipelines with strong reproducibility, observability, and debuggability.
System-aware co-design: Prototype research ideas directly in training and inference stacks (e.g., parallelism strategies, attention kernels, RL training pipelines) and validate them at scale.
Scalability & Infrastructure: Own end-to-end training and inference infrastructure — from data ingestion and checkpointing to multi-node and multi-cloud orchestration.
System-aware research: Prototype research ideas directly in training and inference stacks (e.g., parallelism strategies, attention kernels, RL training pipelines) and validate them at scale.
Production enablement: Work closely with researchers and product engineers to turn new algorithms into reliable, production-ready systems.
Qualifications
Strong foundation in Computer Science and Software Development, including data structures, algorithms, distributed systems, and production-grade coding practices.
5+ years building or operating large-scale AI/ML infrastructure at scale, ideally supporting LLMs with latest model capabilities (agent, coding, reasoning etc).
Deep understanding of GPU architecture, distributed training frameworks (PyTorch, DeepSpeed, Megatron, Ray), and parallelism strategies.
Hands-on experience running inference stacks (vLLM / SGLang, TGI, Triton) and optimizing them via low-level profiling.
Strong software engineering fundamentals in Python and one of C++/Rust/Go, with clean, reliable code shipped to production.
Working knowledge of modern data pipelines, feature stores, and vector databases used in production AI systems.
Comfort automating infrastructure with Kubernetes, Ray, agent workflow automations such as LangChain / LangGraph as well as AI observability stacks (Weights&Biases, Prometheus, Grafana, OpenTelemetry).
Bonus Points
Experience deploying open-source LLMs (Qwen, DeepSeek, Kimi, GLP, Llama etc) or training custom foundation models in coding, reasoning, agent etc.
Contributions to AI/ML systems tooling (compilers, kernels, inference runtimes) or open-source infrastructure projects.
Background in RL, SFT, PEFT / LoRA, training data processing, evaluation, agent harnesses, sandbox environment / tool optimizations that hardens the end-to-end production AI systems.
Why join us?
Expert Mentorship: Partner with AI veterans who have trained trillion-parameter models at scale and applied it to solve real-worldproblems in the billon-user products
Compute Freedom: Access to abundant GPU cluster resources—don't let your creativity be limited by compute.
Flat Culture: Flat management structure that rejects office politics and values only technology and results.
Competitive Compensation: Competitive full-time offers, huge upside, and extra equity incentives when hitting key milestones.
Show more Show less
Role Description You will sit at the intersection of systems engineering and applied AI, building specialized AI infrastructure that keeps frontier models fast, reliable, and cost-effective. Key responsibilities:
AI Performance Efficiency: Improve LLM training and inference efficiency through better memory utilization, optimized parallelism, and kernel-level innovations to serve frontier models in both GPU-poor and GPU-rich scenarios.
Training stability & RL robustness: Build scalable, stable training and RL pipelines with strong reproducibility, observability, and debuggability.
System-aware co-design: Prototype research ideas directly in training and inference stacks (e.g., parallelism strategies, attention kernels, RL training pipelines) and validate them at scale.
Scalability & Infrastructure: Own end-to-end training and inference infrastructure — from data ingestion and checkpointing to multi-node and multi-cloud orchestration.
System-aware research: Prototype research ideas directly in training and inference stacks (e.g., parallelism strategies, attention kernels, RL training pipelines) and validate them at scale.
Production enablement: Work closely with researchers and product engineers to turn new algorithms into reliable, production-ready systems.
Qualifications
Strong foundation in Computer Science and Software Development, including data structures, algorithms, distributed systems, and production-grade coding practices.
5+ years building or operating large-scale AI/ML infrastructure at scale, ideally supporting LLMs with latest model capabilities (agent, coding, reasoning etc).
Deep understanding of GPU architecture, distributed training frameworks (PyTorch, DeepSpeed, Megatron, Ray), and parallelism strategies.
Hands-on experience running inference stacks (vLLM / SGLang, TGI, Triton) and optimizing them via low-level profiling.
Strong software engineering fundamentals in Python and one of C++/Rust/Go, with clean, reliable code shipped to production.
Working knowledge of modern data pipelines, feature stores, and vector databases used in production AI systems.
Comfort automating infrastructure with Kubernetes, Ray, agent workflow automations such as LangChain / LangGraph as well as AI observability stacks (Weights&Biases, Prometheus, Grafana, OpenTelemetry).
Bonus Points
Experience deploying open-source LLMs (Qwen, DeepSeek, Kimi, GLP, Llama etc) or training custom foundation models in coding, reasoning, agent etc.
Contributions to AI/ML systems tooling (compilers, kernels, inference runtimes) or open-source infrastructure projects.
Background in RL, SFT, PEFT / LoRA, training data processing, evaluation, agent harnesses, sandbox environment / tool optimizations that hardens the end-to-end production AI systems.
Why join us?
Expert Mentorship: Partner with AI veterans who have trained trillion-parameter models at scale and applied it to solve real-worldproblems in the billon-user products
Compute Freedom: Access to abundant GPU cluster resources—don't let your creativity be limited by compute.
Flat Culture: Flat management structure that rejects office politics and values only technology and results.
Competitive Compensation: Competitive full-time offers, huge upside, and extra equity incentives when hitting key milestones.
Show more Show less