SambaNova Explained: The Dataflow Chips Behind High-Speed AI Inference

Accelerating AI inference is one of the most critical technical challenges for enterprise AI adoption. In this context, SambaNova Systems’ SambaNova Cloud, announced in September 2024, has garnered significant attention by achieving inference speeds more than 10 times faster than industry-standard GPU-based systems.

SambaNova Cloud boasts performance capable of running Llama 3.1 405B models at 132 tokens per second and 70B models at 461 tokens per second¹, significantly exceeding the average 20 tokens per second of traditional GPU providers². Behind this technological innovation lies proprietary dataflow architecture and years of research by Stanford University researchers.

The Founding and Journey of SambaNova Systems

Founders and Stanford University Research

SambaNova Systems was co-founded in 2017 by Rodrigo Liang (CEO) and two prominent Stanford University professors, Kunle Olukotun and Chris Ré³.

Rodrigo Liang (CEO & Co-founder)

Born in Taiwan and raised in Brazil, Rodrigo Liang earned bachelor’s and master’s degrees in electrical engineering from Stanford University before establishing a long career in high-performance computing⁴. At the core of his career are his experiences at Sun Microsystems and Oracle.

At Sun Microsystems, he was involved in developing the world’s first multi-core processors, and later at Oracle, he served as Senior Vice President, overseeing SPARC processor and ASIC development divisions⁵. His experience designing cutting-edge processors and ASICs for enterprise servers later led to the development of dedicated AI chips at SambaNova.

Kunle Olukotun (Chief Technologist & Co-founder)

Stanford University Professor of Electrical Engineering and Computer Science, Kunle Olukotun is widely known as the “father of the multi-core processor”⁶. Also a member of the National Academy of Engineering, he led the Stanford Hydra Chip Multiprocessor research project and laid the foundation for current multi-core technology.

Chris Ré (Co-founder)

Stanford University Associate Professor of Computer Science Chris Ré is a machine learning and database expert as a member of Stanford AI Lab⁷. He belongs to the Statistical Machine Learning Group and Pervasive Parallelism Lab, conducting research in large-scale data processing and AI.

Funding Journey

The company has successfully raised large-scale funding since its founding:

Year	Round	Amount	Lead Investors
2018	Series A	$56M	GV, Temasek, Intel Capital
2020	Series B	$150M	Intel Capital-led
2021	Series C	$300M	SoftBank Vision Fund 2
2024	Series D	$676M	SoftBank, BlackRock, Intel Capital

Total funding raised exceeds $1 billion, reaching a valuation of $5 billion in 2021⁸. This scale represents one of the world’s highest levels for AI-specialized startups.

Technical Innovation: SN40L RDU and Dataflow Architecture

Design Philosophy Beyond GPU Limitations

The core of SambaNova’s technical advantage lies in a dedicated chip called the SN40L RDU (Reconfigurable Dataflow Unit)⁹. Traditional GPUs (Graphics Processing Units), as their name suggests, were designed for graphics processing and have the following structural inefficiencies for AI inference:

Redundant Memory Access: GPUs require multiple redundant calls to memory during AI processing
Legacy Architecture: Designs oriented toward graphics processing are not optimized for AI-specific computational patterns

The SN40L fundamentally solves these inefficiencies by adopting dataflow architecture¹⁰. In dataflow architecture, computations are executed based on data flow, eliminating the redundant memory access required by traditional GPUs.

Innovation in 3-Tier Memory Design

Another technical feature of the SN40L is its 3-tier memory architecture¹¹:

Large Memory (DRAM): Capable of storing terabytes of models in a single system
High Bandwidth Memory (HBM): Enables high-speed data access
Ultra-fast Memory (SRAM): Optimizes processing of most frequently accessed data

This design enables both execution of large-scale AI models and simultaneous execution of multiple models on a single system.

Features and Competitive Advantages of SambaNova Cloud Service

Service Tiers and Accessibility

SambaNova Cloud, launched in September 2024, is offered in three tiers¹²:

Tier	Launch Date	Features	Target Users
Free	September 2024	Free API access, no waiting list	Developers & Researchers
Developer	End of 2024	Higher rate limits	Application Developers
Enterprise	September 2024	Production-ready, scalable	Enterprises

Performance Benchmarks and Differentiation

The inference speeds provided by the company significantly exceed industry standards:

Llama 3.1 Model Performance Comparison

405B Model: SambaNova 132t/s vs GPU average 20t/s (6.6x faster)
70B Model: SambaNova 461t/s vs industry average (significant performance gap)

DeepSeek R1 671B Model

SambaNova: 250t/s vs GPU average 19t/s (more than 13x faster)¹³

These performance differences have been confirmed by third-party evaluations from independent AI assessment company Artificial Analysis¹⁴.

High-Speed Processing Without Sacrificing Precision

While many competitors achieve speed improvements by reducing model precision, SambaNova maintains full precision model execution¹⁵. This represents an important differentiating factor for practical AI applications.

Corporate Evaluation and Future Prospects

External Recognition and Industry Position

SambaNova Systems has received multiple prestigious awards for its technological innovation:

Forbes AI 50 (2025): Most influential AI company
Fast Company Most Innovative Companies (2025): 4th place in Computing category
Fortune Future 50 (2024): Ranked 6th¹⁶

Market Opportunities and Growth Strategy

The AI inference market continues rapid growth, and as enterprise AI adoption accelerates, demand for fast and efficient inference services is expanding. SambaNova aims for market expansion with the following strategies:

Focus on Enterprise Market: Production-ready services for large enterprises
Developer Ecosystem Building: Developer acquisition through free tier
Latest Model Support: Always supporting the largest open-source models

Kunle Olukotun, the company’s Chief Technology Officer, states: “Competitors cannot provide 405B models to developers due to their inefficient chips. Only SambaNova can provide the best open-source models at full precision and high speed”¹⁷.

Why SambaNova is Gaining Attention Now

Timing and Market Need Alignment

SambaNova is currently attracting attention due to several overlapping factors:

Enterprise AI Adoption Acceleration: Since ChatGPT’s emergence, enterprise AI utilization has entered practical stages, with inference speed becoming a bottleneck
Large Model Proliferation: 400B+ parameter models have reached practical levels, dramatically increasing demand for infrastructure that can efficiently execute them
Cost Efficiency Importance: Reducing AI operational costs has become a critical enterprise concern

Clear Technical Differentiation

SambaNova’s technical advantages are quantitatively measurable and clear. Speed differences of more than 10x represent a major differentiating factor that makes adoption decisions easier for customers.

SambaNova Systems exemplifies how deep academic expertise can translate into transformative commercial technology. The Stanford trio’s decision to tackle AI’s fundamental hardware limitations rather than compete on software alone has yielded remarkable results - inference speeds over 10x faster than competitors, enabling real-time AI applications previously thought impossible.

With over $1 billion raised and a valuation exceeding $5 billion, SambaNova has the resources to challenge established players. But their true differentiator lies in the RDU architecture’s ability to process entire models in memory, eliminating the bottlenecks that plague GPU-based systems. As enterprises increasingly demand not just AI capabilities but instantaneous AI responses, SambaNova’s specialized approach positions them to capture significant market share. Their success demonstrates that in the AI infrastructure race, revolutionary architecture can trump incremental optimization.