SambaNova Systems: Breaking Down the RDU Chip and Architecture Powering High-Speed AI Inference

What is SambaNova Systems?

SambaNova Systems is an AI hardware and software company founded in 2017 by researchers from Stanford University. The company provides high-performance AI inference platforms built around its proprietary RDU (Reconfigurable Dataflow Unit) chips, opening new horizons in enterprise AI computing.

Founders and Leadership

Three Co-founders

SambaNova Systems was founded by three experts with extensive experience in AI and semiconductor fields.

Rodrigo Liang (CEO and Co-founder)

Holds MS and BS degrees in Electrical Engineering from Stanford University
Over 20 years of semiconductor engineering experience at Sun Microsystems and Oracle
Led teams building high-performance processors for enterprise systems

Kunle Olukotun (Co-founder and Chief Technologist)

Professor of Electrical Engineering and Computer Science at Stanford University
Known as the “father of the multi-core processor”
Leader of the Stanford Hydra Chip Multiprocessor (CMP) research project
Founder of Afara Websystems, acquired by Sun in 2002

Christopher Ré (Co-founder)

Associate Professor of Computer Science at Stanford University
MacArthur Genius Award recipient
Affiliated with Stanford AI Lab and Statistical Machine Learning Group

Proprietary Technology: RDU Architecture

What is RDU (Reconfigurable Dataflow Unit)?

At the core of SambaNova’s technological advantage is the proprietary RDU chip. Unlike traditional GPUs or CPUs, it employs a dataflow processing approach specifically designed for AI inference.

3-Tier Memory Architecture

The SN40L RDU’s defining feature is its revolutionary 3-tier memory system:

On-chip SRAM: 520MiB of high-speed accessible on-chip memory
Co-packaged HBM: 64GiB of high-bandwidth memory
Off-package DDR DRAM: Up to 1.5TiB of DDR memory (using pluggable DIMMs)

This architecture enables model loading from DDR to HBM at speeds exceeding 1TB/sec, significantly eliminating memory bottlenecks.

Key Components

PCU (Pattern Compute Units)

Execute innermost parallel operations in applications
Configured as multi-stage reconfigurable SIMD pipelines
Leverage both loop-level and pipeline parallelism

PMU (Pattern Memory Units)

Highly specialized scratchpads
Provide on-chip memory capacity and perform specialized functions
Minimize data movement and reduce latency

Exceptional Inference Speed

Measured Performance

SambaNova achieves industry-leading inference speeds for large language models:

DeepSeek-R1 671B: 198 tokens/sec (measured on SambaNova Cloud)
Llama 3.1 405B: 129 tokens/sec/user (world record)
General LLMs: Up to 450 tokens/sec (8-chip configuration)
First token time: Approximately 0.2 seconds

Hardware Efficiency

Compared to traditional GPU-based systems, SambaNova achieves remarkable efficiency:

Reduces hardware requirements for DeepSeek-R1 671B from 40 racks (320 latest GPUs) to 1 rack (16 RDUs)
Each SN40L RDU socket delivers 638 BF16 TFLOPS peak performance
Low power consumption with high token generation efficiency (tokens per kWh)

Product Lineup

Supporting Both Cloud and On-Premises

SambaNova offers a product suite addressing various deployment models:

SambaCloud: Cloud-based AI inference platform
SambaStack: Comprehensive AI computing platform
SambaManaged: Turnkey AI solution for data centers
SambaRack: High-efficiency AI inference hardware

Corporate Valuation and Funding

Status as a Unicorn

In April 2021, SambaNova raised $676 million in Series D funding, reaching a valuation of $5.1 billion. Investors in this round included:

Lead investor: SoftBank Vision Fund 2
Participants: Temasek, GIC, BlackRock, Intel Capital, GV, Walden International, WRVI

The company has raised a total of $1.13 billion across 4 rounds, becoming “the world’s best-funded AI systems and services platform startup.”

Mission and Vision

Democratizing AI

SambaNova’s mission is to “democratize access to enterprise AI.” The company’s goals include:

Enabling companies of all sizes to harness the power of AI
Bridging the gap between cutting-edge technology and accessibility
Running AI applications efficiently from data centers to the cloud

Customer Base

Key customers include:

National laboratories
Government agencies
Enterprise companies
Developer community

Value Created Through Innovation

SambaNova’s technology addresses the following AI inference challenges:

Memory wall problem: Solved through 3-tier memory architecture
Power consumption: Efficient power utilization via dataflow processing
Scalability: Flexible expansion through modular design
Deployment speed: AI infrastructure can be built in months

SambaNova Systems has developed proprietary technology for accelerating and optimizing AI inference, built on research from Stanford University. The RDU chip and 3-tier memory architecture achieve significant performance improvements and power reduction compared to traditional GPU-based systems. As a unicorn valued at $5.1 billion, the company continues contributing to the proliferation and development of enterprise AI.