technologies · difficulty ◆◆
Pipecat — build real-time voice & multimodal agents
Scaffold a talking agent, wire it to any STT/LLM/TTS, and ship a real project.
You can stand up a talking, listening AI bot in under a minute — and wire it to any STT, LLM, or TTS service you already pay for.
AI-generated draft · reviewed and editedThis text was generated with AI as a first draft, then reviewed and edited by the author, who holds editorial responsibility for it.$ uv tool install "pipecat-ai[cli]"- 100+
- orchestrated AI services
- 20+
- speech-to-text services
- 25+
- LLM providers
- 20+
- text-to-speech services
- 6
- client SDKs
- BSD-2
- license
What it is
Pipecat is an open-source Python framework (BSD-2 license) for building real-time and multimodal AI agents. It is not a model — it is an orchestration framework that connects speech-to-text, an LLM, and text-to-speech into a single agent. You assemble voice assistants, multi-agent systems with handoffs, and multimodal interfaces that see, hear, and speak. It works with any AI provider and any hosting environment; you pay only for the services you choose.
The client/server shape
A typical Pipecat application has a client and a server. The client connects users via a browser, mobile app, or phone. The server runs a Pipecat pipeline that processes audio, runs LLMs, and generates speech in real time. Your hosting — Pipecat Cloud or self-hosted — manages deployment and scales instances to handle concurrent sessions.
The pipeline idea
Every Pipecat agent is a Pipeline of modular processors. A transport connects (WebRTC or WebSocket), a speech-to-text processor transcribes, an LLM thinks, and a text-to-speech processor voices the answer. You control the data flow between them. Because every pipeline is an agent, you can nest them — hand off to specialists, fan out in parallel, or run sidecars.
Why it matters
Real-time voice is awkward to build from scratch: WebRTC plumbing, speech detection, low-latency audio, and provider integration. Pipecat packages that. It orchestrates 100+ AI services with ultra-low latency, and ships Pipecat Flows (structured conversations with state) and Pipecat Cloud (managed hosting) on top of the core runtime.
Example
$ pipecat init quickstartProject name: pipecat-quickstart
Bot type: Web/Mobile
Transport: SmallWebRTC, Daily
Pipeline architecture: Cascade (STT -> LLM -> TTS)
Speech-to-Text: Deepgram
Language model: OpenAI
Text-to-Speech: Cartesia
Deploy to Pipecat Cloud: Yes
Project created successfully!
pipecat-quickstart
Next steps:
Server setup:
- Go to server: cd server
- Install dependencies: uv sync
- Create .env file: cp .env.example .env
- Edit .env and add your API keys
- Run your bot: uv run bot.pyReal output from pipecat init quickstart (CLI 1.7.0). The questions are interactive, and the scaffold writes AGENTS.md and CLAUDE.md at the project root while bot.py, .env.example, Dockerfile and pcc-deploy.toml all land inside server/ - which is why every later command runs from there.
$ bot.py — wire up the pipeline# Create the AI services (STT → LLM → TTS)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
settings=CartesiaTTSService.Settings(
voice=os.getenv("CARTESIA_VOICE_ID",
"71a7ad14-091c-4e8e-a314-022ece01c121"),
),
)
llm = OpenAIResponsesLLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAIResponsesLLMService.Settings(
model=os.getenv("OPENAI_MODEL", "gpt-4.1"),
system_instruction="You are a helpful assistant in a voice conversation."
),
)
# A conversation context, split into user + assistant halves
context = LLMContext()
user_agg, assistant_agg = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
# Compose the pipeline — this is the whole agent
pipeline = Pipeline([
transport.input(), # Receive audio from the browser
stt, # Speech-to-text (Deepgram)
user_agg, # Add the user message to context
llm, # Language model (OpenAI)
tts, # Text-to-speech (Cartesia)
transport.output(), # Send audio back to the browser
assistant_agg, # Add the bot response to context
])Pipelines read the same way as the official quickstart bot.py: Deepgram STT, an LLM, Cartesia TTS with the same default voice id, and the same aggregator pair. The quickstart itself instantiates OpenAILLMService; OpenAIResponsesLLMService is the Responses API variant (pipecat.services.openai.responses.llm) and takes api_key plus a Settings block the same way. The pipeline is the agent.
$ Tuning voice activity (VAD)from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
vad = SileroVADAnalyzer(
params=VADParams(
confidence=0.7, # minimum confidence for voice detection
start_secs=0.2, # wait before confirming speech start
stop_secs=0.2, # wait before confirming speech stop
min_volume=0.6, # minimum volume threshold
)
)
user_agg, assistant_agg = LLMContextAggregatorPair(
context, user_params=LLMUserAggregatorParams(vad_analyzer=vad)
)VAD decides when the user started and stopped talking. The four values shown are the library defaults (0.7 confidence, 0.2s start and stop, 0.6 minimum volume), so tune them against your own mic and noise floor rather than copying them expecting a change.
Common flags
- pipecat init <name>
- Scaffold a runnable project in one command
- uv tool install "pipecat-ai[cli]"
- Install the framework + CLI
- uv add "pipecat-ai[daily,deepgram,openai]"
- Add transports and service extras
- uv sync / uv run bot.py
- Install deps and run the bot
- cp .env.example .env
- Copy the environment template before filling keys
History
Built and maintained by Daily
Pipecat is built and maintained by Daily, the WebRTC infrastructure company, and open-sourced under the pipecat-ai organization. The framework is free under the BSD-2 license and works with any AI provider and any hosting environment — you pay only for the services you choose. It grew out of the practical need to turn a WebRTC call into a full voice agent without hand-rolling the audio plumbing.
The ecosystem around the core
Pipecat is more than the runtime. Pipecat Client SDKs cover JavaScript, React, React Native, iOS, Android, and C++. Pipecat Flows builds structured conversations with defined paths and state management. Pipecat Cloud is managed hosting with built-in infrastructure. The community lives on Discord and GitHub, and the docs are hosted on Mintlify at docs.pipecat.ai.
Under the hood
Frames are the unit of data flow
Everything flowing through a pipeline is a Frame. System frames (InputAudioRawFrame, UserStartedSpeakingFrame, InterruptionFrame) are queued with high priority. Data frames (OutputAudioRawFrame, TextFrame, TranscriptionFrame, LLMTextFrame, TTSSpeakFrame) and control frames (StartFrame, EndFrame, CancelFrame) are queued and processed in order. This frame-based model is what makes real-time control precise.
Processors are composable building blocks
A FrameProcessor is the smallest unit. You subclass it, handle specific frame types, and push frames onward. Processors can fork — the same audio can feed STT and a recording buffer — and pipelines can branch with ParallelPipeline, e.g. routing the same answer through an English TTS and a Spanish TTS.
The conversation turn
An LLMContext holds the conversation. Aggregators split it into user and assistant halves. VAD (voice-activity detection) decides when the user has finished speaking, and pipecat ships additional turn detectors beside the Silero VAD - a smart-turn analyzer and a Krisp VIVA turn detector - which exist for exactly the interruption handling that raw VAD alone handles poorly. You stream the context, run the LLM, and pipe the reply into TTS, all while audio flows over WebRTC or WebSocket at ultra-low latency.
Fun facts
Pros & cons
pros
- + Voice-first: STT, TTS, and conversation logic are built in
- + Orchestrates 100+ AI services with ultra-low latency
- + Pluggable: WebRTC or WebSocket transports, any provider
- + Multi-agent ready: pipelines nest, hand off, fan out
- + Free BSD-2 license; works with any hosting environment
cons
- − More moving parts than a text-only agent pipeline
- − Provider extras mean you manage several service keys
- − Streaming voice debugging is harder than batch LLM work
Takeaways
- 1Fastest start: install the CLI, run pipecat init quickstart, cd server, cp .env.example .env, uv sync, uv run bot.py
- 2Think in pipelines: Transport, STT, LLM, TTS as your four core processors
- 3Tune VAD (SileroVADAnalyzer) so the agent knows when to stop and start listening
- 4Add services as extras: uv add "pipecat-ai[daily,deepgram,openai]"
- 5Compose multi-agent behavior with ParallelPipeline and FrameProcessors
- 6Ship with pipecat cloud deploy, or self-host on your own infrastructure
