kmail.at
contact

elsewhere

EnglishEN
← learning

technologies · difficulty ◆◆

Pipecat — build real-time voice & multimodal agents

Scaffold a talking agent, wire it to any STT/LLM/TTS, and ship a real project.

You can stand up a talking, listening AI bot in under a minute — and wire it to any STT, LLM, or TTS service you already pay for.

AI-generated draft · reviewed and editedThis text was generated with AI as a first draft, then reviewed and edited by the author, who holds editorial responsibility for it.

2026-08-23 · 10 min read

$ uv tool install "pipecat-ai[cli]"
100+
orchestrated AI services
20+
speech-to-text services
25+
LLM providers
20+
text-to-speech services
6
client SDKs
BSD-2
license

What it is

Pipecat is an open-source Python framework (BSD-2 license) for building real-time and multimodal AI agents. It is not a model — it is an orchestration framework that connects speech-to-text, an LLM, and text-to-speech into a single agent. You assemble voice assistants, multi-agent systems with handoffs, and multimodal interfaces that see, hear, and speak. It works with any AI provider and any hosting environment; you pay only for the services you choose.

The client/server shape

A typical Pipecat application has a client and a server. The client connects users via a browser, mobile app, or phone. The server runs a Pipecat pipeline that processes audio, runs LLMs, and generates speech in real time. Your hosting — Pipecat Cloud or self-hosted — manages deployment and scales instances to handle concurrent sessions.

The pipeline idea

Every Pipecat agent is a Pipeline of modular processors. A transport connects (WebRTC or WebSocket), a speech-to-text processor transcribes, an LLM thinks, and a text-to-speech processor voices the answer. You control the data flow between them. Because every pipeline is an agent, you can nest them — hand off to specialists, fan out in parallel, or run sidecars.

Why it matters

Real-time voice is awkward to build from scratch: WebRTC plumbing, speech detection, low-latency audio, and provider integration. Pipecat packages that. It orchestrates 100+ AI services with ultra-low latency, and ships Pipecat Flows (structured conversations with state) and Pipecat Cloud (managed hosting) on top of the core runtime.

Example

$ pipecat init quickstart
Project name: pipecat-quickstart
Bot type: Web/Mobile
Transport: SmallWebRTC, Daily
Pipeline architecture: Cascade (STT -> LLM -> TTS)
Speech-to-Text: Deepgram
Language model: OpenAI
Text-to-Speech: Cartesia
Deploy to Pipecat Cloud: Yes

Project created successfully!
   pipecat-quickstart

Next steps:

  Server setup:
  - Go to server: cd server
  - Install dependencies: uv sync
  - Create .env file: cp .env.example .env
  - Edit .env and add your API keys
  - Run your bot: uv run bot.py

Real output from pipecat init quickstart (CLI 1.7.0). The questions are interactive, and the scaffold writes AGENTS.md and CLAUDE.md at the project root while bot.py, .env.example, Dockerfile and pcc-deploy.toml all land inside server/ - which is why every later command runs from there.

$ bot.py — wire up the pipeline
# Create the AI services (STT → LLM → TTS)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    settings=CartesiaTTSService.Settings(
        voice=os.getenv("CARTESIA_VOICE_ID",
                         "71a7ad14-091c-4e8e-a314-022ece01c121"),
    ),
)
llm = OpenAIResponsesLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    settings=OpenAIResponsesLLMService.Settings(
        model=os.getenv("OPENAI_MODEL", "gpt-4.1"),
        system_instruction="You are a helpful assistant in a voice conversation."
    ),
)

# A conversation context, split into user + assistant halves
context = LLMContext()
user_agg, assistant_agg = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

# Compose the pipeline — this is the whole agent
pipeline = Pipeline([
    transport.input(),      # Receive audio from the browser
    stt,                    # Speech-to-text (Deepgram)
    user_agg,               # Add the user message to context
    llm,                    # Language model (OpenAI)
    tts,                    # Text-to-speech (Cartesia)
    transport.output(),     # Send audio back to the browser
    assistant_agg,          # Add the bot response to context
])

Pipelines read the same way as the official quickstart bot.py: Deepgram STT, an LLM, Cartesia TTS with the same default voice id, and the same aggregator pair. The quickstart itself instantiates OpenAILLMService; OpenAIResponsesLLMService is the Responses API variant (pipecat.services.openai.responses.llm) and takes api_key plus a Settings block the same way. The pipeline is the agent.

$ Tuning voice activity (VAD)
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams

vad = SileroVADAnalyzer(
    params=VADParams(
        confidence=0.7,   # minimum confidence for voice detection
        start_secs=0.2,   # wait before confirming speech start
        stop_secs=0.2,    # wait before confirming speech stop
        min_volume=0.6,   # minimum volume threshold
    )
)
user_agg, assistant_agg = LLMContextAggregatorPair(
    context, user_params=LLMUserAggregatorParams(vad_analyzer=vad)
)

VAD decides when the user started and stopped talking. The four values shown are the library defaults (0.7 confidence, 0.2s start and stop, 0.6 minimum volume), so tune them against your own mic and noise floor rather than copying them expecting a change.

Common flags

pipecat init <name>
Scaffold a runnable project in one command
uv tool install "pipecat-ai[cli]"
Install the framework + CLI
uv add "pipecat-ai[daily,deepgram,openai]"
Add transports and service extras
uv sync / uv run bot.py
Install deps and run the bot
cp .env.example .env
Copy the environment template before filling keys

History

Built and maintained by Daily

Pipecat is built and maintained by Daily, the WebRTC infrastructure company, and open-sourced under the pipecat-ai organization. The framework is free under the BSD-2 license and works with any AI provider and any hosting environment — you pay only for the services you choose. It grew out of the practical need to turn a WebRTC call into a full voice agent without hand-rolling the audio plumbing.

The ecosystem around the core

Pipecat is more than the runtime. Pipecat Client SDKs cover JavaScript, React, React Native, iOS, Android, and C++. Pipecat Flows builds structured conversations with defined paths and state management. Pipecat Cloud is managed hosting with built-in infrastructure. The community lives on Discord and GitHub, and the docs are hosted on Mintlify at docs.pipecat.ai.

Under the hood

Frames are the unit of data flow

Everything flowing through a pipeline is a Frame. System frames (InputAudioRawFrame, UserStartedSpeakingFrame, InterruptionFrame) are queued with high priority. Data frames (OutputAudioRawFrame, TextFrame, TranscriptionFrame, LLMTextFrame, TTSSpeakFrame) and control frames (StartFrame, EndFrame, CancelFrame) are queued and processed in order. This frame-based model is what makes real-time control precise.

Processors are composable building blocks

A FrameProcessor is the smallest unit. You subclass it, handle specific frame types, and push frames onward. Processors can fork — the same audio can feed STT and a recording buffer — and pipelines can branch with ParallelPipeline, e.g. routing the same answer through an English TTS and a Spanish TTS.

The conversation turn

An LLMContext holds the conversation. Aggregators split it into user and assistant halves. VAD (voice-activity detection) decides when the user has finished speaking, and pipecat ships additional turn detectors beside the Silero VAD - a smart-turn analyzer and a Krisp VIVA turn detector - which exist for exactly the interruption handling that raw VAD alone handles poorly. You stream the context, run the LLM, and pipe the reply into TTS, all while audio flows over WebRTC or WebSocket at ultra-low latency.

Fun facts

Pros & cons

pros

  • + Voice-first: STT, TTS, and conversation logic are built in
  • + Orchestrates 100+ AI services with ultra-low latency
  • + Pluggable: WebRTC or WebSocket transports, any provider
  • + Multi-agent ready: pipelines nest, hand off, fan out
  • + Free BSD-2 license; works with any hosting environment

cons

  • − More moving parts than a text-only agent pipeline
  • − Provider extras mean you manage several service keys
  • − Streaming voice debugging is harder than batch LLM work

Takeaways

  1. 1Fastest start: install the CLI, run pipecat init quickstart, cd server, cp .env.example .env, uv sync, uv run bot.py
  2. 2Think in pipelines: Transport, STT, LLM, TTS as your four core processors
  3. 3Tune VAD (SileroVADAnalyzer) so the agent knows when to stop and start listening
  4. 4Add services as extras: uv add "pipecat-ai[daily,deepgram,openai]"
  5. 5Compose multi-agent behavior with ParallelPipeline and FrameProcessors
  6. 6Ship with pipecat cloud deploy, or self-host on your own infrastructure

Related commands

← all learning