AI Voice Agents

An AI voice agent joins an RTC room as a participant. It listens to the people in the room, thinks with a large language model, and answers out loud. It can call your backend through tools, and you can change its instructions, make it say something, or interrupt it while it runs.

Where this fits: your backend creates the room and mints tokens as usual (Rooms & Participants). Adding an agent is one call from your backend. Your app sees the agent as one more participant whose identity starts with agent-.

How it works

• Livqeno adds a participant named agent-<id> to the room. It subscribes to audio and publishes audio. It never publishes video.
• Speech-to-text (Deepgram or OpenAI) turns what people say into text. The turn ends when the speech-to-text provider detects the end of a phrase and there is turnDetection.silenceMs of silence.
• The LLM (Anthropic claude-sonnet-5-5 by default, or OpenAI) writes the reply. Text-to-speech (Cartesia, ElevenLabs or OpenAI) speaks it.
• With interruptible on, someone talking over the agent stops its audio (barge-in).
• The agent doesn't trigger room.participant_* webhooks and isn't counted as an RTC participant. It's billed as agent minutes.
• A room can have up to 2 active agents. The operator can change this with AI_MAX_AGENTS_PER_ROOM.

Add an agent

const agent = await livqeno.agents.create(roomId, {
  name: 'Support bot',
  instructions: 'You are a friendly support agent for Acme. Keep answers short.',
  greeting: 'Hi! How can I help?',
  llm: { provider: 'anthropic', model: 'claude-sonnet-5-5', temperature: 0.6, maxTokens: 400 },
  stt: { provider: 'deepgram', language: 'en' },
  tts: { provider: 'cartesia' },
  turnDetection: { silenceMs: 600 },
  interruptible: true,
  maxDurationSeconds: 3600,
});

agent = livqeno.agents.create(room_id, {
    "name": "Support bot",
    "instructions": "You are a friendly support agent for Acme. Keep answers short.",
    "greeting": "Hi! How can I help?",
    "tts": {"provider": "cartesia"},
})

curl -X POST "$LIVQENO_API_URL/v1/rooms/$ROOM_ID/agents" \
  -H "Authorization: Bearer $LIVQENO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Support bot", "instructions": "You are a friendly support agent.", "greeting": "Hi! How can I help?"}'

The response is 201 with the agent in STARTING. It includes the agent's id (agt_…), roomId, identity (agent-agt_…), status, its configuration, and usage so far.

A provider with no key on this deployment and no key in providerKeys fails with 422 AI_PROVIDER_NOT_CONFIGURED. A room that already has the maximum number of agents fails with 409 AGENT_LIMIT_EXCEEDED.

Show the agent's state

Every client in the room receives the agent's state as it changes:

{ "type": "agent.state", "agentId": "agt_7Lw2", "participantId": "agent-agt_7Lw2", "state": "thinking" }

state is listening, thinking, speaking or ended. What the agent says also arrives as transcription.segment messages with participantId set to the agent's identity (see Live Transcription).

In the client SDKs this is the agentStateChanged(participant, state) room event. participant.isAgent is true for any identity starting with agent-, and participant.agentState holds the latest state (unset until the first one). The event fires only when the state changes. A state that arrives before the agent's participantJoined is held and delivered just after it.

Sometimes you only have an identity string, not a full Participant — in a webhook payload, say, or a chat message's senderId. In that case, use isAgentIdentity() from @livqeno/rtc instead — it does the same check as participant.isAgent above. (There's also isTranscriberIdentity() for the transcriber; see Live Transcription.)

import { isAgentIdentity, AGENT_IDENTITY_PREFIX } from '@livqeno/rtc';
 
isAgentIdentity('agent-agt_7Lw2'); // true
AGENT_IDENTITY_PREFIX; // 'agent-'

room.on('agentStateChanged', (participant, state) => {
  setAgentStatus(participant.identity, state); // 'listening' | 'thinking' | 'speaking' | 'ended'
});
 
const agent = room.remoteParticipants.find((p) => p.isAgent);
agent?.on('agentStateChanged', (state) => updateRing(state));

import { AgentVisualizer, useAgentState, useRemoteParticipants } from '@livqeno/react';
import '@livqeno/react/video/styles.css';
 
function AgentPanel() {
  const agent = useRemoteParticipants().find((p) => p.isAgent);
  const state = useAgentState(agent); // undefined until the first update
  if (!agent) return null;
  return (
    <div className="livqeno-video">
      <AgentVisualizer participant={agent} />
      {state === 'thinking' && <p>One moment…</p>}
    </div>
  );
}

import { useAgentState, useRemoteParticipants } from '@livqeno/react-native';
 
function AgentStatus({ room }) {
  const agent = useRemoteParticipants(room).find((p) => p.isAgent);
  const state = useAgentState(agent);
  return agent ? <Text>{agent.identity}: {state ?? 'connecting'}</Text> : null;
}

room.agentStateChanges.listen((update) {
  // update.participant.isAgent == true
  setAgentStatus(update.participant.identity, update.state); // LivqenoAgentState.speaking, …
});
 
final agent = room.participants.where((p) => p.isAgent).firstOrNull;
agent?.agentState; // LivqenoAgentState?, null until the first update

for await event in room.events() {
    if case let .agentStateChanged(participant, state) = event {
        setAgentStatus(participant.identity, state) // .listening, .thinking, .speaking, .ended
    }
}
 
let agent = room.remoteParticipants.values.first { $0.isAgent }
agent?.agentState // AgentState?, nil until the first update

room.events.filterIsInstance<RoomEvent.AgentStateChanged>().collect { e ->
    setAgentStatus(e.participant.identity, e.state) // AgentState.LISTENING, THINKING, SPEAKING, ENDED
}
 
val agent = room.remoteParticipants.firstOrNull { it.isAgent }
agent?.agentStateFlow // StateFlow<AgentState?>, null until the first update

Steer it live

await livqeno.agents.update(agent.id, { instructions: 'Answer in Spanish from now on.' });
await livqeno.agents.say(agent.id, 'One moment while I check that.');
await livqeno.agents.interrupt(agent.id);
await livqeno.agents.delete(agent.id);

Changing an agent that has already ended fails with 409 AGENT_NOT_ACTIVE.

An agent also leaves by itself when the room is closed, after 10 minutes with nobody publishing audio, or at maxDurationSeconds.

Tools

Give the agent tools to look things up or take actions in your system:

await livqeno.agents.create(roomId, {
  name: 'Order bot',
  instructions: 'Help customers track their orders. Use lookup_order for any order number.',
  tools: [
    {
      name: 'lookup_order',
      description: 'Look up an order by its number',
      inputSchema: { type: 'object', properties: { orderId: { type: 'string' } }, required: ['orderId'] },
      webhookUrl: 'https://api.example.com/livqeno/tools',
    },
  ],
});

When the model calls a tool, Livqeno sends a POST to its webhookUrl:

{ "agentId": "agt_7Lw2", "roomId": "5b1f0c7e-…", "tool": "lookup_order", "input": { "orderId": "A-1001" }, "callId": "call_01" }

• The request is signed exactly like a webhook, in the swalvo-signature header (t=…,v1=…), with the signing secret of your project's oldest active webhook endpoint in that environment. Verify it the same way.
• Reply within 5 seconds. The JSON response body is the tool result the model sees.
• On a timeout or an error, the model gets { "error": "tool_unavailable" } and carries on.

Webhooks

reason is requested, room_ended, no_audio_publishers, max_duration or worker_ended. See Events for every webhook.

Usage

An agent is billed per minute that it's in the room, whoever's provider keys it uses. It counts as agent minutes in your usage.