← back to console

Integrate Voice Core

Voice Core runs in two modes from one runtime: the standalone console on this deployment, and an embedded mode consumed by any front-end (such as the Animavio web app) over the public v1 API. Provider credentials never leave the server.

1 — HTTP surface

GET/api/public/v1/agentsAgent catalogue (id, voice profile, endpointing, tools)
POST/api/public/v1/sttmultipart audio + language → transcript
GET/api/public/v1/modelsChat models accepted as `modelOverride`
GET/api/public/v1/toolsTool definitions (ElevenLabs shape) + derived JSON Schema
POST/api/public/v1/turnJSON turn request → SSE (safety, tier, deltas, usage)
POST/api/public/v1/ttsJSON text → SSE base64 PCM @ 24 kHz

CORS is enforced from VOICE_CORE_ALLOWED_ORIGINS and, when VOICE_CORE_API_KEY is set, every call must carry X-Api-Key. With neither configured the API stays open for local testing.

2 — Browser SDK (recommended)

Copy src/sdk/animavio-voice.ts into the host app. It is a single dependency-free file: mic capture, VAD endpointing, barge-in, scheduled PCM playback, turn streaming and cost metering.

import { VoiceSession } from "./animavio-voice";

const session = new VoiceSession({
  baseUrl: "https://voice-core.example.com",
  apiKey: import.meta.env.VITE_VOICE_CORE_KEY,
  agentId: "elena",
  language: "de",
  // Grounding resolves before the agent greets (spec C7)
  resolveContext: async () => ({
    petDigest: await loadPetDigest(),
  }),
  onSnapshot: (s) => render(s),      // status, transcript, urgency, cost
  // ElevenLabs-style client tools: you implement the body, the returned value
  // becomes the tool result the console/host app shows.
  clientTools: {
    Provide_AnimalRecommendation: (args) => renderPlan(args),
    Provide_AnimalNutritionPlan: (args) => renderPlan(args),
    Escalate_ToVet: (args) => showClinicRouting(args),
  },
  // Or observe every tool call (server + client) without handling it
  onToolCall: (call) => log(call.name, call.args),
  onError: (msg) => toast(msg),
  // Optional: pin one model instead of tier routing (GET /v1/models)
  modelOverride: "openai/gpt-5.5",
  // Optional: configure which model serves each tier (defaults: flash-lite for
  // T0/T1, flash for T2/T4, Claude Sonnet 4.6 for T3 clinical turns)
  tierModels: { T1: "google/gemini-3.1-flash-lite", T3: "anthropic/claude-sonnet-4-6" },
  // Optional: cheaper TTS engine (TTS is the costliest leg of a turn)
  // "elevenlabs/flash-v2.5" (default) | "openai/gpt-4o-mini-tts"
  // "google/gemini-2.5-flash-tts" | "google/gemini-2.5-flash-lite-preview-tts"
  voiceEngine: "elevenlabs/flash-v2.5",
  sttEngine: "openai/gpt-4o-mini-transcribe",
});

await session.start();               // mic + greeting
session.sendUserMessage("Hallo");    // text turn
session.sendContextualUpdate({       // silent host-app context
  kind: "phase_transition",
  text: "User opened the vaccination screen.",
});
session.setPetDigest("Name: Balu · dog · 4 y · 31 kg"); // live grounding swap
session.setModelOverride("google/gemini-3.1-flash-lite"); // live A/B swap
session.setVoiceEngine("google/gemini-2.5-flash-tts");    // live voice/cost swap
session.setSttEngine("openai/gpt-4o-mini-transcribe");   // live recogniser swap
session.interrupt();                 // manual barge-in
await session.stop();

3 — Tools

Tools are declarative artefacts in the ElevenLabs agent-tool shape (src/config/tools.ts). The runtime derives a JSON Schema from parameters[], exposes it to the model as a function, dispatches the call and streams it to the client as tool_call / tool_result SSE events. Server tools (Get_AnimalContext) resolve in-runtime; client tools are forwarded to the host app and acknowledged so the agent can speak its post-tool summary in the same turn.

data: {"type":"tool_call","name":"Provide_CoachingPlan","tool_type":"client",
        "disable_interruptions":true,
        "args":{"animal":"dog","urgency":"Routine","medicalReferral":false,
                "confidence":0.7,"recommendedActions":["[Management] ...","[Training] ..."]}}
data: {"type":"tool_result","name":"Provide_CoachingPlan","result":{"delivered":true}}

Fields documented as “JSON array of …” are parsed into real arrays before they reach the host app. Tools with disable_interruptions hold the floor until the post-tool speech finishes, and End_Session closes the session after it speaks.

4 — React binding

A thin hook wraps the same session object, so the console and any embedding React app share identical behaviour.

const voice = useVoiceAgent({
  agentId: "hannah",
  language: "en",
  baseUrl: "https://voice-core.example.com",
  apiKey: VOICE_CORE_KEY,
});

voice.status; voice.transcript; voice.urgency; voice.costMicros;

5 — Raw API (non-JS clients)

curl -N https://voice-core.example.com/api/public/v1/turn \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $VOICE_CORE_API_KEY" \
  -d '{
    "agentId": "elena",
    "language": "de",
    "tierModels": { "T3": "anthropic/claude-sonnet-4-6" },
    "voiceEngine": "google/gemini-2.5-flash-tts",
    "sttEngine": "openai/gpt-4o-mini-transcribe",
    "messages": [{ "role": "user", "content": "Mein Hund frisst nicht." }]
  }'

# SSE events: turn_start → text_delta* → turn_done
# turn_done.usage = { tier, model_used, llm_ms, out_chars, cost_micros }

6 — Integration notes