Noesis

Stream protocol

Connect a live call over the WebSocket: audio formats, the messages you send, the events you receive, limits, credits and close codes.

One WebSocket per call. Stream the callee's audio for the whole call, and receive the first answer and every change.

Example
wss://api.noesis.detectif.ai/v1/detect/stream

Open the socket as soon as the call is answered and stream the callee's audio in real time. Keep it open for the whole call: a transfer to a person can happen minutes in. Authenticate with the Authorization header on the upgrade request, or with a stream token in ?token=; see Authentication.

Query parameters

Parameter Values Default
encoding mulaw, alaw or pcm_s16le mulaw
sample_rate 8000, 16000, 22050, 24000, 44100 or 48000 8000
channels 1: the callee only. 2: the callee and your agent on separate channels 1
remote_channel The callee's channel when channels=2: 0 or 1 0
language Optional hint, en or hi. Leave it out for automatic (English, Hindi, Hinglish) automatic
call_id Your reference for the call, up to 200 characters, returned in every event none
transcripts true to also receive what the callee says, for debugging false
record false to not record this call. See Data and recording true
token A stream token, for clients that cannot set the Authorization header none

callback_url and other webhook settings are not available; a socket that sets them is refused with unsupported_parameter. Answers arrive on the socket, and later through sessions.

Audio

Audio Accepted
Encoding G.711 μ-law (mulaw), G.711 A-law (alaw), or 16-bit linear PCM, little-endian (pcm_s16le)
Sample rate 8, 16, 22.05, 24, 44.1 or 48 kHz
Channels Mono (the callee only), or stereo with your agent on the other channel
Framing Binary WebSocket frames, or Twilio and Plivo media-stream JSON

Compressed audio such as Opus, MP3, AAC or G.729 is not accepted: decode it to PCM first. Most telephony platforms already give you μ-law or PCM. Set encoding and sample_rate to what you send; the audio is not converted.

Messages you send

Message When
Binary audio The callee's audio, 20 to 100 ms per frame, in real time from the moment the call is answered
{"event": "media", ...} Twilio or Plivo media-stream JSON, unchanged. The inbound track is the callee; an outbound track is your agent. A start message sets call_id from the call's ID; stop ends the call
{"type": "agent.speech", "state": "started"} and {"type": "agent.speech", "state": "ended"} Optional: when your agent starts and stops talking, if its audio is not on a second channel or track. It helps Noesis recognise a person sooner
{"type": "end"} On hang-up. You receive the summary and the socket closes
{"type": "ping"} Keep-alive, answered with {"type": "pong"}

Events you receive

JSON
// once, when the socket opens
{"type": "ready", "session_id": "9f2c4e7a-...", "call_id": "your-id", "t": 0.0,
 "audio": {"encoding": "mulaw", "sample_rate": 8000, "channels": 1}}

// the first answer, then one event per change
{"type": "classification", "seq": 2, "t": 13.0,
 "label": "hold", "category": "machine", "confidence": 0.8,
 "previous_label": "ivr", "change": "transition",
 "session_id": "9f2c4e7a-...", "call_id": "your-id"}

// after {"type": "end"} (or a Twilio or Plivo "stop")
{"type": "summary", "label": "human", "category": "human", "timeline": [ ... ]}
{"type": "usage", "billed_seconds": 27}
Field Meaning
session_id Noesis's ID for the call. Keep it to look the call up later in sessions
call_id Your reference, from the URL or the Twilio or Plivo start message
seq Increases with every event. Act on the highest you have seen
t Seconds since the socket opened
label, category The answer. See Answers
confidence 0 to 1
change initial: the first answer. transition: person and machine swapped, or a transfer to hold. refinement: a more specific label, such as machine to voicemail
billed_seconds In the usage event at the end: the seconds of call audio billed for this call
Other fields Events may carry more fields than these. Ignore the ones you do not use

With transcripts=true you also receive {"type": "transcript", ...} events with what the callee said.

A typical transfer

Time Label On the line
3.2 s ivr "For credit cards, press one."
13.0 s hold "Please hold while we connect you to a customer associate."
23.4 s human "Hi, this is Priya. How can I help you?"

Ending a call

Send {"type": "end"} when the call ends (a Twilio or Plivo stop does the same). You receive the summary and a usage event, then the socket closes with code 1000.

If you close the socket without end, the call still ends and its result is kept in the session, but the summary is not sent to you.

Limits

Limit Value
Binary audio frame Up to 192,000 bytes; 20 to 100 ms of audio is best
JSON message Up to 384,000 characters
One call Up to 2 hours
Silence on the socket The socket closes after 15 minutes without any message
Calls at the same time Set by your plan. A call over the limit is refused with 429 and close code 4429

A frame or message that is too large gets an error event with status 413; the socket stays open.

Credits

Live calls use credits for the seconds of audio Noesis receives. A call that sends less than a second of audio is not billed. Your balance is on the dashboard Overview page.

If your balance runs out during a call, you receive {"type": "error", "code": "insufficient_credits", ...} and the socket closes with code 4402. The audio already received is billed. Carry on with the call without detection, and add credits before the next one.

Close codes

Close code HTTP status in the error event Meaning What to do
1000 The call ended normally after end, or the socket was silent for 15 minutes, or the call passed 2 hours (408) Open one socket per call
1003 400 A parameter is invalid or not available, such as callback_url; the message names it Fix the URL
1013 503 Busy Carry on with the call without detection
4401 401 The key or stream token is missing, invalid, expired or already used Check the key, or create a new token
4402 402 The credit balance ran out (insufficient_credits) Add credits in the dashboard
4403 403 The key is not allowed to use Noesis, or the email address is not verified Use a key with noesis:detect
4429 429 Too many calls at once, or new calls too fast Wait for a call to end

Before closing, Noesis sends an error event with the details. See Errors.