Noesis / Integrations

Twilio

Stream a Twilio call to Noesis with a Media Stream and a stream token, and receive the answers during the call through a relay.

Twilio Media Streams send a call's audio to a WebSocket. Noesis reads Twilio's messages as they are, and the defaults (μ-law, 8 kHz) match.

Twilio cannot send an Authorization header, so you pass a stream token as a <Parameter> instead. Twilio does not pass messages from the stream back to you, so to act on the answers during the call, stream through a relay on your server (step 3).

1. Stream the call to Noesis

Add a <Stream> inside <Start>, so the call carries on while its audio is streamed. track="both_tracks" lets Noesis hear when your agent speaks, and the call ID defaults to the CallSid.

XML
<Response>
  <Start>
    <Stream url="wss://api.noesis.detectif.ai/v1/detect/stream" track="both_tracks">
      <Parameter name="token" value="ntk_..."/>
    </Stream>
  </Start>
  <!-- your existing call flow continues here -->
</Response>

To not record the call, add <Parameter name="record" value="false"/>. See Data and recording.

2. Create a token for each call

A token opens one call and expires after 5 minutes, so create it in your voice webhook, right before you return the TwiML. This Flask webhook uses the Twilio helper library (pip install flask requests twilio):

Python
# voice.py: Twilio voice webhook that streams each call to Noesis
import os

import requests
from flask import Flask
from twilio.twiml.voice_response import Start, VoiceResponse

app = Flask(__name__)

@app.post("/voice")
def voice():
    response = VoiceResponse()
    try:
        r = requests.post(
            "https://api.noesis.detectif.ai/v1/detect/stream-tokens",
            headers={"Authorization": f"Bearer {os.environ['DETECTIF_API_KEY']}"},
            json={},
            timeout=5,
        )
        r.raise_for_status()
        start = Start()
        stream = start.stream(url="wss://api.noesis.detectif.ai/v1/detect/stream",
                              track="both_tracks")
        stream.parameter(name="token", value=r.json()["token"])
        response.append(start)
    except requests.RequestException:
        pass                                        # never block the call on Noesis
    response.say("Your existing call flow continues here.")
    return str(response), 200, {"Content-Type": "text/xml"}

3. Receive the answers during the call

With the stream pointed straight at Noesis, the answers go back to Twilio, which ignores them. To act on them, point the <Stream> at a relay on your server instead. The relay forwards Twilio's messages to Noesis unchanged, with your API key in the header, so it needs no token:

XML
<Response>
  <Start>
    <Stream url="wss://relay.example.com/twilio" track="both_tracks"/>
  </Start>
  <!-- your existing call flow continues here -->
</Response>

The relay uses noesis_relay.py from the relay helper:

Python
# twilio_relay.py: Twilio Media Stream -> your server -> Noesis
import asyncio

import websockets

from noesis_relay import NoesisCall

async def on_answer(event):
    if event["type"] == "classification":
        print(event["call_id"], event["label"])     # your own logic; call_id is the CallSid

async def handle(twilio):
    call = NoesisCall(on_answer)                    # Twilio's format is the default
    await call.open()
    try:
        async for message in twilio:                # Twilio's JSON messages, unchanged
            await call.send(message)
    except websockets.ConnectionClosed:
        pass
    await call.close()

async def main():
    async with websockets.serve(handle, "0.0.0.0", 8080):
        await asyncio.Future()                      # run until stopped

asyncio.run(main())

Act on the latest seq for each CallSid. For example, update the live call with Twilio's REST API when the answer is voicemail. To not record calls through the relay, create the call with NoesisCall(on_answer, record="false").