Portfolio/blog-posts/17-conversational-ai-turn-taking.md

Conversational AI 2.0: Natürliches Turn-Taking und Interruption Handling

Meta-Description: Fortgeschrittene Techniken für natürliche Sprachinteraktion. Turn-Taking-Modelle, intelligente Unterbrechungserkennung und Full-Duplex-Kommunikation.

Keywords: Conversational AI, Turn-Taking, Interruption Handling, Full-Duplex, Voice Agent, Natural Conversation, VAD, TRP

---

Einführung

Die größte Herausforderung für Voice AI ist nicht die Spracherkennung – es ist das Timing. Wann ist der User fertig? Wann darf der Agent sprechen? Wie reagiert man auf Unterbrechungen?

"OpenAIs neue Audio-Architektur 2026 zielt auf niedrigere Latenz und natürlicheres Back-and-Forth ab – weg vom Walkie-Talkie-Modell."

---

Das Problem mit Silence-Based Detection

Warum einfache Stille nicht funktioniert

┌─────────────────────────────────────────────────────────────┐
│              SILENCE-BASED VS. INTELLIGENT DETECTION        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Silence-Based:                                             │
│  User: "Ich möchte..." [500ms Pause] "...einen Kaffee"     │
│  Agent: [Unterbricht bei Pause] "Wie kann ich helfen?"     │
│  ❌ Frustrierend für User                                   │
│                                                             │
│  Intelligent Turn-Taking:                                   │
│  User: "Ich möchte..." [500ms Pause] "...einen Kaffee"     │
│  Agent: [Wartet auf semantisches Ende]                      │
│  User: "...mit Milch."                                      │
│  Agent: "Einen Kaffee mit Milch, kommt sofort!"            │
│  ✅ Natürliche Konversation                                 │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Transition-Relevance Places (TRPs)

Natürliche Gespräche haben Übergangspunkte, die nicht nur durch Stille definiert sind:

Signal	Beispiel	Zuverlässigkeit
Syntaktisch	Vollständiger Satz	Hoch
Prosodisch	Fallende Intonation	Mittel
Pragmatisch	Frage gestellt	Hoch
Semantisch	Gedanke abgeschlossen	Mittel
Stille	> 700ms Pause	Niedrig

---

Turn-Taking Architektur

// src/turn-taking/detector.ts
interface TurnTakingSignals {
  // Audio-basiert
  silenceDurationMs: number;
  pitchContour: 'rising' | 'falling' | 'flat';
  speechRate: number;

  // Text-basiert (von ASR)
  lastUtterance: string;
  isQuestion: boolean;
  isComplete: boolean;

  // Kontext
  conversationHistory: Message[];
  expectedResponseType: 'answer' | 'continuation' | 'acknowledgment';
}

interface TurnDecision {
  action: 'wait' | 'respond' | 'backchannel';
  confidence: number;
  reasoning: string;
}

class IntelligentTurnDetector {
  private model: TurnPredictionModel;

  constructor() {
    this.model = new TurnPredictionModel();
  }

  async predict(signals: TurnTakingSignals): Promise<TurnDecision> {
    // Multi-Signal-Analyse
    const features = this.extractFeatures(signals);

    // Heuristiken für schnelle Entscheidungen
    if (signals.silenceDurationMs > 2000) {
      return { action: 'respond', confidence: 0.95, reasoning: 'Long silence' };
    }

    if (signals.isQuestion && signals.silenceDurationMs > 300) {
      return { action: 'respond', confidence: 0.9, reasoning: 'Question asked' };
    }

    // ML-basierte Prediction für komplexe Fälle
    const prediction = await this.model.predict(features);

    return prediction;
  }

  private extractFeatures(signals: TurnTakingSignals) {
    return {
      // Normalisierte Features für ML
      silenceNormalized: Math.min(signals.silenceDurationMs / 2000, 1),
      pitchFalling: signals.pitchContour === 'falling' ? 1 : 0,
      syntacticCompleteness: this.analyzeSyntax(signals.lastUtterance),
      semanticCompleteness: this.analyzeSemantics(signals.lastUtterance),
      questionProbability: signals.isQuestion ? 1 : 0,
      // ...weitere Features
    };
  }

  private analyzeSyntax(utterance: string): number {
    // Einfache Heuristik: Endet mit Satzzeichen?
    if (/[.!?]$/.test(utterance.trim())) return 1;

    // Unvollständiger Satz
    if (/\b(und|aber|oder|weil|dass)\s*$/.test(utterance)) return 0;

    return 0.5;
  }

  private analyzeSemantics(utterance: string): number {
    // Kann mit LLM verbessert werden
    const incompletePatterns = [
      /ich möchte\s*$/i,
      /ich brauche\s*$/i,
      /können Sie\s*$/i,
      /wie wäre es\s*$/i
    ];

    for (const pattern of incompletePatterns) {
      if (pattern.test(utterance)) return 0;
    }

    return 0.7;
  }
}

---

Interruption Handling

Typen von Unterbrechungen

enum InterruptionType {
  // Kooperativ
  AGREEMENT = 'agreement',        // "Ja, genau!"
  ASSISTANCE = 'assistance',      // User hilft Agent beim Formulieren
  CLARIFICATION = 'clarification', // "Moment, was meinen Sie?"

  // Disruptiv
  DISAGREEMENT = 'disagreement',  // "Nein, das stimmt nicht"
  TOPIC_CHANGE = 'topic_change',  // "Egal, andere Frage..."
  CORRECTION = 'correction',      // "Nicht 3, ich sagte 5"

  // Technisch
  BARGE_IN = 'barge_in'          // User will Agent stoppen
}

interface InterruptionEvent {
  type: InterruptionType;
  timestamp: number;
  userUtterance: string;
  agentWasSpeaking: boolean;
  agentUtteranceProgress: number; // 0-1
}

Intelligente Reaktionen

// src/turn-taking/interruption-handler.ts
class InterruptionHandler {
  async handleInterruption(
    event: InterruptionEvent,
    agent: VoiceAgent
  ): Promise<void> {
    // 1. Agent sofort stoppen
    await agent.stopSpeaking();

    // 2. Typ klassifizieren
    const type = await this.classifyInterruption(event);

    // 3. Entsprechend reagieren
    switch (type) {
      case InterruptionType.AGREEMENT:
        // Kurz bestätigen, dann weitermachen
        await agent.say("Genau.");
        await agent.continueFromLastPoint();
        break;

      case InterruptionType.CLARIFICATION:
        // Erklärung geben
        await agent.say("Lass mich das erklären...");
        await agent.clarifyLastStatement();
        break;

      case InterruptionType.CORRECTION:
        // Korrektur akzeptieren
        await agent.say("Entschuldigung, ich korrigiere...");
        await agent.processUserInput(event.userUtterance);
        break;

      case InterruptionType.BARGE_IN:
        // Komplett neuen Input verarbeiten
        await agent.processUserInput(event.userUtterance);
        break;

      case InterruptionType.TOPIC_CHANGE:
        // Context wechseln
        await agent.say("Okay, zum neuen Thema...");
        await agent.processUserInput(event.userUtterance);
        break;
    }
  }

  private async classifyInterruption(
    event: InterruptionEvent
  ): Promise<InterruptionType> {
    // Schnelle Heuristiken
    const text = event.userUtterance.toLowerCase();

    if (/^(ja|genau|richtig|stimmt)/.test(text)) {
      return InterruptionType.AGREEMENT;
    }

    if (/^(nein|falsch|nicht|stop)/.test(text)) {
      return InterruptionType.DISAGREEMENT;
    }

    if (/^(was|wie|warum|moment)/.test(text)) {
      return InterruptionType.CLARIFICATION;
    }

    // LLM für komplexere Fälle
    return await this.classifyWithLLM(event);
  }
}

---

Full-Duplex Communication

Das NVIDIA PersonaPlex Konzept

// Full-Duplex: Agent kann gleichzeitig hören und sprechen
class FullDuplexAgent {
  private isSpeaking = false;
  private isListening = true; // Immer an
  private audioBuffer: Buffer[] = [];

  async processAudioStream(
    input: AsyncIterable<Buffer>,
    output: (chunk: Buffer) => void
  ) {
    // Parallele Verarbeitung
    const [transcription, speechOutput] = await Promise.all([
      this.transcribeStream(input),
      this.generateSpeech()
    ]);

    // Während Agent spricht, weiter zuhören
    for await (const audioChunk of input) {
      // VAD prüfen
      if (this.detectVoiceActivity(audioChunk)) {
        // User spricht während Agent spricht
        if (this.isSpeaking) {
          await this.handleOverlap(audioChunk);
        }
      }
    }
  }

  private async handleOverlap(userAudio: Buffer) {
    // Analyse: Ist es eine Unterbrechung?
    const energy = this.calculateEnergy(userAudio);

    if (energy > this.thresholdForInterruption) {
      // Lautstärke des Agents reduzieren
      this.reduceAgentVolume(0.3);

      // Warten ob User weiterspricht
      await this.waitForUserIntent(500);

      if (this.userContinuesSpeaking) {
        // Komplett stoppen
        await this.stopSpeaking();
      } else {
        // War nur Backchannel, weitermachen
        this.restoreAgentVolume();
      }
    }
  }
}

---

Backchannel Responses

Natürliche Bestätigungen während User spricht

interface BackchannelConfig {
  enabled: boolean;
  responses: string[];
  triggerInterval: number; // ms
  maxPerTurn: number;
}

class BackchannelGenerator {
  private config: BackchannelConfig = {
    enabled: true,
    responses: ['Mhm', 'Ja', 'Verstehe', 'Okay', 'Aha'],
    triggerInterval: 3000,
    maxPerTurn: 3
  };

  private backchannelCount = 0;
  private lastBackchannel = 0;

  shouldGenerateBackchannel(
    signals: TurnTakingSignals
  ): { should: boolean; response: string } {
    const now = Date.now();

    // Limits prüfen
    if (this.backchannelCount >= this.config.maxPerTurn) {
      return { should: false, response: '' };
    }

    if (now - this.lastBackchannel < this.config.triggerInterval) {
      return { should: false, response: '' };
    }

    // Trigger-Bedingungen
    const shouldTrigger =
      signals.silenceDurationMs > 200 &&
      signals.silenceDurationMs < 500 &&
      !signals.isComplete &&
      signals.lastUtterance.length > 20;

    if (shouldTrigger) {
      this.backchannelCount++;
      this.lastBackchannel = now;

      const response = this.selectResponse(signals);
      return { should: true, response };
    }

    return { should: false, response: '' };
  }

  private selectResponse(signals: TurnTakingSignals): string {
    // Kontext-abhängige Auswahl
    if (signals.lastUtterance.includes('Problem')) {
      return 'Oh je';
    }

    if (signals.lastUtterance.includes('?')) {
      return 'Mhm';
    }

    // Random für Variation
    const idx = Math.floor(Math.random() * this.config.responses.length);
    return this.config.responses[idx];
  }

  resetForNewTurn() {
    this.backchannelCount = 0;
  }
}

---

Production-Ready Implementation

// src/voice-agent-v2.ts
import { DeepgramStreamer } from './services/deepgram';
import { ElevenLabsStreamer } from './services/elevenlabs';
import { IntelligentTurnDetector } from './turn-taking/detector';
import { InterruptionHandler } from './turn-taking/interruption-handler';
import { BackchannelGenerator } from './turn-taking/backchannel';

export class ConversationalVoiceAgent {
  private deepgram = new DeepgramStreamer();
  private elevenlabs = new ElevenLabsStreamer();
  private turnDetector = new IntelligentTurnDetector();
  private interruptionHandler = new InterruptionHandler();
  private backchannel = new BackchannelGenerator();

  private state: 'listening' | 'processing' | 'speaking' = 'listening';
  private currentUtterance = '';

  async start(
    audioInput: AsyncIterable<Buffer>,
    onAudioOutput: (chunk: Buffer) => void
  ) {
    await this.deepgram.startStream(
      {
        model: 'nova-2',
        language: 'de',
        smart_format: true,
        interim_results: true,
        endpointing: 300 // Schnelles Feedback
      },
      async (text, isFinal) => {
        this.currentUtterance = text;

        // Interim: Prüfe auf Unterbrechung
        if (!isFinal && this.state === 'speaking') {
          await this.interruptionHandler.handleInterruption(
            {
              type: InterruptionType.BARGE_IN,
              timestamp: Date.now(),
              userUtterance: text,
              agentWasSpeaking: true,
              agentUtteranceProgress: 0.5
            },
            this
          );
          return;
        }

        // Final: Turn-Taking-Entscheidung
        if (isFinal) {
          const decision = await this.turnDetector.predict({
            silenceDurationMs: 0,
            pitchContour: 'falling',
            speechRate: 1,
            lastUtterance: text,
            isQuestion: text.includes('?'),
            isComplete: true,
            conversationHistory: [],
            expectedResponseType: 'answer'
          });

          if (decision.action === 'respond') {
            this.state = 'processing';
            await this.generateResponse(text, onAudioOutput);
          }
        } else {
          // Backchannel prüfen
          const bc = this.backchannel.shouldGenerateBackchannel({
            silenceDurationMs: 300,
            pitchContour: 'flat',
            speechRate: 1,
            lastUtterance: text,
            isQuestion: false,
            isComplete: false,
            conversationHistory: [],
            expectedResponseType: 'continuation'
          });

          if (bc.should) {
            // Leises Backchannel ohne State-Wechsel
            await this.speakQuietly(bc.response, onAudioOutput);
          }
        }
      }
    );

    for await (const chunk of audioInput) {
      this.deepgram.sendAudio(chunk);
    }
  }

  private async speakQuietly(
    text: string,
    onOutput: (chunk: Buffer) => void
  ) {
    // Niedrige Lautstärke für Backchannel
    for await (const chunk of this.elevenlabs.streamSpeech(text, {
      voiceId: 'default',
      modelId: 'eleven_flash_v2_5',
      stability: 0.3,
      similarityBoost: 0.5,
      latencyOptimization: 4
    })) {
      onOutput(this.reduceVolume(chunk, 0.4));
    }
  }

  private reduceVolume(audio: Buffer, factor: number): Buffer {
    // PCM volume reduction
    const samples = new Int16Array(audio.buffer);
    for (let i = 0; i < samples.length; i++) {
      samples[i] = Math.round(samples[i] * factor);
    }
    return Buffer.from(samples.buffer);
  }

  async stopSpeaking() {
    // Implementation für sofortigen Stop
    this.state = 'listening';
  }

  private async generateResponse(
    input: string,
    onOutput: (chunk: Buffer) => void
  ) {
    this.state = 'speaking';
    // LLM + TTS Pipeline...
    this.state = 'listening';
    this.backchannel.resetForNewTurn();
  }
}

---

Metriken für Turn-Taking Qualität

Metrik	Zielwert	Beschreibung
Turn-Taking Latenz	< 500ms	Zeit von User-Ende bis Agent-Start
False Interruptions	< 5%	Agent unterbricht User fälschlich
Missed TRPs	< 10%	Agent verpasst Übergangspunkte
Interruption Recovery	< 1s	Zeit bis normale Konversation

---

Fazit

Natürliches Turn-Taking erfordert:

1. Multi-Signal-Analyse: Nicht nur Stille, sondern Syntax + Semantik + Prosodie
2. Intelligente Unterbrechungserkennung: Kooperativ vs. Disruptiv unterscheiden
3. Backchannel-Responses: Aktives Zuhören signalisieren
4. Full-Duplex: Gleichzeitig hören und sprechen können

Das Walkie-Talkie-Modell ist Geschichte.

---

Bildprompts

1. "Two people in natural conversation with speech bubbles overlapping, timing visualization, warm illustration style"
2. "AI voice assistant with sound waves showing bidirectional flow, full-duplex concept, modern tech art"
3. "Conversation flow diagram with turn-taking points marked, linguistic analysis visualization"

---

Quellen

• NVIDIA PersonaPlex Research
• Amazon Nova 2 Sonic Announcement
• Retell AI Turn-Taking Model
• Interruption Handling Research (arXiv)
• Turn-Taking in Conversational Systems (MDPI)