damjan_savicandClaude Opus 4.5
e1bbe5455d
Vollständige Next.js 15 Portfolio-Website mit: - Blog-System mit 100+ Artikeln - Supabase-Integration - Responsive Design mit Tailwind CSS - TypeScript-Konfiguration - Testing-Setup mit Vitest und Playwright Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
564 lines
16 KiB
Markdown
564 lines
16 KiB
Markdown
# Conversational AI 2.0: Natürliches Turn-Taking und Interruption Handling
|
||
|
||
**Meta-Description:** Fortgeschrittene Techniken für natürliche Sprachinteraktion. Turn-Taking-Modelle, intelligente Unterbrechungserkennung und Full-Duplex-Kommunikation.
|
||
|
||
**Keywords:** Conversational AI, Turn-Taking, Interruption Handling, Full-Duplex, Voice Agent, Natural Conversation, VAD, TRP
|
||
|
||
---
|
||
|
||
## Einführung
|
||
|
||
Die größte Herausforderung für Voice AI ist nicht die Spracherkennung – es ist das **Timing**. Wann ist der User fertig? Wann darf der Agent sprechen? Wie reagiert man auf Unterbrechungen?
|
||
|
||
> "OpenAIs neue Audio-Architektur 2026 zielt auf niedrigere Latenz und natürlicheres Back-and-Forth ab – weg vom Walkie-Talkie-Modell."
|
||
|
||
---
|
||
|
||
## Das Problem mit Silence-Based Detection
|
||
|
||
### Warum einfache Stille nicht funktioniert
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ SILENCE-BASED VS. INTELLIGENT DETECTION │
|
||
├─────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Silence-Based: │
|
||
│ User: "Ich möchte..." [500ms Pause] "...einen Kaffee" │
|
||
│ Agent: [Unterbricht bei Pause] "Wie kann ich helfen?" │
|
||
│ ❌ Frustrierend für User │
|
||
│ │
|
||
│ Intelligent Turn-Taking: │
|
||
│ User: "Ich möchte..." [500ms Pause] "...einen Kaffee" │
|
||
│ Agent: [Wartet auf semantisches Ende] │
|
||
│ User: "...mit Milch." │
|
||
│ Agent: "Einen Kaffee mit Milch, kommt sofort!" │
|
||
│ ✅ Natürliche Konversation │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Transition-Relevance Places (TRPs)
|
||
|
||
Natürliche Gespräche haben **Übergangspunkte**, die nicht nur durch Stille definiert sind:
|
||
|
||
| Signal | Beispiel | Zuverlässigkeit |
|
||
|--------|----------|-----------------|
|
||
| **Syntaktisch** | Vollständiger Satz | Hoch |
|
||
| **Prosodisch** | Fallende Intonation | Mittel |
|
||
| **Pragmatisch** | Frage gestellt | Hoch |
|
||
| **Semantisch** | Gedanke abgeschlossen | Mittel |
|
||
| **Stille** | > 700ms Pause | Niedrig |
|
||
|
||
---
|
||
|
||
## Turn-Taking Architektur
|
||
|
||
```typescript
|
||
// src/turn-taking/detector.ts
|
||
interface TurnTakingSignals {
|
||
// Audio-basiert
|
||
silenceDurationMs: number;
|
||
pitchContour: 'rising' | 'falling' | 'flat';
|
||
speechRate: number;
|
||
|
||
// Text-basiert (von ASR)
|
||
lastUtterance: string;
|
||
isQuestion: boolean;
|
||
isComplete: boolean;
|
||
|
||
// Kontext
|
||
conversationHistory: Message[];
|
||
expectedResponseType: 'answer' | 'continuation' | 'acknowledgment';
|
||
}
|
||
|
||
interface TurnDecision {
|
||
action: 'wait' | 'respond' | 'backchannel';
|
||
confidence: number;
|
||
reasoning: string;
|
||
}
|
||
|
||
class IntelligentTurnDetector {
|
||
private model: TurnPredictionModel;
|
||
|
||
constructor() {
|
||
this.model = new TurnPredictionModel();
|
||
}
|
||
|
||
async predict(signals: TurnTakingSignals): Promise<TurnDecision> {
|
||
// Multi-Signal-Analyse
|
||
const features = this.extractFeatures(signals);
|
||
|
||
// Heuristiken für schnelle Entscheidungen
|
||
if (signals.silenceDurationMs > 2000) {
|
||
return { action: 'respond', confidence: 0.95, reasoning: 'Long silence' };
|
||
}
|
||
|
||
if (signals.isQuestion && signals.silenceDurationMs > 300) {
|
||
return { action: 'respond', confidence: 0.9, reasoning: 'Question asked' };
|
||
}
|
||
|
||
// ML-basierte Prediction für komplexe Fälle
|
||
const prediction = await this.model.predict(features);
|
||
|
||
return prediction;
|
||
}
|
||
|
||
private extractFeatures(signals: TurnTakingSignals) {
|
||
return {
|
||
// Normalisierte Features für ML
|
||
silenceNormalized: Math.min(signals.silenceDurationMs / 2000, 1),
|
||
pitchFalling: signals.pitchContour === 'falling' ? 1 : 0,
|
||
syntacticCompleteness: this.analyzeSyntax(signals.lastUtterance),
|
||
semanticCompleteness: this.analyzeSemantics(signals.lastUtterance),
|
||
questionProbability: signals.isQuestion ? 1 : 0,
|
||
// ...weitere Features
|
||
};
|
||
}
|
||
|
||
private analyzeSyntax(utterance: string): number {
|
||
// Einfache Heuristik: Endet mit Satzzeichen?
|
||
if (/[.!?]$/.test(utterance.trim())) return 1;
|
||
|
||
// Unvollständiger Satz
|
||
if (/\b(und|aber|oder|weil|dass)\s*$/.test(utterance)) return 0;
|
||
|
||
return 0.5;
|
||
}
|
||
|
||
private analyzeSemantics(utterance: string): number {
|
||
// Kann mit LLM verbessert werden
|
||
const incompletePatterns = [
|
||
/ich möchte\s*$/i,
|
||
/ich brauche\s*$/i,
|
||
/können Sie\s*$/i,
|
||
/wie wäre es\s*$/i
|
||
];
|
||
|
||
for (const pattern of incompletePatterns) {
|
||
if (pattern.test(utterance)) return 0;
|
||
}
|
||
|
||
return 0.7;
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Interruption Handling
|
||
|
||
### Typen von Unterbrechungen
|
||
|
||
```typescript
|
||
enum InterruptionType {
|
||
// Kooperativ
|
||
AGREEMENT = 'agreement', // "Ja, genau!"
|
||
ASSISTANCE = 'assistance', // User hilft Agent beim Formulieren
|
||
CLARIFICATION = 'clarification', // "Moment, was meinen Sie?"
|
||
|
||
// Disruptiv
|
||
DISAGREEMENT = 'disagreement', // "Nein, das stimmt nicht"
|
||
TOPIC_CHANGE = 'topic_change', // "Egal, andere Frage..."
|
||
CORRECTION = 'correction', // "Nicht 3, ich sagte 5"
|
||
|
||
// Technisch
|
||
BARGE_IN = 'barge_in' // User will Agent stoppen
|
||
}
|
||
|
||
interface InterruptionEvent {
|
||
type: InterruptionType;
|
||
timestamp: number;
|
||
userUtterance: string;
|
||
agentWasSpeaking: boolean;
|
||
agentUtteranceProgress: number; // 0-1
|
||
}
|
||
```
|
||
|
||
### Intelligente Reaktionen
|
||
|
||
```typescript
|
||
// src/turn-taking/interruption-handler.ts
|
||
class InterruptionHandler {
|
||
async handleInterruption(
|
||
event: InterruptionEvent,
|
||
agent: VoiceAgent
|
||
): Promise<void> {
|
||
// 1. Agent sofort stoppen
|
||
await agent.stopSpeaking();
|
||
|
||
// 2. Typ klassifizieren
|
||
const type = await this.classifyInterruption(event);
|
||
|
||
// 3. Entsprechend reagieren
|
||
switch (type) {
|
||
case InterruptionType.AGREEMENT:
|
||
// Kurz bestätigen, dann weitermachen
|
||
await agent.say("Genau.");
|
||
await agent.continueFromLastPoint();
|
||
break;
|
||
|
||
case InterruptionType.CLARIFICATION:
|
||
// Erklärung geben
|
||
await agent.say("Lass mich das erklären...");
|
||
await agent.clarifyLastStatement();
|
||
break;
|
||
|
||
case InterruptionType.CORRECTION:
|
||
// Korrektur akzeptieren
|
||
await agent.say("Entschuldigung, ich korrigiere...");
|
||
await agent.processUserInput(event.userUtterance);
|
||
break;
|
||
|
||
case InterruptionType.BARGE_IN:
|
||
// Komplett neuen Input verarbeiten
|
||
await agent.processUserInput(event.userUtterance);
|
||
break;
|
||
|
||
case InterruptionType.TOPIC_CHANGE:
|
||
// Context wechseln
|
||
await agent.say("Okay, zum neuen Thema...");
|
||
await agent.processUserInput(event.userUtterance);
|
||
break;
|
||
}
|
||
}
|
||
|
||
private async classifyInterruption(
|
||
event: InterruptionEvent
|
||
): Promise<InterruptionType> {
|
||
// Schnelle Heuristiken
|
||
const text = event.userUtterance.toLowerCase();
|
||
|
||
if (/^(ja|genau|richtig|stimmt)/.test(text)) {
|
||
return InterruptionType.AGREEMENT;
|
||
}
|
||
|
||
if (/^(nein|falsch|nicht|stop)/.test(text)) {
|
||
return InterruptionType.DISAGREEMENT;
|
||
}
|
||
|
||
if (/^(was|wie|warum|moment)/.test(text)) {
|
||
return InterruptionType.CLARIFICATION;
|
||
}
|
||
|
||
// LLM für komplexere Fälle
|
||
return await this.classifyWithLLM(event);
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Full-Duplex Communication
|
||
|
||
### Das NVIDIA PersonaPlex Konzept
|
||
|
||
```typescript
|
||
// Full-Duplex: Agent kann gleichzeitig hören und sprechen
|
||
class FullDuplexAgent {
|
||
private isSpeaking = false;
|
||
private isListening = true; // Immer an
|
||
private audioBuffer: Buffer[] = [];
|
||
|
||
async processAudioStream(
|
||
input: AsyncIterable<Buffer>,
|
||
output: (chunk: Buffer) => void
|
||
) {
|
||
// Parallele Verarbeitung
|
||
const [transcription, speechOutput] = await Promise.all([
|
||
this.transcribeStream(input),
|
||
this.generateSpeech()
|
||
]);
|
||
|
||
// Während Agent spricht, weiter zuhören
|
||
for await (const audioChunk of input) {
|
||
// VAD prüfen
|
||
if (this.detectVoiceActivity(audioChunk)) {
|
||
// User spricht während Agent spricht
|
||
if (this.isSpeaking) {
|
||
await this.handleOverlap(audioChunk);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
private async handleOverlap(userAudio: Buffer) {
|
||
// Analyse: Ist es eine Unterbrechung?
|
||
const energy = this.calculateEnergy(userAudio);
|
||
|
||
if (energy > this.thresholdForInterruption) {
|
||
// Lautstärke des Agents reduzieren
|
||
this.reduceAgentVolume(0.3);
|
||
|
||
// Warten ob User weiterspricht
|
||
await this.waitForUserIntent(500);
|
||
|
||
if (this.userContinuesSpeaking) {
|
||
// Komplett stoppen
|
||
await this.stopSpeaking();
|
||
} else {
|
||
// War nur Backchannel, weitermachen
|
||
this.restoreAgentVolume();
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Backchannel Responses
|
||
|
||
### Natürliche Bestätigungen während User spricht
|
||
|
||
```typescript
|
||
interface BackchannelConfig {
|
||
enabled: boolean;
|
||
responses: string[];
|
||
triggerInterval: number; // ms
|
||
maxPerTurn: number;
|
||
}
|
||
|
||
class BackchannelGenerator {
|
||
private config: BackchannelConfig = {
|
||
enabled: true,
|
||
responses: ['Mhm', 'Ja', 'Verstehe', 'Okay', 'Aha'],
|
||
triggerInterval: 3000,
|
||
maxPerTurn: 3
|
||
};
|
||
|
||
private backchannelCount = 0;
|
||
private lastBackchannel = 0;
|
||
|
||
shouldGenerateBackchannel(
|
||
signals: TurnTakingSignals
|
||
): { should: boolean; response: string } {
|
||
const now = Date.now();
|
||
|
||
// Limits prüfen
|
||
if (this.backchannelCount >= this.config.maxPerTurn) {
|
||
return { should: false, response: '' };
|
||
}
|
||
|
||
if (now - this.lastBackchannel < this.config.triggerInterval) {
|
||
return { should: false, response: '' };
|
||
}
|
||
|
||
// Trigger-Bedingungen
|
||
const shouldTrigger =
|
||
signals.silenceDurationMs > 200 &&
|
||
signals.silenceDurationMs < 500 &&
|
||
!signals.isComplete &&
|
||
signals.lastUtterance.length > 20;
|
||
|
||
if (shouldTrigger) {
|
||
this.backchannelCount++;
|
||
this.lastBackchannel = now;
|
||
|
||
const response = this.selectResponse(signals);
|
||
return { should: true, response };
|
||
}
|
||
|
||
return { should: false, response: '' };
|
||
}
|
||
|
||
private selectResponse(signals: TurnTakingSignals): string {
|
||
// Kontext-abhängige Auswahl
|
||
if (signals.lastUtterance.includes('Problem')) {
|
||
return 'Oh je';
|
||
}
|
||
|
||
if (signals.lastUtterance.includes('?')) {
|
||
return 'Mhm';
|
||
}
|
||
|
||
// Random für Variation
|
||
const idx = Math.floor(Math.random() * this.config.responses.length);
|
||
return this.config.responses[idx];
|
||
}
|
||
|
||
resetForNewTurn() {
|
||
this.backchannelCount = 0;
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Production-Ready Implementation
|
||
|
||
```typescript
|
||
// src/voice-agent-v2.ts
|
||
import { DeepgramStreamer } from './services/deepgram';
|
||
import { ElevenLabsStreamer } from './services/elevenlabs';
|
||
import { IntelligentTurnDetector } from './turn-taking/detector';
|
||
import { InterruptionHandler } from './turn-taking/interruption-handler';
|
||
import { BackchannelGenerator } from './turn-taking/backchannel';
|
||
|
||
export class ConversationalVoiceAgent {
|
||
private deepgram = new DeepgramStreamer();
|
||
private elevenlabs = new ElevenLabsStreamer();
|
||
private turnDetector = new IntelligentTurnDetector();
|
||
private interruptionHandler = new InterruptionHandler();
|
||
private backchannel = new BackchannelGenerator();
|
||
|
||
private state: 'listening' | 'processing' | 'speaking' = 'listening';
|
||
private currentUtterance = '';
|
||
|
||
async start(
|
||
audioInput: AsyncIterable<Buffer>,
|
||
onAudioOutput: (chunk: Buffer) => void
|
||
) {
|
||
await this.deepgram.startStream(
|
||
{
|
||
model: 'nova-2',
|
||
language: 'de',
|
||
smart_format: true,
|
||
interim_results: true,
|
||
endpointing: 300 // Schnelles Feedback
|
||
},
|
||
async (text, isFinal) => {
|
||
this.currentUtterance = text;
|
||
|
||
// Interim: Prüfe auf Unterbrechung
|
||
if (!isFinal && this.state === 'speaking') {
|
||
await this.interruptionHandler.handleInterruption(
|
||
{
|
||
type: InterruptionType.BARGE_IN,
|
||
timestamp: Date.now(),
|
||
userUtterance: text,
|
||
agentWasSpeaking: true,
|
||
agentUtteranceProgress: 0.5
|
||
},
|
||
this
|
||
);
|
||
return;
|
||
}
|
||
|
||
// Final: Turn-Taking-Entscheidung
|
||
if (isFinal) {
|
||
const decision = await this.turnDetector.predict({
|
||
silenceDurationMs: 0,
|
||
pitchContour: 'falling',
|
||
speechRate: 1,
|
||
lastUtterance: text,
|
||
isQuestion: text.includes('?'),
|
||
isComplete: true,
|
||
conversationHistory: [],
|
||
expectedResponseType: 'answer'
|
||
});
|
||
|
||
if (decision.action === 'respond') {
|
||
this.state = 'processing';
|
||
await this.generateResponse(text, onAudioOutput);
|
||
}
|
||
} else {
|
||
// Backchannel prüfen
|
||
const bc = this.backchannel.shouldGenerateBackchannel({
|
||
silenceDurationMs: 300,
|
||
pitchContour: 'flat',
|
||
speechRate: 1,
|
||
lastUtterance: text,
|
||
isQuestion: false,
|
||
isComplete: false,
|
||
conversationHistory: [],
|
||
expectedResponseType: 'continuation'
|
||
});
|
||
|
||
if (bc.should) {
|
||
// Leises Backchannel ohne State-Wechsel
|
||
await this.speakQuietly(bc.response, onAudioOutput);
|
||
}
|
||
}
|
||
}
|
||
);
|
||
|
||
for await (const chunk of audioInput) {
|
||
this.deepgram.sendAudio(chunk);
|
||
}
|
||
}
|
||
|
||
private async speakQuietly(
|
||
text: string,
|
||
onOutput: (chunk: Buffer) => void
|
||
) {
|
||
// Niedrige Lautstärke für Backchannel
|
||
for await (const chunk of this.elevenlabs.streamSpeech(text, {
|
||
voiceId: 'default',
|
||
modelId: 'eleven_flash_v2_5',
|
||
stability: 0.3,
|
||
similarityBoost: 0.5,
|
||
latencyOptimization: 4
|
||
})) {
|
||
onOutput(this.reduceVolume(chunk, 0.4));
|
||
}
|
||
}
|
||
|
||
private reduceVolume(audio: Buffer, factor: number): Buffer {
|
||
// PCM volume reduction
|
||
const samples = new Int16Array(audio.buffer);
|
||
for (let i = 0; i < samples.length; i++) {
|
||
samples[i] = Math.round(samples[i] * factor);
|
||
}
|
||
return Buffer.from(samples.buffer);
|
||
}
|
||
|
||
async stopSpeaking() {
|
||
// Implementation für sofortigen Stop
|
||
this.state = 'listening';
|
||
}
|
||
|
||
private async generateResponse(
|
||
input: string,
|
||
onOutput: (chunk: Buffer) => void
|
||
) {
|
||
this.state = 'speaking';
|
||
// LLM + TTS Pipeline...
|
||
this.state = 'listening';
|
||
this.backchannel.resetForNewTurn();
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Metriken für Turn-Taking Qualität
|
||
|
||
| Metrik | Zielwert | Beschreibung |
|
||
|--------|----------|--------------|
|
||
| **Turn-Taking Latenz** | < 500ms | Zeit von User-Ende bis Agent-Start |
|
||
| **False Interruptions** | < 5% | Agent unterbricht User fälschlich |
|
||
| **Missed TRPs** | < 10% | Agent verpasst Übergangspunkte |
|
||
| **Interruption Recovery** | < 1s | Zeit bis normale Konversation |
|
||
|
||
---
|
||
|
||
## Fazit
|
||
|
||
Natürliches Turn-Taking erfordert:
|
||
|
||
1. **Multi-Signal-Analyse**: Nicht nur Stille, sondern Syntax + Semantik + Prosodie
|
||
2. **Intelligente Unterbrechungserkennung**: Kooperativ vs. Disruptiv unterscheiden
|
||
3. **Backchannel-Responses**: Aktives Zuhören signalisieren
|
||
4. **Full-Duplex**: Gleichzeitig hören und sprechen können
|
||
|
||
Das Walkie-Talkie-Modell ist Geschichte.
|
||
|
||
---
|
||
|
||
## Bildprompts
|
||
|
||
1. "Two people in natural conversation with speech bubbles overlapping, timing visualization, warm illustration style"
|
||
2. "AI voice assistant with sound waves showing bidirectional flow, full-duplex concept, modern tech art"
|
||
3. "Conversation flow diagram with turn-taking points marked, linguistic analysis visualization"
|
||
|
||
---
|
||
|
||
## Quellen
|
||
|
||
- [NVIDIA PersonaPlex Research](https://research.nvidia.com/labs/adlr/personaplex/)
|
||
- [Amazon Nova 2 Sonic Announcement](https://aws.amazon.com/blogs/aws/introducing-amazon-nova-2-sonic-next-generation-speech-to-speech-model-for-conversational-ai/)
|
||
- [Retell AI Turn-Taking Model](https://www.retellai.com/blog/how-retell-ais-turn-taking-model-ensures-seamless-calls)
|
||
- [Interruption Handling Research (arXiv)](https://arxiv.org/html/2501.01568v1)
|
||
- [Turn-Taking in Conversational Systems (MDPI)](https://www.mdpi.com/2227-7080/13/12/591)
|