Repository navigation
feat(twilio): keep a reply the caller cut off as the part they heard - #1467
Conversation
ConversationRelay stops playback where the caller cuts in, but the whole reply still went into the conversation history, so the next turn believed the question had been asked in full and never came back to it. The relay session now gets the last say over what the history keeps: for a reply cut off while it was still being streamed it returns the text that actually went out, with the rest named in a mark. The dialog is appended after that callback, so the stored message is the marked one, while what is spoken is unchanged. Only a reply cut off mid-stream can be marked; once the whole reply is out, Twilio holds the text it has yet to speak and a cut-in there leaves nothing of ours to go by. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
reviewed |
There was a problem hiding this comment.
🟡 Changes recommended
Interrupt handling can misclassify timed-out replies as caller-interrupted and incorrectly treat queued speech the caller never heard as delivered in conversation history.
2 open issues
What changed in this PR
This PR adds history rewriting to the Twilio conversation flow so interrupted streaming replies can record an unsent remainder without changing the text passed to speech playback.
Changes:
- Adds a
RewriteReplycallback for adjusting stored replies. - Applies history rewrites while preserving the original spoken response.
- Tracks mid-reply interruptions and builds annotations from streamed text.
| File | Description |
|---|---|
src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioConversationProcessor.cs |
Applies history rewrites while preserving spoken content. |
src/Plugins/BotSharp.Plugin.Twilio/Services/ConversationRelaySession.cs |
Tracks interruptions and annotates unsent reply text. |
src/Plugins/BotSharp.Plugin.Twilio/Models/AgentTurnCallbacks.cs |
Defines the optional reply-rewriting callback. |
Review effort: Balanced
| @@ -235,6 +236,7 @@ private async Task HandleMessageAsync(string json) | |||
| if (_isProcessing && _streamedThisTurn) | |||
There was a problem hiding this comment.
Fixed in #1474: an interrupt no longer marks the turn once its stream was already dropped (the turn timed out and the apology is playing) or the session ended, so the orphaned reply is stored as it is.
🤖 Addressed by Claude Code
| { | ||
| return content; | ||
| } | ||
| sent = (_utteranceOpen ? _streamSegment.ToString() : _lastStreamSegment).Trim(); |
There was a problem hiding this comment.
Fixed in #1474: the interrupt's utteranceUntilInterrupt is kept, and the mark cuts there when it is the start of what went out; when it is missing or doesn't match, the cut falls back to where the sending stopped (and that is logged).
🤖 Addressed by Claude Code
…er a timeout Addresses the review on #1467. - Text goes out faster than it is spoken, so what was sent when the caller cut in is more than they heard. The interrupt's utteranceUntilInterrupt says where playback stopped; the history mark now cuts there when it is the start of what went out, and where the sending stopped otherwise. - An interrupt once the stream was already dropped (the turn timed out and the apology is playing) or the session ended no longer marks the orphaned turn's reply as cut in on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The problem
ConversationRelay stops playback where the caller cuts in, but the whole reply still went into the conversation history. The next turn then believed the question had been asked in full, so the part the caller never heard was never asked again.
What this does
A reply cut off while it was still being streamed is kept in the history as the text that actually went out, with the rest named in a mark:
AgentTurnCallbacks.RewriteReplyis the flow's last say over what the history keeps of the reply.TwilioConversationProcessorcalls it from theonResponseReceivedcallback.ConversationService.SendMessageappends the dialog after that callback, so the message changed there is the one stored.AssistantMessage.Contentkeeps the reply as the model wrote it, so what is spoken does not change and the mark can never reach TTS.ConversationRelaySession.MarkWhatTheCallerCutOffbuilds it from_streamSegment/_lastStreamSegment, which hold exactly what was sent to Twilio. A reply that does not start with what was streamed (text set by a tool, or a reply rewritten after the model wrote it) is left as it is, asGetUnstreamedTextalready does for the same case._cutInMidReplyflag rather than_dropStream, whichFailAsyncalso sets: an orphaned turn that timed out still reaches the callback when it finishes, and must not be recorded as something the caller interrupted.What it does not cover
Only a reply cut off mid-stream can be marked. Once the whole reply is out, Twilio holds the text it has yet to speak, and a cut-in there leaves nothing on our side to go by; placing that cut would mean estimating from
durationUntilInterruptMsand rewriting a message that is already stored.Testing
The behaviour is covered downstream, where the relay session is driven with a fake WebSocket: a reply cut in on mid-stream is stored as the heard part plus the mark, and a reply the caller lets finish is stored unchanged. There is no test project here that covers the Twilio plugin.
🤖 Generated with Claude Code