feat(clients): event sound effects + optional text-to-speech

Add audible cues and optional spoken announcements for session events
(join/leave, channel + PM sent/recv, login, logout/connection-lost,
mic on/off, voice-activity, PTT) across all three clients, driven off
the shared C ABI vc_event stream so the mapping stays consistent.

TTS is off by default; when enabled it announces events and reads
message/PM bodies aloud. Master toggles + a sound-volume slider; the
per-utterance voice-activity and PTT cues default off. WAVs ship from
assets/sounds/.

Windows (built + verified): new VoiceCat.App/Notifications/ layer
(FeedbackSettings -> %AppData%\VoiceCat\feedback.json, SoundPlayerPool
via System.Media.SoundPlayer, SpeechAnnouncer via Prismatoid 0.3.0,
EventFeedback dispatcher); MainForm hooks; NotificationSettingsForm
under Settings > Notifications; csproj adds the Prismatoid PackageRef
and copies the WAVs into sounds\.

macOS + iOS (written, not yet built -- needs a Mac): shared
VoiceCatCore/Feedback/ (SoundEvent, EventFeedback = AVAudioPlayer pool
+ native AVSpeechSynthesizer, FeedbackSettings over UserDefaults); WAVs
bundled via Package.swift resources (.process). Hooks in SessionState/
AppState (iOS) and MainWindowController (macOS); settings UI in
SettingsView (iOS) and SettingsWindowController (macOS).

No core/server code touched; ctest --preset dev unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-22 15:20:11 +02:00
parent 725bd8e925
commit 50416c33a2
45 changed files with 812 additions and 19 deletions

View File

@@ -10,6 +10,36 @@ up instantly. Newest status at the top.
## ▶ Where we left off / next action
- **Windows done / Apple awaiting Mac build (2026-06-22):** **Event sound effects + optional
text-to-speech for all clients.** Clients now play a cue per session event and can optionally
speak it (TTS off by default; when on it announces joins/leaves and reads message/PM bodies).
One canonical event→sound mapping (defined off the shared C ABI `vc_event` stream) is mirrored
across all three clients; `self` vs others is `user_id == self_user_id`, and outgoing messages
echo back as events so sent/recv cues need no separate send-path hook. Conservative defaults
(join/leave, channel/PM sent+recv, login, logout, connection-lost, mic on/off ON; per-utterance
self voice-activity `va_start/va_stop` and the PTT cue OFF). WAVs ship from `assets/sounds/`.
- **Windows (built + verified):** new `VoiceCat.App/Notifications/` (`FeedbackSettings`
`%AppData%\VoiceCat\feedback.json`, `SoundPlayerPool` via `System.Media.SoundPlayer`,
`SpeechAnnouncer` via the **Prismatoid** NuGet 0.3.0, `EventFeedback` dispatcher); hooks in
`Forms/MainForm.cs`; `Forms/NotificationSettingsForm.cs` under a new **Settings ▸ Notifications**
menu. `.csproj` adds the Prismatoid PackageRef and copies the WAVs into `sounds\`. `dotnet build`
clean; WAVs + `Prismatoid.dll` confirmed in output. Note: `SoundPlayer` has no gain control, so
volume is honoured as a mute gate (0 = silent) — swap to NAudio if finer/overlap control is needed.
- **macOS + iOS (written, NOT yet built — needs a Mac):** shared `Sources/VoiceCatCore/Feedback/`
(`SoundEvent`, `EventFeedback` = `AVAudioPlayer` pool + native `AVSpeechSynthesizer`,
`FeedbackSettings` over `UserDefaults`); WAVs copied into `Sources/VoiceCatCore/Sounds/` and
bundled via `Package.swift` `resources: [.process("Sounds")]` (`Bundle.module`). Hooks: iOS
`SessionState.handleEvent` (+ split `userJoined`/`userLeft`, added a `.disconnected` cue case),
`AppState` auth-success login cue, PTT cue in `setPushToTalk`; macOS `MainWindowController`
handlers + NSEvent PTT monitor. Settings UI: iOS `SettingsView` Notifications section
(`@AppStorage`), macOS `SettingsWindowController` checkboxes + volume slider. No `.pbxproj`
edits needed (shared files are SPM-managed; app files already in the projects).
- **Next:** on a Mac, `clients/apple/scripts/build-xcframework.sh --all` then build
VoiceCatMac/VoiceCatiOS; fix compile fallout. **Watch the iOS audio session:** cues/TTS play over
the live VPIO `playAndRecord` session — verify they mix and don't duck/interrupt the call or get
silenced by the mute switch (most likely bug site). Then run `ctest --preset dev` (unchanged —
no core/server code touched).
- **Done (2026-06-22):** **UDP media now shares the TCP port (self-host port-forward fix).** Symptom: a
remote self-hosted server (`iamtalon.me:8384`, TCP+UDP 8384 forwarded) accepted TCP connections but
passed no voice. Root cause: `Config::media_port` defaulted to `0` = OS-assigned, and `main.cpp`'s