feat(clients): event sound effects + optional text-to-speech
Add audible cues and optional spoken announcements for session events (join/leave, channel + PM sent/recv, login, logout/connection-lost, mic on/off, voice-activity, PTT) across all three clients, driven off the shared C ABI vc_event stream so the mapping stays consistent. TTS is off by default; when enabled it announces events and reads message/PM bodies aloud. Master toggles + a sound-volume slider; the per-utterance voice-activity and PTT cues default off. WAVs ship from assets/sounds/. Windows (built + verified): new VoiceCat.App/Notifications/ layer (FeedbackSettings -> %AppData%\VoiceCat\feedback.json, SoundPlayerPool via System.Media.SoundPlayer, SpeechAnnouncer via Prismatoid 0.3.0, EventFeedback dispatcher); MainForm hooks; NotificationSettingsForm under Settings > Notifications; csproj adds the Prismatoid PackageRef and copies the WAVs into sounds\. macOS + iOS (written, not yet built -- needs a Mac): shared VoiceCatCore/Feedback/ (SoundEvent, EventFeedback = AVAudioPlayer pool + native AVSpeechSynthesizer, FeedbackSettings over UserDefaults); WAVs bundled via Package.swift resources (.process). Hooks in SessionState/ AppState (iOS) and MainWindowController (macOS); settings UI in SettingsView (iOS) and SettingsWindowController (macOS). No core/server code touched; ctest --preset dev unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
30
PROGRESS.md
30
PROGRESS.md
@@ -10,6 +10,36 @@ up instantly. Newest status at the top.
|
||||
|
||||
## ▶ Where we left off / next action
|
||||
|
||||
- **Windows done / Apple awaiting Mac build (2026-06-22):** **Event sound effects + optional
|
||||
text-to-speech for all clients.** Clients now play a cue per session event and can optionally
|
||||
speak it (TTS off by default; when on it announces joins/leaves and reads message/PM bodies).
|
||||
One canonical event→sound mapping (defined off the shared C ABI `vc_event` stream) is mirrored
|
||||
across all three clients; `self` vs others is `user_id == self_user_id`, and outgoing messages
|
||||
echo back as events so sent/recv cues need no separate send-path hook. Conservative defaults
|
||||
(join/leave, channel/PM sent+recv, login, logout, connection-lost, mic on/off ON; per-utterance
|
||||
self voice-activity `va_start/va_stop` and the PTT cue OFF). WAVs ship from `assets/sounds/`.
|
||||
- **Windows (built + verified):** new `VoiceCat.App/Notifications/` (`FeedbackSettings` →
|
||||
`%AppData%\VoiceCat\feedback.json`, `SoundPlayerPool` via `System.Media.SoundPlayer`,
|
||||
`SpeechAnnouncer` via the **Prismatoid** NuGet 0.3.0, `EventFeedback` dispatcher); hooks in
|
||||
`Forms/MainForm.cs`; `Forms/NotificationSettingsForm.cs` under a new **Settings ▸ Notifications**
|
||||
menu. `.csproj` adds the Prismatoid PackageRef and copies the WAVs into `sounds\`. `dotnet build`
|
||||
clean; WAVs + `Prismatoid.dll` confirmed in output. Note: `SoundPlayer` has no gain control, so
|
||||
volume is honoured as a mute gate (0 = silent) — swap to NAudio if finer/overlap control is needed.
|
||||
- **macOS + iOS (written, NOT yet built — needs a Mac):** shared `Sources/VoiceCatCore/Feedback/`
|
||||
(`SoundEvent`, `EventFeedback` = `AVAudioPlayer` pool + native `AVSpeechSynthesizer`,
|
||||
`FeedbackSettings` over `UserDefaults`); WAVs copied into `Sources/VoiceCatCore/Sounds/` and
|
||||
bundled via `Package.swift` `resources: [.process("Sounds")]` (`Bundle.module`). Hooks: iOS
|
||||
`SessionState.handleEvent` (+ split `userJoined`/`userLeft`, added a `.disconnected` cue case),
|
||||
`AppState` auth-success login cue, PTT cue in `setPushToTalk`; macOS `MainWindowController`
|
||||
handlers + NSEvent PTT monitor. Settings UI: iOS `SettingsView` Notifications section
|
||||
(`@AppStorage`), macOS `SettingsWindowController` checkboxes + volume slider. No `.pbxproj`
|
||||
edits needed (shared files are SPM-managed; app files already in the projects).
|
||||
- **Next:** on a Mac, `clients/apple/scripts/build-xcframework.sh --all` then build
|
||||
VoiceCatMac/VoiceCatiOS; fix compile fallout. **Watch the iOS audio session:** cues/TTS play over
|
||||
the live VPIO `playAndRecord` session — verify they mix and don't duck/interrupt the call or get
|
||||
silenced by the mute switch (most likely bug site). Then run `ctest --preset dev` (unchanged —
|
||||
no core/server code touched).
|
||||
|
||||
- **Done (2026-06-22):** **UDP media now shares the TCP port (self-host port-forward fix).** Symptom: a
|
||||
remote self-hosted server (`iamtalon.me:8384`, TCP+UDP 8384 forwarded) accepted TCP connections but
|
||||
passed no voice. Root cause: `Config::media_port` defaulted to `0` = OS-assigned, and `main.cpp`'s
|
||||
|
||||
Reference in New Issue
Block a user