Audio (Monitor & Devices)
Shiny.Audio is the standalone audio backbone under Shiny.Speech — usable on its own for
recording and playback without the speech stack. Alongside capture (IAudioSource) and concurrent
playback (IAudioPlayer) — both with normalized VU metering — it provides a live microphone monitor
(IAudioMonitor) and audio route enumeration/selection (IAudioDevices), all discoverable
through a single IAudio facade.
Registration
Section titled “Registration”builder .UseMauiApp<App>() .UseShiny(); // required on Android for RECORD_AUDIO permission routing
builder.Services.AddAudioServices(); // IAudioPlayer + IAudioSource + IAudioMonitor + IAudioDevices + IAudioRecorder + IAudioDuplex + IAudioAddSpeechServices() already registers the capture/playback pieces; call AddAudioServices() when
you also want the monitor/devices/facade. The individual registrations still exist
(AddAudioSource(), AddAudioPlayer(), AddAudioMonitor(), AddAudioDevices(), AddAudioDuplex()), and every focused
interface remains injectable directly — the facade doesn’t replace them.
The IAudio facade
Section titled “The IAudio facade”Inject one service to discover the whole surface. Each member is the same service you could inject
directly; lifetimes are preserved (player/monitor/devices singletons, IAudioSource a fresh
transient per access).
using Shiny.Audio;
public class AudioViewModel(IAudio audio){ Task Play() => audio.Player.PlayAsync("https://example.com/clip.mp3"); Task StartMic() => audio.Monitor.Start(); var Outputs => audio.Devices.GetOutputs();}Monitor and Devices are implemented on iOS / Mac Catalyst and Android only, and Duplex on
iOS, Mac Catalyst, macOS and Android; accessing them on other platforms throws
PlatformNotSupportedException.
Microphone monitor (IAudioMonitor)
Section titled “Microphone monitor (IAudioMonitor)”Routes the microphone straight to the current audio output in near-real-time — a PA / megaphone. Ideal for “talk into the phone, hear it over a Bluetooth speaker”.
using Shiny.Audio;
public partial class MicViewModel(IAudio audio) : ObservableObject{ [ObservableProperty] double level; // bind to a VU bar
public async Task Toggle() { var monitor = audio.Monitor; if (monitor.IsMonitoring) { await monitor.Stop(); return; }
if (await monitor.RequestAccess() != AccessState.Available) return;
monitor.InputLevelChanged += (_, l) => MainThread.BeginInvokeOnMainThread(() => Level = l);
// No Processing → routes to a Bluetooth A2DP speaker (see the trade-off below). await monitor.Start(new AudioMonitorOptions { Gain = 1.0 }); }}The output goes to the current route — the **built-in speaker, a wired output, or a connected
Bluetooth speaker (A2DP) using the phone’s own mic. Gain (0.0–1.0) is adjustable live;
InputLevelChanged emits a normalized level for a meter.
Implementation: iOS/Mac Catalyst use AVAudioEngine (input node → main mixer → output) in a
PlayAndRecord session (no DefaultToSpeaker, so it follows a Bluetooth/wired route; Default mode
unless processing is requested); the engine rebuilds on route changes so monitoring follows a
newly-connected output. Android uses AudioRecord → AudioTrack. The audio session is snapshotted
and restored on Stop.
AirPlay (HomePod / Apple TV) is not supported for a live mic — iOS only permits AirPlay for playback, not while recording. Use Bluetooth for a wireless live PA.
Capturing for a model, not a listener
Section titled “Capturing for a model, not a listener”If the captured audio feeds a model — speaker recognition/verification, wake words, anything that
produces an embedding — capture it with AudioProcessingOptions.Analysis:
var pcm = await source.StartCaptureAsync(AudioProcessingOptions.Analysis, ct);That is no effects plus AllowBluetooth = false, and both halves matter:
- AEC / noise suppression / AGC are adaptive and non-linear. They exist to normalize away speaker and channel characteristics — exactly what a speaker embedding encodes. Because they adapt per session, two recordings of one person come out different.
- A Bluetooth mic runs over HFP at 8 kHz narrowband. Fricatives (/s/, /ʃ/, /f/) live above that, so the bandwidth you actually record depends on what happens to be paired.
On iOS/Mac Catalyst, requesting no effects puts the session in Measurement mode (minimum system
input processing) rather than VoiceChat; AllowBluetooth = false additionally drops the Bluetooth
options from the session category. Android’s raw mic source and Windows capture never route to a
Bluetooth mic implicitly, so Analysis is a no-op there.
None of this applies to transcription — for STT, the effects are usually what you want.
Microphone metering (IAudioSource)
Section titled “Microphone metering (IAudioSource)”Capture raises InputLevelChanged with a normalized 0.0–1.0 level while it streams PCM — a mic
VU meter for a record button, a “we’re hearing you” indicator, or a silence detector. Unlike the
playback meter it needs no capability check: it’s computed from the captured PCM itself, so it works
on every platform, including the browser.
using Shiny.Audio;
public partial class RecorderViewModel(IAudio audio) : ObservableObject{ [ObservableProperty] double level; // bind to a VU bar
public async Task Record(CancellationToken ct) { var source = audio.Source; if (await source.RequestAccess() != AccessState.Available) return;
source.InputLevelChanged += (_, l) => MainThread.BeginInvokeOnMainThread(() => Level = l);
var pcm = await source.StartCaptureAsync(cancellationToken: ct); // consume the 16 kHz / 16-bit / mono stream... await source.StopCaptureAsync(); Level = 0; }}Events are raised on the capture thread and throttled to roughly 20/sec (peak-held in between), so a bound bar stays smooth without flooding the UI thread. If you’d rather meter the PCM yourself — say you’re buffering it elsewhere — the same mapping is public:
var level = AudioLevel.FromPcm16(buffer.AsSpan(0, bytesRead)); // 0.0 - 1.0AudioLevel maps linear RMS to dBFS against a -50 dB noise floor. Every meter in the library goes
through it, so mic and playback bars read on the same scale.
Audio devices (IAudioDevices)
Section titled “Audio devices (IAudioDevices)”Its most reliable use is displaying where audio is flowing and reacting to route changes — e.g.
“Output: JBL Flip · Bluetooth”. Read CurrentInput / CurrentOutput (each an AudioDevice with a
normalized Type) and subscribe to Changed:
using Shiny.Audio;
var devices = audio.Devices;
string input = devices.CurrentInput?.Name ?? "—"; // .Type: BuiltInMic, Bluetooth, …string output = devices.CurrentOutput?.Name ?? "—"; // .Type: BuiltInSpeaker, BluetoothA2dp, …
devices.Changed += (_, _) => Refresh(); // route added / removed / switchedEach AudioDevice carries Id, Name, Io (input/output), a normalized Type (BuiltInMic,
BluetoothA2dp, WiredHeadphones, BuiltInSpeaker, …) and IsCurrent.
Classifying the route — wired, Bluetooth, or built-in
Section titled “Classifying the route — wired, Bluetooth, or built-in”AudioDeviceType covers wired routes as well as Bluetooth: WiredHeadphones (output only) and
WiredHeadset (output plus a mic). Both map on each platform — iOS from
AVAudioSession.PortHeadphones / PortHeadsetMic, Android from WiredHeadphones / WiredHeadset.
Rather than matching every enum member by hand, use the classification helpers. Each exists on both
AudioDevice and AudioDeviceType:
using Shiny.Audio;
var output = audio.Devices.CurrentOutput;
if (output?.IsWired() == true) { /* 3.5mm jack, Lightning, or USB-C */ }if (output?.IsBluetooth() == true) { /* HFP/SCO or A2DP */ }if (output?.IsBuiltIn() == true) { /* phone speaker or earpiece — nothing attached */ }
// "Is audio private to the user?" — e.g. before speaking a TTS reply out loud.if (output?.IsHeadphones() == true) await audio.Player.PlayAsync(replyUrl);
// Will capture come from the accessory, or fall back to the phone's own mic?bool accessoryMic = output?.HasMicrophone() == true;| Helper | True for |
|---|---|
IsWired() |
WiredHeadphones, WiredHeadset, Usb |
IsBluetooth() |
Bluetooth (HFP/SCO), BluetoothA2dp |
IsBuiltIn() |
BuiltInMic, BuiltInSpeaker, BuiltInReceiver |
IsHeadphones() |
WiredHeadphones, WiredHeadset, Bluetooth, BluetoothA2dp |
HasMicrophone() |
WiredHeadset, Bluetooth |
Combine with Changed to react to plug/unplug — it fires on both platforms (iOS route-change
notification, Android device callback).
Selection (GetInputs/GetOutputs + IAudioMonitor.SetInputDevice/SetOutputDevice) is a full
selector on Android only:
await audio.Monitor.SetInputDevice(chosenInput); // Android + iOSawait audio.Monitor.SetOutputDevice(chosenOutput); // Android only (best-effort elsewhere)Concurrent playback (IAudioPlayer / IAudioPlayback)
Section titled “Concurrent playback (IAudioPlayer / IAudioPlayback)”Clips play concurrently. Starting one does not stop the others, so background music, a sound effect and a voice line can overlap — and each one is still cancellable on its own.
StartAsync returns an IAudioPlayback handle as soon as the clip is playing; PlayAsync is the
same thing plus a wait on that handle’s Completion, so it waits for its clip without touching
anything else.
using Shiny.Audio;
public class SoundboardViewModel(IAudioPlayer player){ IAudioPlayback? music;
public async Task StartMusic(CancellationToken ct) => this.music = await player.StartAsync("https://example.com/theme.mp3", ct);
// Overlaps the music rather than interrupting it public Task Ping(CancellationToken ct) => player.PlayAsync(Path.Combine(FileSystem.AppDataDirectory, "ping.mp3"), ct);
public Task StopMusic() => this.music?.StopAsync() ?? Task.CompletedTask; // effects play on public Task StopEverything() => player.StopAsync();}| Member | Meaning |
|---|---|
IAudioPlayer.StartAsync(stream | source, ct) |
Start a clip, return its IAudioPlayback |
IAudioPlayer.PlayAsync(stream | source, ct) |
StartAsync + await Completion — waits for this clip only |
IAudioPlayer.Active |
Everything currently playing, oldest first |
IAudioPlayer.StopAsync() |
Stops every clip this player started |
IAudioPlayback.StopAsync() |
Stops one clip; the rest keep playing |
IAudioPlayback.Completion |
Completes when that clip ends, stops, or is cancelled |
IAudioPlayback.Id / Source / IsPlaying |
Identity, the URL/path (null for a stream), live state |
A CancellationToken passed to StartAsync/PlayAsync stops that clip only, and completes it
normally — Completion does not throw OperationCanceledException. IAudioPlayback is
IAsyncDisposable, so disposing a handle stops its clip.
AudioLevelChanged stays a single stream per player: while clips overlap it reports the loudest
of them, which is what a VU meter should show.
Volume (IAudioPlayer)
Section titled “Volume (IAudioPlayer)”IAudioPlayer exposes the device media volume — the level the hardware buttons control, normalized
0.0–1.0 and independent of any per-request TTS Volume. Reading works on every platform; setting
is platform-limited — always guard a set with IsVolumeControlSupported.
using Shiny.Audio;
public partial class VolumeViewModel(IAudio audio) : ObservableObject{ [ObservableProperty] float volume; // bind to a Slider public bool CanSet => audio.Player.IsVolumeControlSupported;
public VolumeViewModel(IAudio audio) : this(audio) { Volume = audio.Player.Volume; // reading works everywhere
audio.Player.VolumeChanged += (_, v) => // hardware buttons, OS UI, or a successful set MainThread.BeginInvokeOnMainThread(() => Volume = v); }
partial void OnVolumeChanged(float value) { if (audio.Player.IsVolumeControlSupported) // iOS / Mac Catalyst setter throws audio.Player.Volume = value; }}| Platform | Read | Set | VolumeChanged source |
Backing API |
|---|---|---|---|---|
| Android | ✅ | ✅ | system settings observer | AudioManager STREAM_MUSIC (integer-stepped → set values quantize) |
| Windows | ✅ | ✅ | endpoint volume callback | WASAPI IAudioEndpointVolume (default render endpoint) |
| macOS | ✅ | ✅ † | CoreAudio property listener | HAL virtual main volume of the default output device |
| iOS / Mac Catalyst | ✅ | ❌ | KVO on outputVolume |
AVAudioSession.OutputVolume (read-only) |
| Browser (WASM) | ✅ | ✅ | echoed on set | HTMLAudioElement.volume — app-local, not the OS volume |
Full-duplex voice for calls (IAudioDuplex)
Section titled “Full-duplex voice for calls (IAudioDuplex)”IAudioDuplex runs microphone capture and live PCM playback on one voice-processing session, so
the platform’s echo canceller removes what the speaker plays from what the microphone hears. It’s the
audio half of a phone / FaceTime-style call, where PCM frames arrive from the network every ~20 ms and
must play straight away.
using Shiny;using Shiny.Audio;
public class CallAudio(IAudioDuplex duplex) // or IAudio.Duplex{ IAudioDuplexSession? session;
public async Task Start(Func<ReadOnlyMemory<byte>, CancellationToken, ValueTask> send, CancellationToken ct) { if (await duplex.RequestAccess() != AccessState.Available) return;
session = await duplex.StartAsync(new AudioDuplexOptions { SampleRate = 16000, // or 8000 Speakerphone = true // loudspeaker rather than the earpiece }, ct);
// Mic → network: one 20 ms PCM16 mono frame per item, already echo-cancelled. _ = Task.Run(async () => { await foreach (var frame in session.Captured.ReadAllAsync(ct)) await send(frame, ct); }, ct); }
// Network → speaker: any thread, any size, never blocks. public void OnReceived(ReadOnlyMemory<byte> pcm16) => session?.Play(pcm16.Span);
public void ToggleSpeaker() => session!.Speakerphone = !session.Speakerphone; // live
// A mic from IAudioDevices.GetInputs(), or null for the system default. Live. public void UseMic(AudioDevice? mic) => session!.InputDevice = mic;
public Task Stop() => session?.StopAsync() ?? Task.CompletedTask; // restores the audio session}AudioDuplexOptions |
Default | |
|---|---|---|
SampleRate |
16000 |
8000 or 16000, both directions |
Processing |
AudioProcessingOptions.VoiceChat |
Echo cancellation, noise suppression, AGC. Null / nothing enabled = no echo cancellation |
Speakerphone |
true |
Loudspeaker rather than the earpiece. A connected headset still wins |
InputDevice |
null |
The microphone, one of IAudioDevices.GetInputs(). Null is the system default. Changeable live |
FrameDuration |
20 ms | The size of each Captured item: whole milliseconds, 10–100 |
PlaybackLatency |
60 ms | The jitter-buffer target |
MaxPlaybackLatency |
200 ms | The most that may queue; must be greater than PlaybackLatency |
The playback jitter buffer
Section titled “The playback jitter buffer”Play hands audio to a jitter buffer that the output pulls at the hardware rate:
- Priming. Nothing plays until about
PlaybackLatencyis queued, so the first late packet doesn’t starve the output straight away. - Underruns. When the queue runs dry, whatever there is plays, then silence, and it primes again before playing more. Lost and late packets come out as silence, never as a stall.
- Overflow. When a burst takes the queue past
MaxPlaybackLatency, the oldest audio is dropped back down toPlaybackLatency, so delay can’t build up over a long call. Bufferedreports how much is queued.
The other direction is bounded too: Captured holds about a second of frames, and a reader that falls
behind loses the oldest. It completes when the session stops.
Choosing the microphone
Section titled “Choosing the microphone”AudioDuplexOptions.InputDevice picks the mic the call starts on, and IAudioDuplexSession.InputDevice
switches it mid-call without stopping capture or playback:
var inputs = audio.Devices.GetInputs();session.InputDevice = inputs.FirstOrDefault(d => d.Type == AudioDeviceType.Bluetooth); // null = default
session.InputDeviceChanged += (_, _) =>{ // The chosen mic was unplugged or disconnected: the call is back on the default mic, // and session.InputDevice is null. Raised on a platform thread.};- Null is the system default. An output device throws
ArgumentException. A mic that isn’t connected falls back to the default, andInputDevicethen readsnull. - A headset’s mic and speaker move together on iOS, Mac Catalyst and Android. Choosing a Bluetooth,
wired or USB headset mic routes the call’s audio to that headset as well, and
Speakerphonehas no effect while it’s chosen. Choosing the built-in mic while a headset is connected takes the call off the headset, to the loudspeaker or the earpiece perSpeakerphone. Withnull, a connected headset wins, as before. InputDeviceChangedfires only when the platform changes the mic underneath you, never for your own sets.- How long a switch takes is the platform’s. iOS reroutes the session and the engine rebuilds after
the route change, a short blip. macOS restarts its voice-processing engine on the new device, a gap of a
few tens of milliseconds; queued playback is kept. Android reroutes the running
AudioRecordin place.
Platforms
Section titled “Platforms”| Platform | How it’s built |
|---|---|
| iOS / Mac Catalyst | AVAudioSession PlayAndRecord in VoiceChat mode (DefaultToSpeaker for speakerphone, HFP Bluetooth when AllowBluetooth), one AVAudioEngine with voice processing on its input node, a tap resampled by AVAudioConverter, and an AVAudioPlayerNode fed from the jitter buffer. The engine is rebuilt after a route change. InputDevice is the session’s preferred input (setPreferredInput) |
| macOS | The same engine, without AVAudioSession. Speakerphone has no effect. InputDevice points the voice-processing unit at that CoreAudio device; only the input changes, since macOS output is a separate device |
| Android | AudioRecord (VOICE_COMMUNICATION) and a streaming AudioTrack (USAGE_VOICE_COMMUNICATION, CONTENT_TYPE_SPEECH) on one audio session, with AcousticEchoCanceler, NoiseSuppressor and AutomaticGainControl when the device has them. MODE_IN_COMMUNICATION and transient audio focus while running; speakerphone through setCommunicationDevice (API 31+) or setSpeakerphoneOn. InputDevice is the record’s setPreferredDevice, and it steers the communication device as described above. Below API 31 the session doesn’t start a Bluetooth SCO link, so a Bluetooth headset mic may not be reachable there. Add MODIFY_AUDIO_SETTINGS to the manifest |
| Windows, Linux, Browser | Not available |
- One session at a time. A second
StartAsyncthrowsInvalidOperationExceptionuntil the first stops. Invalid options throwArgumentException. - Stopping restores what the session changed: the audio session category and mode on Apple; the audio mode, communication device and audio focus on Android. Disposing the session stops it.
- Echo-cancellation quality belongs to the platform and the device. Without processing, use a headset.
Linux support ships in a separate Shiny.Audio.Linux package. It implements all four services
over PulseAudio/PipeWire, falling back to ALSA where no sound server is running — headless
servers, minimal containers, Raspberry Pi images.
It is a separate package because it carries a managed MP3 decoder (NLayer): Linux has no system
media decoder to hand a stream to, unlike MediaPlayer on Windows/Android or AVAudioPlayer on
Apple.
dotnet add package Shiny.Audio.Linuxusing Shiny;
// Registers IAudioSource + IAudioPlayer + IAudioDevices + IAudioMonitor + IAudio.// A no-op when not running on Linux, so it is safe in shared startup code.builder.Services.AddLinuxAudio();There is no native speech on Linux
Section titled “There is no native speech on Linux”Linux has no OS speech engine to wrap the way iOS, Android and Windows do, so AddSpeechToText()
and AddTextToSpeech() register nothing there. Pair AddLinuxAudio() with any cloud provider —
they are pure HTTP over the PCM stream and work unchanged:
builder.Services.AddLinuxAudio();builder.Services.AddAzureSpeech("subscription-key", "region");Behaviour notes
Section titled “Behaviour notes”- No resampler ships in the library. Streams open at exactly the format required — 16 kHz mono
for capture, a decoded clip’s own rate for playback. PulseAudio converts server-side; ALSA’s
defaultdevice converts through itsplugplugin. - Playback decodes MP3 and WAV only. That covers what Azure, OpenAI and ElevenLabs return by
default. Any other container throws
NotSupportedExceptionnaming the format. - Playback metering works here (unlike Windows/Browser) because the player decodes to PCM in managed code and sees every sample.
- Voice processing is best-effort. PulseAudio publishes echo cancellation as a virtual source
from
module-echo-cancel; when that module is loaded,AudioProcessingOptions.EchoCancellationselects it, otherwise capture is raw. - No permission prompt. Microphone access on Linux is a filesystem/group concern, so
RequestAccess()returnsAvailablewhenever a backend is present. ShowOutputPicker()is a no-op — there is no system route picker equivalent to iOS’sAVRoutePickerView; routing lives in the desktop environment’s sound settings.IAudioDevices.Changedfires on PulseAudio/PipeWire only, since ALSA has no change notification.
Runtime dependencies
Section titled “Runtime dependencies”The distro’s own audio libraries — libpulse-simple.so.0 (usually the libpulse0 package) or
libasound.so.2 (usually libasound2). In a container, install one of them explicitly.
Probe at startup when the app must degrade gracefully rather than throw on first use:
if (!LinuxAudioServiceCollectionExtensions.IsLinuxAudioAvailable) logger.LogWarning("No Linux audio backend found — voice features disabled");Platform support
Section titled “Platform support”| Capability | iOS / Mac Catalyst | Android | Windows | Browser | Linux |
|---|---|---|---|---|---|
IAudioPlayer / IAudioSource |
✅ | ✅ | ✅ | ✅ | ✅ |
IAudioSource.InputLevelChanged (mic VU) |
✅ | ✅ | ✅ | ✅ | ✅ |
IAudioPlayer.AudioLevelChanged (playback VU) |
✅ | ✅ | ❌ | ❌ | ✅ |
IAudioPlayer.Volume (read) |
✅ | ✅ | ✅ | ✅ | ✅ |
IAudioPlayer.Volume (set) |
❌ ¹ | ✅ | ✅ | ✅ | ✅ ² |
IAudioMonitor (live monitor) |
✅ | ✅ | ❌ | ❌ | ✅ |
IAudioDevices (routes) |
✅ | ✅ | ❌ | ❌ | ✅ |
IAudioDuplex (call audio) |
✅ ³ | ✅ | ❌ | ❌ | ❌ |
¹ Native macOS (not Mac Catalyst) can set the volume via CoreAudio — see the Volume section.
² Linux volume is settable on PulseAudio/PipeWire (the default sink’s volume). On the ALSA
fallback IsVolumeControlSupported is false and the setter throws — reading returns the last value
set in-process rather than a real device level. Guard the set, exactly as on iOS.
³ Native macOS too.


