Client v5: BLE, BLE Hosting, HTTP, Jobs - Linux, MacOS, & Blazor Support! Full AOT, RX on BLE only & MANY other features! Power up!

Getting Started


GitHub
Core
Cloud
Azure
OpenAI
ElevenLabs
Typecast
Microsoft.Extensions.AI

Frameworks

.NET

.NET MAUI

Operating Systems

Android

iOS

Windows

Web

Shiny.Speech provides a unified API for speech-to-text, text-to-speech, audio capture, and audio playback across Android, iOS, Windows, and Browser (Blazor WebAssembly) — with pluggable cloud providers for Azure AI Speech, OpenAI, ElevenLabs, and Typecast.

Features

Event-based speech-to-text with Start/Stop lifecycle — multiple subscribers supported
Built-in keyword detection via SpeechRecognitionOptions.Keywords and the KeywordHeard event
Platform-native text-to-speech with voice selection, rate, pitch, and volume control
Raw audio capture from the device microphone (16kHz, 16-bit, mono PCM)
Voice processing on capture — echo cancellation, noise suppression, and automatic gain control via AudioProcessingOptions (cancels TTS bleed for barge-in)
Audio playback for MP3 streams
Pluggable cloud provider architecture — swap between on-device and cloud STT/TTS
Azure AI Speech integration (STT + TTS) with SSML prosody control
OpenAI integration (STT + TTS) powered by Whisper and GPT-4o models
ElevenLabs integration — Scribe speech-to-text and multilingual text-to-speech
Typecast integration — AI voice text-to-speech (TTS only) via the typecast-csharp SDK
Microsoft.Extensions.AI adapter — expose providers as ISpeechToTextClient / ITextToSpeechClient
Runtime credential changes — provider config objects are mutable singletons; rotate API keys / region / model without re-registration
State tracking — IsListening (STT), IsSpeaking (TTS), IsPlaying (audio)
VU meter signal (outgoing) — AudioLevelChanged event on ITextToSpeechService and IAudioPlayer emits a normalized 0.0–1.0 RMS level during playback; IsPlayerAnalysisSupported reports availability per platform
VU meter signal (incoming) — InputLevelChanged event on ISpeechToTextService (gated by IsInputAnalysisSupported), IAudioSource (every platform), and IAudioMonitor emits the same normalized 0.0–1.0 level from the microphone
Permission management via AccessState and RequestAccess()
Convenience extension methods: ListenUntilSilence, StatementAfterKeyword, WaitListenForKeywords, ListenForKeywords
CarPlay compatible — iOS audio routes through the car’s mic/speakers automatically when CarPlay is active

Packages

Package	Purpose
`Shiny.Speech`	Core library — platform-native STT, TTS, audio capture, and playback
`Shiny.Speech.Cloud`	Cloud provider abstractions (included transitively by Azure/ElevenLabs)
`Shiny.Speech.Azure`	Azure AI Speech provider (STT + TTS)
`Shiny.Speech.OpenAI`	OpenAI provider (STT + TTS) — Whisper, GPT-4o Transcribe, GPT-4o Mini TTS
`Shiny.Speech.ElevenLabs`	ElevenLabs provider — Scribe STT + TTS
`Shiny.Speech.Typecast`	Typecast provider (TTS only) — AI voice via the `typecast-csharp` SDK
`Shiny.Speech.MicrosoftAI`	Microsoft.Extensions.AI adapter — `ISpeechToTextClient` / `ITextToSpeechClient`

Setup

Changed files

MyApp/

Platforms/Android/AndroidManifest.xml

1<?xml version="1.0" encoding="utf-8"?>2<manifest xmlns:android="http://schemas.android.com/apk/res/android">3	<application android:allowBackup="true" android:icon="@mipmap/appicon" android:roundIcon="@mipmap/appicon_round" android:supportsRtl="true">4	</application>5	<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>6	<uses-permission android:name="android.permission.BATTERY_STATS" />7	<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />8	<uses-permission android:name="android.permission.INTERNET" />9	<uses-permission android:name="android.permission.RECORD_AUDIO" />10	<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />11</manifest>

UseShiny() is required

Native speech and audio delegate Android permission handling and activity tracking to Shiny.Core’s AndroidPlatform. Reference the Shiny.Hosting.Maui package and call .UseShiny() on your MauiAppBuilder before building:

builder
    .UseMauiApp<App>()
    .UseShiny();

builder.Services.AddSpeechServices();

Without .UseShiny(), RequestAccess() throws a TimeoutException on Android because no AndroidPlatform is registered to receive the permission-result callback. AccessState also comes from Shiny.Core now (namespace Shiny).

Platform Notes

Android’s native speech recognition engine works in short recording segments — it automatically stops listening after a period of silence and must restart for the next segment. This creates brief pauses during continuous listening that are noticeable compared to iOS, which streams continuously.

If smooth, uninterrupted speech recognition is important for your app, use the ElevenLabs cloud provider, which transcribes captured audio without relying on the native engine’s start/stop behavior.

builder.Services.AddElevenLabsSpeechToText("your-api-key");

Avoid Azure on Android: the Azure AI Speech provider does not currently support Android’s 16 KB memory page size. On devices that use 16 KB pages (Android 15+ on supported hardware), the Azure native libraries fail to load — prefer ElevenLabs on Android.

Platform Permissions

Android — Add to AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

MODIFY_AUDIO_SETTINGS is required for the TTS audio-level Visualizer and for the native STT beep suppression.

iOS — Add to Info.plist:

<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone for speech recognition</string>

Windows — Add the Microphone capability to your Package.appxmanifest:

<Capabilities>
    <DeviceCapability Name="microphone" />
</Capabilities>

Browser (Blazor WebAssembly) — No manifest changes and no <script> tag needed. The browser prompts for microphone access automatically, and the JS interop module ships inside the Shiny.Audio package as a static web asset (_content/Shiny.Audio/shiny-audio.js), loaded on demand via JSHost.ImportAsync. Just reference the NuGet package — don’t copy the JS into wwwroot.

IAudioSource captures raw PCM audio in the browser using the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono.

Quick Example

public class MyViewModel
{
    readonly ISpeechToTextService _stt;
    readonly ITextToSpeechService _tts;

    public MyViewModel(ISpeechToTextService stt, ITextToSpeechService tts)
    {
        _stt = stt;
        _tts = tts;
    }

    async Task ListenAndRespond(CancellationToken ct)
    {
        // 1. Request permission
        var access = await _stt.RequestAccess();
        if (access != AccessState.Available)
            return;

        // 2. Listen until the user stops speaking
        var text = await _stt.ListenUntilSilence(
            new SpeechRecognitionOptions
            {
                Culture = CultureInfo.GetCultureInfo("en-US"),
                SilenceTimeout = TimeSpan.FromSeconds(3)
            },
            ct
        );

        if (text != null)
        {
            // 3. Speak the result back
            await _tts.SpeakAsync($"You said: {text}");
        }
    }

    async Task EventBasedListening()
    {
        // Subscribe to events — multiple classes can subscribe simultaneously
        _stt.ResultReceived += (s, result) =>
            Console.WriteLine($"[{(result.IsFinal ? "FINAL" : "partial")}] {result.Text}");

        _stt.KeywordHeard += (s, keyword) =>
            Console.WriteLine($"Keyword: {keyword}");

        // Start listening with keyword detection
        await _stt.Start(new SpeechRecognitionOptions
        {
            Keywords = ["Yes", "No", "Maybe"]
        });

        // Later: stop
        await _stt.Stop();
    }

    async Task WakeWordExample(CancellationToken ct)
    {
        // "Hey Computer, do something" → returns "do something"
        var command = await _stt.StatementAfterKeyword(["Hey Computer"], cancellationToken: ct);

        if (command != null)
            await _tts.SpeakAsync($"You asked: {command}");
    }

    async Task KeywordExample(CancellationToken ct)
    {
        // Wait for a keyword (with optional timeout)
        await _tts.SpeakAsync("Do you agree? Say yes, no, or maybe.");
        var answer = await _stt.WaitListenForKeywords(
            ["Yes", "No", "Maybe"],
            timeout: TimeSpan.FromSeconds(30),
            cancellationToken: ct
        );
    }
}

VU Meters (Audio Levels)

Both directions are metered on the same normalized 0.0–1.0 scale — a “speaking” bar for what comes out of the speaker and a “listening” bar for what goes into the mic.

Outgoing — playback / TTS

ITextToSpeechService and IAudioPlayer raise AudioLevelChanged while audio is playing. Check IsPlayerAnalysisSupported before binding UI — it’s false where metering isn’t available (Windows native TTS, Browser).

Incoming — microphone

ISpeechToTextService raises InputLevelChanged while listening; check IsInputAnalysisSupported. The lower-level IAudioSource.InputLevelChanged (raw capture, supported everywhere, no flag) and IAudioMonitor.InputLevelChanged (live mic-to-output) emit the same signal.

public partial class VoiceViewModel : ObservableObject
{
    [ObservableProperty] double speakingLevel;    // 0.0 .. 1.0
    [ObservableProperty] double listeningLevel;

    public bool IsSpeakingVuSupported { get; }
    public bool IsListeningVuSupported { get; }

    public VoiceViewModel(ITextToSpeechService tts, ISpeechToTextService stt)
    {
        IsSpeakingVuSupported = tts.IsPlayerAnalysisSupported;
        IsListeningVuSupported = stt.IsInputAnalysisSupported;

        tts.AudioLevelChanged += (_, level) =>
            MainThread.BeginInvokeOnMainThread(() => SpeakingLevel = level);

        stt.InputLevelChanged += (_, level) =>
            MainThread.BeginInvokeOnMainThread(() => ListeningLevel = level);
    }
}

<ProgressBar Progress="{Binding SpeakingLevel}"
             IsVisible="{Binding IsSpeakingVuSupported}" />
<ProgressBar Progress="{Binding ListeningLevel}"
             IsVisible="{Binding IsListeningVuSupported}" />

Surface	iOS / macOS	Android	Windows	Browser
Native TTS (`ITextToSpeechService`)	✅ — `AVAudioEngine` + player-node tap	✅ — `OnAudioAvailable` RMS	❌	❌
Cloud TTS (`CloudTextToSpeech`)	✅ — forwarded from `IAudioPlayer`	✅ — forwarded from `IAudioPlayer`	❌	❌
Generic playback (`IAudioPlayer`)	✅ — `AVAudioPlayer.MeteringEnabled`	✅ — `Visualizer` on session	❌	❌
Cloud STT (`CloudSpeechToText`)	✅ — forwarded from `IAudioSource`	✅ — forwarded from `IAudioSource`	✅ — forwarded from `IAudioSource`	✅ — forwarded from `IAudioSource`
Native STT (`ISpeechToTextService`)	✅ — recognizer input-node tap	✅ — `OnRmsChanged`	❌	❌
Capture (`IAudioSource`)	✅	✅	✅	✅
Monitor (`IAudioMonitor`)	✅	✅	n/a	n/a

Cloud recognition meters the IAudioSource feeding the provider, so a mic meter works on every platform. Native recognition depends on what the OS exposes: Apple taps the recognizer’s own input node, Android forwards SpeechRecognizer.OnRmsChanged, while the Windows and Web Speech recognizers own the mic and surface no level at all.

On iOS / macOS the native TTS path routes AVSpeechSynthesizer through AVAudioEngine + AVAudioPlayerNode so a tap can compute RMS. The engine is created lazily and kept warm across utterances — first-utterance latency adds roughly 50–150 ms; subsequent calls are indistinguishable from the legacy direct path.

Level events are raised off the UI thread (capture-side ones are throttled to ~20/sec, peak-held in between) — marshal before mutating bound properties, and reset your bound value to 0 when playback or listening ends so the meter drains.

Voice Processing (Noise Suppression & Echo Cancellation)

Microphone capture can request platform voice-processing effects to strip background noise and, critically, to cancel your text-to-speech output from the mic so it isn’t re-captured while the mic is open (barge-in). Configure it via AudioProcessingOptions — either directly on IAudioSource.StartCaptureAsync(...) or through SpeechRecognitionOptions.AudioProcessing (honored by the cloud providers, which capture through IAudioSource):

await stt.Start(new SpeechRecognitionOptions
{
    Culture = CultureInfo.GetCultureInfo("en-US"),
    AudioProcessing = AudioProcessingOptions.VoiceChat   // AEC + noise suppression + AGC
});

// or, capturing raw audio directly:
var stream = await audioSource.StartCaptureAsync(new AudioProcessingOptions
{
    EchoCancellation = true,      // subtracts speaker/TTS output from the mic signal
    NoiseSuppression = true,      // attenuates steady background noise
    AutomaticGainControl = true   // normalizes capture level
});

Each flag is best-effort and maps to native voice processing:

Effect	iOS / macOS	Android	Windows	Browser
Echo Cancellation	✅ Voice-Processing I/O	✅ `AcousticEchoCanceler` + VoiceCommunication	⚠️ best-effort (Communications pipeline)	✅ WebRTC AEC3
Noise Suppression	✅ (bundled)	✅ `NoiseSuppressor`	⚠️ best-effort	✅
Automatic Gain Control	✅ (bundled)	✅ `AutomaticGainControl`	⚠️ best-effort	✅

Apple bundles all three into a single Voice-Processing I/O unit — enabling any flag enables the whole chain (they can’t be toggled independently).
Android effect availability is device/driver dependent; unavailable effects are skipped. Requesting echo cancellation also routes capture through VoiceCommunication.
Windows AudioGraph exposes no per-effect control; requesting any effect selects the Communications capture category, which engages driver-provided AEC/NS when present.
These are OS/hardware cancellers referencing the real speaker feed, so any device audio (not just library-played TTS) is cancelled.
Native on-device ISpeechToTextService implementations manage their own microphone and are unaffected by this setting — it applies to IAudioSource capture (cloud providers, raw capture).

Changing Credentials at Runtime

Provider config objects are mutable singletons, so you can change the API key (or region / model / voice) at any time — after the user pastes a new key in a settings screen, for example — and the provider picks it up on its next call. No re-registration required. Hold your config instance, or resolve it from DI:

var config = new AzureSpeechConfig { SubscriptionKey = "initial-key", Region = "eastus" };
builder.Services.AddAzureSpeech(config);

// ...later, e.g. after the user rotates the key in settings:
config.SubscriptionKey = "rotated-key";        // next Speak/recognize uses it

// Or resolve it from the container if you didn't keep a reference:
serviceProvider.GetRequiredService<TypecastConfig>().ApiKey = "new-key";

Providers that cache an SDK/HTTP client (ElevenLabs, Typecast) transparently rebuild it when the key changes; Azure and OpenAI read the config on every call.

Samples

AI Coding Assistant

Step 1 — Add the marketplace:

claude plugin marketplace add shinyorg/skills

Step 2 — Install plugins:

claude plugin install shiny-client@shiny

BLE, GPS, Jobs, Notifications, Push, HTTP Transfers, OBD, Music, Health, DataSync — iOS, Android, Windows, MacOS, Linux, Web

claude plugin install shiny-maui@shiny

Shell, Contact Store

claude plugin install controls@shiny

TableView, BottomSheet, PillView, ImageViewer, Scheduler, Markdown, Mermaid Diagrams — MAUI and Blazor

claude plugin install shiny-mediator@shiny

Mediator/CQRS with middleware and source generators

claude plugin install shiny-data@shiny

DocumentDB and Spatial data libraries

claude plugin install shiny-aspire@shiny

Orleans and Gluetun Aspire integrations

claude plugin install shiny-extensions@shiny

DI, Stores, Reflector, Localization, Hosting modules

Step 1 — Add the marketplace:

copilot plugin marketplace add https://github.com/shinyorg/skills