Introducing AI Conversations: Natural Language Interaction for Your Apps! Learn More

Getting Started


GitHub
Core
Cloud
Azure
ElevenLabs

Frameworks

.NET

.NET MAUI

Operating Systems

Android

iOS

Windows

Web

Shiny.Speech provides a unified API for speech-to-text, text-to-speech, audio capture, and audio playback across Android, iOS, Windows, and Browser (Blazor WebAssembly) — with pluggable cloud providers for Azure AI Speech and ElevenLabs.

Features

Platform-native speech-to-text with streaming results via IAsyncEnumerable
Platform-native text-to-speech with voice selection, rate, pitch, and volume control
Raw audio capture from the device microphone (16kHz, 16-bit, mono PCM)
Audio playback for MP3 streams
Pluggable cloud provider architecture — swap between on-device and cloud STT/TTS
Azure AI Speech integration (STT + TTS) with SSML prosody control
ElevenLabs integration (TTS) with multilingual voice support
State tracking — IsListening (STT), IsSpeaking (TTS), IsPlaying (audio)
Permission management via AccessState and RequestAccess()
Listen-until-silence dictation mode for simple transcription scenarios
Continuous recognition mode for real-time streaming transcription
Wake word activation (“Hey Siri” style) — listens for a phrase then captures the command
Keyword detection — listens until a specific keyword is spoken (e.g. Yes/No/Maybe)

Packages

Package	Purpose
`Shiny.Speech`	Core library — platform-native STT, TTS, audio capture, and playback
`Shiny.Speech.Cloud`	Cloud provider abstractions (included transitively by Azure/ElevenLabs)
`Shiny.Speech.Azure`	Azure AI Speech provider (STT + TTS)
`Shiny.Speech.ElevenLabs`	ElevenLabs TTS provider

Setup

Shiny.Speech

Platform Notes

Android’s native speech recognition engine works in short recording segments — it automatically stops listening after a period of silence and must restart for the next segment. This creates brief pauses during continuous listening (ListenUntilSilence, ContinuousRecognize, wake word detection) that are noticeable compared to iOS, which streams continuously.

If smooth, uninterrupted speech recognition is important for your app, use Azure Speech Services which provides continuous streaming recognition on all platforms, eliminating the start/stop behavior.

builder.Services.AddAzureSpeech("your-subscription-key", "eastus");

Platform Permissions

Android — Add to AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

iOS — Add to Info.plist:

<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone for speech recognition</string>

Windows — Add the Microphone capability to your Package.appxmanifest:

<Capabilities>
    <DeviceCapability Name="microphone" />
</Capabilities>

Browser (Blazor WebAssembly) — No manifest changes needed. The browser prompts for microphone access automatically. Include the JS interop module in index.html:

<script src="shiny-speech.js"></script>

IAudioSource captures raw PCM audio in the browser using the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono.

Quick Example

public class MyViewModel
{
    readonly ISpeechToTextService _stt;
    readonly ITextToSpeechService _tts;

    public MyViewModel(ISpeechToTextService stt, ITextToSpeechService tts)
    {
        _stt = stt;
        _tts = tts;
    }

    async Task ListenAndRespond(CancellationToken ct)
    {
        // 1. Request permission
        var access = await _stt.RequestAccess();
        if (access != AccessState.Available)
            return;

        // 2. Listen until the user stops speaking
        var text = await _stt.ListenUntilSilence(
            new SpeechRecognitionOptions
            {
                Culture = CultureInfo.GetCultureInfo("en-US"),
                SilenceTimeout = TimeSpan.FromSeconds(3)
            },
            ct
        );

        if (text != null)
        {
            // 3. Speak the result back
            await _tts.SpeakAsync($"You said: {text}");
        }
    }

    async Task StreamRecognition(CancellationToken ct)
    {
        // Continuous recognition with streaming results
        await foreach (var result in _stt.ContinuousRecognize(cancellationToken: ct))
        {
            Console.WriteLine($"{result.Text} (final: {result.IsFinal}, confidence: {result.Confidence})");
        }
    }

    async Task WakeWordExample(CancellationToken ct)
    {
        // "Hey Siri" style — listens until wake phrase, then captures the command
        var command = await _stt.ListenWithWakeWord("Hey Computer", cancellationToken: ct);
        // "Hey Computer, what's the weather" → "what's the weather"

        if (command != null)
            await _tts.SpeakAsync($"You asked: {command}");
    }

    async Task KeywordExample(CancellationToken ct)
    {
        // Listen until a keyword is detected
        await _tts.SpeakAsync("Do you agree? Say yes, no, or maybe.");
        var answer = await _stt.ListenForKeyword(["Yes", "No", "Maybe"], cancellationToken: ct);
        // "I think yes" → "Yes"
    }
}

Samples

AI Coding Assistant

Step 1 — Add the marketplace:

claude plugin marketplace add shinyorg/skills

Step 2 — Install plugins:

claude plugin install shiny-client@shiny

BLE, GPS, Jobs, Notifications, Push, HTTP Transfers, OBD, Music, Health, DataSync — iOS, Android, Windows, MacOS, Linux, Web

claude plugin install shiny-maui@shiny

Shell, Contact Store

claude plugin install shiny-controls@shiny

TableView, BottomSheet, PillView, ImageViewer, Scheduler, Markdown, Mermaid Diagrams — MAUI and Blazor

claude plugin install shiny-mediator@shiny

Mediator/CQRS with middleware and source generators

claude plugin install shiny-data@shiny

DocumentDB and Spatial data libraries

claude plugin install shiny-aspire@shiny

Orleans and Gluetun Aspire integrations

claude plugin install shiny-extensions@shiny

DI, Stores, Reflector, Localization, Hosting modules

Step 1 — Add the marketplace:

copilot plugin marketplace add https://github.com/shinyorg/skills

Step 2 — Install plugins:

copilot plugin install shiny-client@shiny

BLE, GPS, Jobs, Notifications, Push, HTTP Transfers, OBD, Music, Health, DataSync — iOS, Android, Windows, MacOS, Linux, Web

copilot plugin install shiny-maui@shiny

Shell, Contact Store

copilot plugin install shiny-controls@shiny