AnySpeechRecognizer: Features, Models and How to Use It

What is AnySpeechRecognizer?

AnySpeechRecognizer is a Swift package developed by Hugging Face that implements Apple's speech-to-text API while allowing developers to choose alternative recognition engines. It mirrors the SpeechAnalyzer framework introduced in macOS 26 and iOS 26, using Apple's default engine or local models running on Core ML.

The package also introduces speaker diarization, which tracks who spoke and when. By running open models on device, it lets developers transcribe audio without sending data to external cloud services.

Supported speech recognition models

AnySpeechRecognizer supports multiple open speech-to-text models converted to Core ML format and hosted on the Hugging Face Hub. Developers can select models based on language requirements, latency needs, and available storage.

NVIDIA Parakeet v3 supports 25 European languages with automatic language detection, volatile live results, word timing, and word confidence scores. Moondream post-trained the Parakeet Ultra variant for higher accuracy, while Parakeet Unified EN targets English only. Parakeet models download around 600 MB plus an optional 590 MB encoder for live streams.

Alibaba Qwen3-ASR comes in 0.6B and 1.7B parameters, requiring 0.9 GB to 2.4 GB of storage. It supports 30 languages and 22 Chinese dialects, including Cantonese, Mandarin, Japanese, and Korean, but it runs on the GPU and processes files in chunks up to 40 seconds without word timings or volatile streaming results.

Speaker diarization with Sortformer

AnySpeechRecognizer adds speaker diarization to Apple's API, an ability missing from the default Speech framework. It integrates NVIDIA's Streaming Sortformer model to detect who is speaking during a conversation.

The diarizer tracks up to four distinct speakers simultaneously, even when their voices overlap. A component named SpeakerTimeline aligns diarizer output with transcriber word timestamps, allowing apps to print formatted transcripts segmented by speaker turn.

The package uses Sortformer v2 by default and includes Sortformer v2.1 as an option. Hugging Face notes that Sortformer v2.1 was trained on additional meeting audio and delivered higher accuracy in meeting tests.

How to set up and use AnySpeechRecognizer

AnySpeechRecognizer is designed as a drop-in replacement for Apple's SpeechAnalyzer framework in Swift. Developers can replace Apple's native engine by updating framework imports and initializing custom model configurations.

1. Add the AnySpeechRecognizer package to your Swift project and import both Speech and AnySpeechRecognizer in your source files.
2. Initialize SpeechTranscriber with your chosen model configuration, such as ParakeetSpeechModel(variant: .v3) or QwenSpeechModel.
3. Optionally initialize SpeakerDiarizer using SortformerDiarizationModel() to track speaker turns.
4. Call AssetInventory.assetInstallationRequest with your modules and await downloadAndInstall() to download the Core ML model weights from the Hugging Face Hub.
5. Initialize SpeechAnalyzer with your transcriber and diarizer modules, then pass your audio file or live audio stream buffer to receive transcription sequences.

Limits and system requirements

AnySpeechRecognizer requires Apple operating system versions that support SpeechAnalyzer, specifically macOS 26 or iOS 26 and later. The package was benchmarked by Hugging Face on macOS 27.0.1 using an M4 Max Mac Studio.

Hardware demands vary across models. Qwen3-ASR executes on the GPU and is significantly slower than Parakeet, making it less practical for real-time live microphone transcription. It also lacks support for partial volatile results, word-level timing, and confidence scores.

Parakeet offers faster processing but requires an extra 8-second live encoder module to reduce live latency. Hugging Face reports that while this live encoder cuts analysis time by approximately one-third, it results in a slightly higher word error rate.

Frequently asked questions

Sources: Hugging Face