ASR benchmarking ================ Client and transport model -------------------------- ``client.type: stt`` always selects one provider-agnostic realtime WebSocket client. ``client.provider`` chooses the Vajra, vLLM, Deepgram Flux, Deepgram Nova, ElevenLabs Scribe, Mistral Voxtral, Cartesia Ink, or Together wire adapter; it does not select a different client lifecycle. Requests name an audio file, which the client decodes to PCM16 and streams while receiving partial and final transcripts concurrently. Disabling ``ws_realtime_pacing`` sends those chunks faster, but does not switch to HTTP. There is no separate HTTP or batch STT client. Sample configurations --------------------- The packaged samples use the same trace, pacing, evaluator, and metric contract. Only the endpoint and wire adapter differ. Vajra and vLLM samples sweep native engine concurrency; hosted API samples use one session with repeated requests. Vajra OpenAI-compatible realtime transcription: .. literalinclude:: ../../veeksha/sample_configs/stt_vajra.yml :language: yaml :caption: veeksha/sample_configs/stt_vajra.yml vLLM Realtime transcription: .. literalinclude:: ../../veeksha/sample_configs/stt_vllm_realtime.yml :language: yaml :caption: veeksha/sample_configs/stt_vllm_realtime.yml Run either sample from a source checkout: .. code-block:: console uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_vajra.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_vllm_realtime.yml Hosted API adapters use the same client and evaluator: .. code-block:: console uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_deepgram_flux.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_deepgram_nova.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_elevenlabs.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_mistral.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_cartesia.yml uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_together_nemotron_3_80ms.yml Trace generation ---------------- Generate the public ASR trace with: .. code-block:: console .venv/bin/python scripts/prepare_audio_traces.py --clips-per-dataset 128 This writes ``traces/asr/aa_public/manifest.jsonl`` plus WAV files under ``traces/asr/aa_public/audio/``. ``--max-duration`` controls final clip length and defaults to 30 seconds. For timestamped clips, longer source files are split on word boundaries. With ``--without-word-timestamping``, longer untimestamped clips are skipped. .. code-block:: console .venv/bin/python scripts/prepare_audio_traces.py \ --clips-per-dataset 128 \ --max-duration 30 Use ``--target-duration`` to build clips around a requested duration instead of splitting by ``--max-duration``. Source clips are repeated before NeMo alignment until they cover the target duration, then the aligned result is truncated to the requested length. ``--target-duration`` and ``--max-duration`` are mutually exclusive. .. code-block:: console .venv/bin/python scripts/prepare_audio_traces.py \ --clips-per-dataset 128 \ --target-duration 600 Word-level reference timings for the interactivity metric are generated by default with NeMo forced alignment in Docker. To skip word timestamping: .. code-block:: console .venv/bin/python scripts/prepare_audio_traces.py \ --clips-per-dataset 128 \ --without-word-timestamping AMI can be prepared through the same script. The script downloads and caches the AMI manual word annotations and Mix-Headset WAVs under ``benchmark_output/``: .. code-block:: console .venv/bin/python scripts/prepare_audio_traces.py \ --datasets ami_word_timed This writes ``traces/asr/ami_word_timed/manifest.jsonl`` plus WAV files under ``traces/asr/ami_word_timed/audio/``. The trace uses the public Artificial Analysis cleaned datasets, VoxPopuli and Earnings22, as a recognizable external point of reference. Earnings22 examples are kept as full, potentially long requests rather than being chunked. The trace does not reproduce the full Artificial Analysis benchmark exactly: the proprietary AA-AgentTalk dataset is unavailable, its custom normalizer is not open source, and Veeksha measures behavior consistently across serving engines. Request metrics --------------- Request-level metrics are written to ``request_level_metrics.jsonl``. Headline latency ~~~~~~~~~~~~~~~~ Veeksha reports streaming STT ``TTFB`` as ``interactivity``, not raw ``ttfc``. For every reference word that can be matched to the evolving transcript, Veeksha records: .. code-block:: text word latency = first client timestamp whose transcript contains the matched word - reference end-of-word timestamp The per-request ``interactivity`` value is the mean of those matched-word latencies. The headline STT latency is P50 across request-level ``interactivity`` values. The packaged SLOs additionally gate P90 below one second. Word timestamps and transcript snapshots are both relative to the first PCM byte sent; the WebSocket handshake is therefore excluded. ``ttfc`` remains a useful diagnostic for first transcript activity after the first input PCM byte, but it is not a spoken-word-boundary metric: it rewards a provider that emits an early scrap of text, which is not the same as keeping up with speech. Common audio metrics: * ``ttfc``: time from the first audio byte sent to the first transcript delta whose own payload carries text after control-token cleaning, or to the final transcript when no such delta arrives, in milliseconds. Empty progress or keepalive deltas do not count. * ``end_to_end_latency``: time from client request start to request completion, in milliseconds. * ``generated_audio_duration``: audio duration represented by the request, in milliseconds. For STT this is the input clip duration. * ``rtf``: real-time factor, computed as ``end_to_end_latency / generated_audio_duration``. * ``chunk_count``: number of transcript deltas observed, or one when only a final transcript is returned. * ``input_tokens``: whitespace token count of the final transcript. * ``provider``: normalized serving provider family, such as ``vajra``, ``vllm``, ``deepgram``, ``elevenlabs``, ``mistral``, ``cartesia``, or ``together``. * ``provider_model``: configured transcription model identifier. * ``provider_protocol``: concrete wire protocol selected by ``client.provider``: ``v1_realtime_transcription``, ``openai_v1_realtime_transcription``, ``deepgram_v2_flux_listen``, ``deepgram_v1_listen``, ``elevenlabs_scribe_v2_realtime``, ``mistral_realtime_transcription``, ``cartesia_stt_websocket_manual``, or ``together_openai_v1_realtime_transcription``. ASR-specific metrics: * ``time_to_first_visible_text``: time from the first audio byte sent to the first moment the assembled, cleaned transcript is non-empty, in milliseconds. This is usually equal to ``ttfc``; they differ only when cleaning a single delta disagrees with cleaning the assembled transcript. * ``time_to_first_partial``: time from the client sending EOF or commit to the first non-empty partial transcript received afterward, in milliseconds. This is present only when the provider emits such a partial. * ``time_to_final_transcript``: time from EOF or commit to the final transcript, in milliseconds. * ``partial_transcript``: first non-empty post-EOF partial transcript used for partial WER. * ``final_transcript``: final transcript returned by the provider. * ``expected_transcript``: reference transcript from the trace row. * ``interactivity``: mean latency between when matched reference words finish in the audio and when they first appear in the streamed client transcript, in milliseconds. This requires ``reference_word_timestamps`` and is most meaningful with realtime pacing enabled. * ``interactivity_word_count``: number of matched words used for the per-request ``interactivity`` value. * ``partial_wer``: WER for ``partial_transcript`` against ``expected_transcript``. Present when a partial transcript is available. * ``final_wer``: WER for ``final_transcript`` against ``expected_transcript``. Present for every completed STT request with a reference transcript. The performance evaluator computes final corpus, sample-mean, and duration-weighted WER automatically for every STT sample configuration; no separate quality evaluator is required. Replay viewer ------------- For runs with ``reference_word_timestamps`` and ``transcript_snapshots``, export a browser replay bundle: .. code-block:: console .venv/bin/python tools/asr_replay_viewer/export_asr_replay.py \ benchmark_output/ Then serve the repository root: .. code-block:: console .venv/bin/python -m http.server 8765 --bind 127.0.0.1 Open: .. code-block:: text http://127.0.0.1:8765/tools/asr_replay_viewer/?data=benchmark_output//metrics/asr_replay.json The viewer shows the ground-truth word timeline, streamed transcript timeline, and an interactive latency graph. The graph loads ECharts from a pinned CDN URL at runtime, so the viewer needs network access when opened. Partials and finals ------------------- Streaming providers may emit many transcript deltas before the audio stream is committed. Veeksha concatenates those deltas to track ``time_to_first_visible_text`` and the eventual transcript, but ``partial_transcript`` is specifically the first non-empty transcript observed after EOF or commit. ``final_transcript`` is the provider's final message when available; otherwise, the concatenated deltas are used as a fallback. Aggregate metrics ----------------- Aggregate metrics are emitted in the performance summary. General audio aggregates are percentile summaries over request-level values: * ``ttfc (Mean/P50/P90/P99)`` * ``end_to_end_latency (Mean/P50/P90/P99)`` * ``generated_audio_duration (Mean/P50/P90/P99)`` * ``rtf (Mean/P50/P90/P99)`` * ``chunk_count (Mean/P50/P90/P99)`` * ``time_to_first_visible_text (Mean/P50/P90/P99)``, when available * ``time_to_first_partial (Mean/P50/P90/P99)``, when available * ``time_to_final_transcript (Mean/P50/P90/P99)``, when available * ``interactivity (Mean/P50/P90/P99)``, for word-timestamped references ASR WER aggregates: * ``asr_final_sample_count``: number of scored ASR request samples. * ``asr_final_sample_mean_wer``: unweighted mean WER across scored samples. * ``asr_final_corpus_wer``: corpus-level WER from summed edit counts and summed reference words. * ``asr_final_duration_weighted_wer``: WER weighted by audio duration. * ``asr_partial_*``: the same aggregate modes for partial transcripts, over samples where partial WER is available. * ``asr_dataset__final_*`` and ``asr_dataset__partial_*``: dataset-specific forms of the same metrics. For comparisons between serving engines, use the same aggregate metric across runs. ``asr_final_duration_weighted_wer`` is the closest analogue to the duration-weighted WER used by public ASR leaderboards. Manifest fields --------------- Each manifest row represents one audio request. Key fields: * ``session_id``: trace session identifier. * ``audio_file``: WAV path relative to the manifest directory. * ``dataset``: source dataset key, such as ``aa_voxpopuli`` or ``aa_earnings22``. * ``expected_transcript``: reference transcript used for WER. * ``duration_s``: clip duration in seconds. * ``sample_id``: unique row identifier. * ``reference_word_timestamps``: optional reference timings relative to request audio start, for example ``{"word": "hello", "start_ms": 120.0, "end_ms": 340.0}``. When the audio trace flavor sets ``target_duration_s``, each row is trimmed at generation time. Only the first ``target_duration_s`` seconds of audio are streamed: the STT client slices decoded PCM using the ``input_audio_start_ms`` and ``input_audio_end_ms`` request metadata written by the generator. ``expected_transcript`` and ``reference_word_timestamps`` are trimmed to words ending within that prefix. Rows must provide ``reference_word_timestamps``, and every clip must be at least ``target_duration_s`` long. References ---------- * `Artificial Analysis speech-to-text methodology `_ * `VoxPopuli-Cleaned-AA `_ * `Earnings22-Cleaned-AA `_