ASR benchmarking

Client and transport model

client.type: stt always selects one provider-agnostic realtime WebSocket client. client.provider chooses the Vajra, vLLM, Deepgram Flux, Deepgram Nova, ElevenLabs Scribe, Mistral Voxtral, Cartesia Ink, or Together wire adapter; it does not select a different client lifecycle.

Requests name an audio file, which the client decodes to PCM16 and streams while receiving partial and final transcripts concurrently. Disabling ws_realtime_pacing sends those chunks faster, but does not switch to HTTP. There is no separate HTTP or batch STT client.

Sample configurations

The packaged samples use the same trace, pacing, evaluator, and metric contract. Only the endpoint and wire adapter differ. Vajra and vLLM samples sweep native engine concurrency; hosted API samples use one session with repeated requests.

Vajra OpenAI-compatible realtime transcription:

veeksha/sample_configs/stt_vajra.yml
seed: 42
output_dir: benchmark_output/stt_vajra

client:
  type: stt
  provider: vajra_openai_realtime
  api_base: http://localhost:8003
  # Drives Vajra through the OpenAI-compatible realtime transcription WebSocket.
  model: mistralai/Voxtral-Mini-4B-Realtime-2602
  sample_rate: 16000
  ws_chunk_size: 2048
  ws_realtime_pacing: true
  request_timeout: 7200

session_generator:
  type: trace
  trace_file: traces/asr/aa_public/manifest.jsonl
  wrap_mode: true
  flavor:
    type: audio
    audio_dir: ""

traffic_scheduler:
  type: concurrent
  target_concurrent_sessions: !expand [1, 2, 4, 8, 16, 32, 64]
  rampup_seconds: 0

evaluators:
  - type: performance
    target_channels: [audio]
    stream_metrics: true
    stream_metrics_interval: 10.0
    slos:
      # Headline interactivity: spoken word end to the first streamed
      # transcript snapshot in which that word is visible.
      - name: P90 spoken-word-end to first returned text under 1 second
        metric: interactivity
        percentile: 0.90
        value: 1000
        type: constant

runtime:
  max_sessions: -1
  benchmark_timeout: 300
  post_timeout_grace_seconds: 120
  num_client_threads: 2
  num_dispatcher_threads: 2
  num_completion_threads: 2

trace_recorder:
  enabled: false

wandb:
  enabled: false
  project: stt-benchmark
  run_name: stt_vajra_openai_realtime_voxtral
  tags: [stt, vajra, voxtral, openai_realtime]
  log_artifacts: false

vLLM Realtime transcription:

veeksha/sample_configs/stt_vllm_realtime.yml
seed: 42
output_dir: benchmark_output/stt_vllm_realtime

client:
  type: stt
  provider: vllm_realtime
  api_base: http://localhost:8025
  model: mistralai/Voxtral-Mini-4B-Realtime-2602
  sample_rate: 16000
  ws_chunk_size: 2048
  ws_realtime_pacing: true
  request_timeout: 7200

session_generator:
  type: trace
  trace_file: traces/asr/aa_public/manifest.jsonl
  wrap_mode: true
  flavor:
    type: audio
    audio_dir: ""

traffic_scheduler:
  type: concurrent
  target_concurrent_sessions: !expand [1, 2, 4, 8, 16, 32, 64]
  rampup_seconds: 0

evaluators:
  - type: performance
    target_channels: [audio]
    stream_metrics: true
    stream_metrics_interval: 10.0
    slos:
      # Headline interactivity: spoken word end to the first streamed
      # transcript snapshot in which that word is visible.
      - name: P90 spoken-word-end to first returned text under 1 second
        metric: interactivity
        percentile: 0.90
        value: 1000
        type: constant

runtime:
  max_sessions: -1
  benchmark_timeout: 300
  post_timeout_grace_seconds: 120
  num_client_threads: 2
  num_dispatcher_threads: 2
  num_completion_threads: 2

trace_recorder:
  enabled: false

wandb:
  enabled: false
  project: stt-benchmark
  run_name: stt_vllm_voxtral
  tags: [stt, vllm_realtime, voxtral]
  log_artifacts: false

Run either sample from a source checkout:

uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_vajra.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_vllm_realtime.yml

Hosted API adapters use the same client and evaluator:

uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_deepgram_flux.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_deepgram_nova.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_elevenlabs.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_mistral.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_cartesia.yml
uvx -p 3.14t veeksha benchmark --config veeksha/sample_configs/stt_together_nemotron_3_80ms.yml

Trace generation

Generate the public ASR trace with:

.venv/bin/python scripts/prepare_audio_traces.py --clips-per-dataset 128

This writes traces/asr/aa_public/manifest.jsonl plus WAV files under traces/asr/aa_public/audio/.

--max-duration controls final clip length and defaults to 30 seconds. For timestamped clips, longer source files are split on word boundaries. With --without-word-timestamping, longer untimestamped clips are skipped.

.venv/bin/python scripts/prepare_audio_traces.py \
  --clips-per-dataset 128 \
  --max-duration 30

Use --target-duration to build clips around a requested duration instead of splitting by --max-duration. Source clips are repeated before NeMo alignment until they cover the target duration, then the aligned result is truncated to the requested length. --target-duration and --max-duration are mutually exclusive.

.venv/bin/python scripts/prepare_audio_traces.py \
  --clips-per-dataset 128 \
  --target-duration 600

Word-level reference timings for the interactivity metric are generated by default with NeMo forced alignment in Docker. To skip word timestamping:

.venv/bin/python scripts/prepare_audio_traces.py \
  --clips-per-dataset 128 \
  --without-word-timestamping

AMI can be prepared through the same script. The script downloads and caches the AMI manual word annotations and Mix-Headset WAVs under benchmark_output/:

.venv/bin/python scripts/prepare_audio_traces.py \
  --datasets ami_word_timed

This writes traces/asr/ami_word_timed/manifest.jsonl plus WAV files under traces/asr/ami_word_timed/audio/.

The trace uses the public Artificial Analysis cleaned datasets, VoxPopuli and Earnings22, as a recognizable external point of reference. Earnings22 examples are kept as full, potentially long requests rather than being chunked. The trace does not reproduce the full Artificial Analysis benchmark exactly: the proprietary AA-AgentTalk dataset is unavailable, its custom normalizer is not open source, and Veeksha measures behavior consistently across serving engines.

Request metrics

Request-level metrics are written to request_level_metrics.jsonl.

Headline latency

Veeksha reports streaming STT TTFB as interactivity, not raw ttfc. For every reference word that can be matched to the evolving transcript, Veeksha records:

word latency =
    first client timestamp whose transcript contains the matched word
    - reference end-of-word timestamp

The per-request interactivity value is the mean of those matched-word latencies. The headline STT latency is P50 across request-level interactivity values. The packaged SLOs additionally gate P90 below one second. Word timestamps and transcript snapshots are both relative to the first PCM byte sent; the WebSocket handshake is therefore excluded.

ttfc remains a useful diagnostic for first transcript activity after the first input PCM byte, but it is not a spoken-word-boundary metric: it rewards a provider that emits an early scrap of text, which is not the same as keeping up with speech.

Common audio metrics:

  • ttfc: time from the first audio byte sent to the first transcript delta whose own payload carries text after control-token cleaning, or to the final transcript when no such delta arrives, in milliseconds. Empty progress or keepalive deltas do not count.

  • end_to_end_latency: time from client request start to request completion, in milliseconds.

  • generated_audio_duration: audio duration represented by the request, in milliseconds. For STT this is the input clip duration.

  • rtf: real-time factor, computed as end_to_end_latency / generated_audio_duration.

  • chunk_count: number of transcript deltas observed, or one when only a final transcript is returned.

  • input_tokens: whitespace token count of the final transcript.

  • provider: normalized serving provider family, such as vajra, vllm, deepgram, elevenlabs, mistral, cartesia, or together.

  • provider_model: configured transcription model identifier.

  • provider_protocol: concrete wire protocol selected by client.provider: v1_realtime_transcription, openai_v1_realtime_transcription, deepgram_v2_flux_listen, deepgram_v1_listen, elevenlabs_scribe_v2_realtime, mistral_realtime_transcription, cartesia_stt_websocket_manual, or together_openai_v1_realtime_transcription.

ASR-specific metrics:

  • time_to_first_visible_text: time from the first audio byte sent to the first moment the assembled, cleaned transcript is non-empty, in milliseconds. This is usually equal to ttfc; they differ only when cleaning a single delta disagrees with cleaning the assembled transcript.

  • time_to_first_partial: time from the client sending EOF or commit to the first non-empty partial transcript received afterward, in milliseconds. This is present only when the provider emits such a partial.

  • time_to_final_transcript: time from EOF or commit to the final transcript, in milliseconds.

  • partial_transcript: first non-empty post-EOF partial transcript used for partial WER.

  • final_transcript: final transcript returned by the provider.

  • expected_transcript: reference transcript from the trace row.

  • interactivity: mean latency between when matched reference words finish in the audio and when they first appear in the streamed client transcript, in milliseconds. This requires reference_word_timestamps and is most meaningful with realtime pacing enabled.

  • interactivity_word_count: number of matched words used for the per-request interactivity value.

  • partial_wer: WER for partial_transcript against expected_transcript. Present when a partial transcript is available.

  • final_wer: WER for final_transcript against expected_transcript. Present for every completed STT request with a reference transcript.

The performance evaluator computes final corpus, sample-mean, and duration-weighted WER automatically for every STT sample configuration; no separate quality evaluator is required.

Replay viewer

For runs with reference_word_timestamps and transcript_snapshots, export a browser replay bundle:

.venv/bin/python tools/asr_replay_viewer/export_asr_replay.py \
  benchmark_output/<run-dir>

Then serve the repository root:

.venv/bin/python -m http.server 8765 --bind 127.0.0.1

Open:

http://127.0.0.1:8765/tools/asr_replay_viewer/?data=benchmark_output/<run-dir>/metrics/asr_replay.json

The viewer shows the ground-truth word timeline, streamed transcript timeline, and an interactive latency graph. The graph loads ECharts from a pinned CDN URL at runtime, so the viewer needs network access when opened.

Partials and finals

Streaming providers may emit many transcript deltas before the audio stream is committed. Veeksha concatenates those deltas to track time_to_first_visible_text and the eventual transcript, but partial_transcript is specifically the first non-empty transcript observed after EOF or commit. final_transcript is the provider’s final message when available; otherwise, the concatenated deltas are used as a fallback.

Aggregate metrics

Aggregate metrics are emitted in the performance summary. General audio aggregates are percentile summaries over request-level values:

  • ttfc (Mean/P50/P90/P99)

  • end_to_end_latency (Mean/P50/P90/P99)

  • generated_audio_duration (Mean/P50/P90/P99)

  • rtf (Mean/P50/P90/P99)

  • chunk_count (Mean/P50/P90/P99)

  • time_to_first_visible_text (Mean/P50/P90/P99), when available

  • time_to_first_partial (Mean/P50/P90/P99), when available

  • time_to_final_transcript (Mean/P50/P90/P99), when available

  • interactivity (Mean/P50/P90/P99), for word-timestamped references

ASR WER aggregates:

  • asr_final_sample_count: number of scored ASR request samples.

  • asr_final_sample_mean_wer: unweighted mean WER across scored samples.

  • asr_final_corpus_wer: corpus-level WER from summed edit counts and summed reference words.

  • asr_final_duration_weighted_wer: WER weighted by audio duration.

  • asr_partial_*: the same aggregate modes for partial transcripts, over samples where partial WER is available.

  • asr_dataset_<dataset>_final_* and asr_dataset_<dataset>_partial_*: dataset-specific forms of the same metrics.

For comparisons between serving engines, use the same aggregate metric across runs. asr_final_duration_weighted_wer is the closest analogue to the duration-weighted WER used by public ASR leaderboards.

Manifest fields

Each manifest row represents one audio request. Key fields:

  • session_id: trace session identifier.

  • audio_file: WAV path relative to the manifest directory.

  • dataset: source dataset key, such as aa_voxpopuli or aa_earnings22.

  • expected_transcript: reference transcript used for WER.

  • duration_s: clip duration in seconds.

  • sample_id: unique row identifier.

  • reference_word_timestamps: optional reference timings relative to request audio start, for example {"word": "hello", "start_ms": 120.0, "end_ms": 340.0}.

When the audio trace flavor sets target_duration_s, each row is trimmed at generation time. Only the first target_duration_s seconds of audio are streamed: the STT client slices decoded PCM using the input_audio_start_ms and input_audio_end_ms request metadata written by the generator. expected_transcript and reference_word_timestamps are trimmed to words ending within that prefix. Rows must provide reference_word_timestamps, and every clip must be at least target_duration_s long.

References