Design & Architecture

A deep dive into how Veeksha works internally. Understanding these concepts will help you make informed decisions when configuring benchmarks and interpreting results.

• System Architecture
  • High-level components
  • Component descriptions
  • Request lifecycle
  • Threading model
  • Output pipeline
• Sessions and Graphs
  • The session model
  • Session graphs as DAGs
  • Linear sessions
  • History inheritance
  • Single-request sessions
  • Branching sessions
  • Session generators
  • Request scheduling within sessions
• Content Generation
  • Channel architecture
  • Text channel generator
  • Length generators
  • Content generation process
  • Shared prefix for prefix caching
  • Tokenizer integration
  • Trace-based content
  • Output length control
• Traffic Scheduling
  • Scheduling modes
  • Rate-based scheduling
  • Concurrency-based scheduling
  • Intra-session scheduling
  • Session cancellation
  • Ready queue and dispatch
  • History population
  • Timing verification
• Evaluation and Metrics
  • Evaluator architecture
  • Performance metrics
  • Configuring evaluators
  • SLO definitions
  • Output files
  • Prefill statistics
  • Accuracy evaluation
  • Streaming metrics
  • Health checks