tinyllama::TtftBreakdown Struct Reference

Where the time went between generate() being called and its first output token being emitted (docs/TTFT.md). More...

#include <api.h>

Collaboration diagram for tinyllama::TtftBreakdown:

Public Attributes
std::chrono::steady_clock::time_point	t_entry
std::chrono::steady_clock::time_point	t_first_token
int	num_prompt_tokens = 0
bool	first_token = false
bool	batch_prefill = false
double	tokenize_ms = 0.0
double	kv_reset_ms = 0.0
double	embed_ms = 0.0
double	prefill_wall_ms = 0.0
double	prefill_gpu_span_ms = -1.0
double	sample_ms = 0.0
double	ttft_ms = 0.0
int	prefix_prev_cached_tokens = 0
int	prefix_reusable_tokens = 0
double	prefix_hit_rate = 0.0
int	prefix_reused_tokens = 0
int	spec_drafted = 0
int	spec_accepted = 0
int	spec_passes = 0
double	spec_draft_ms = 0.0
int	spec_draft_calls = 0
std::map< std::string, double >	op_ms

Detailed Description

Where the time went between generate() being called and its first output token being emitted (docs/TTFT.md).

Every *_ms field is host wall-clock (steady_clock) except prefill_gpu_span_ms, which is the CUDA-event span of the batched prefill forward on the default stream. The span runs from the first to the last GPU op of the forward, so it INCLUDES any time the GPU sat idle waiting on the host (logging, synchronising copies) inside the forward; it is not the sum of kernel durations. Kernel busy time comes from a separate nsys pass. The phases tile the TTFT window with nothing left over except negligible glue, so ttft_ms ~= sum of the phase fields.

Definition at line 37 of file api.h.

Member Data Documentation

batch_prefill

bool tinyllama::TtftBreakdown::batch_prefill = false

Definition at line 42 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

embed_ms

double tinyllama::TtftBreakdown::embed_ms = 0.0

Definition at line 45 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

first_token

bool tinyllama::TtftBreakdown::first_token = false

Definition at line 41 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate(), and main().

kv_reset_ms

double tinyllama::TtftBreakdown::kv_reset_ms = 0.0

Definition at line 44 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

num_prompt_tokens

int tinyllama::TtftBreakdown::num_prompt_tokens = 0

Definition at line 40 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

op_ms

std::map<std::string, double> tinyllama::TtftBreakdown::op_ms

Definition at line 74 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefill_gpu_span_ms

double tinyllama::TtftBreakdown::prefill_gpu_span_ms = -1.0

Definition at line 48 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefill_wall_ms

double tinyllama::TtftBreakdown::prefill_wall_ms = 0.0

Definition at line 46 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefix_hit_rate

double tinyllama::TtftBreakdown::prefix_hit_rate = 0.0

Definition at line 59 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefix_prev_cached_tokens

int tinyllama::TtftBreakdown::prefix_prev_cached_tokens = 0

Definition at line 57 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefix_reusable_tokens

int tinyllama::TtftBreakdown::prefix_reusable_tokens = 0

Definition at line 58 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

prefix_reused_tokens

int tinyllama::TtftBreakdown::prefix_reused_tokens = 0

Definition at line 60 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

sample_ms

double tinyllama::TtftBreakdown::sample_ms = 0.0

Definition at line 49 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

spec_accepted

int tinyllama::TtftBreakdown::spec_accepted = 0

Definition at line 64 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

spec_draft_calls

int tinyllama::TtftBreakdown::spec_draft_calls = 0

Definition at line 69 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

spec_draft_ms

double tinyllama::TtftBreakdown::spec_draft_ms = 0.0

Definition at line 68 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

spec_drafted

int tinyllama::TtftBreakdown::spec_drafted = 0

Definition at line 63 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

spec_passes

int tinyllama::TtftBreakdown::spec_passes = 0

Definition at line 65 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

t_entry

std::chrono::steady_clock::time_point tinyllama::TtftBreakdown::t_entry

Definition at line 38 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

t_first_token

std::chrono::steady_clock::time_point tinyllama::TtftBreakdown::t_first_token

Definition at line 39 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

tokenize_ms

double tinyllama::TtftBreakdown::tokenize_ms = 0.0

Definition at line 43 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

ttft_ms

double tinyllama::TtftBreakdown::ttft_ms = 0.0

Definition at line 50 of file api.h.

Referenced by tinyllama::TinyLlamaSession::generate().

---

The documentation for this struct was generated from the following file:

• api.h