llm_symbol Struct Reference

#include <tokenizer.h>

Collaboration diagram for llm_symbol:

Public Types
using	index = int

Public Attributes
index	prev
index	next
const char *	text
size_t	n

Detailed Description

Definition at line 21 of file tokenizer.h.

Member Typedef Documentation

index

using llm_symbol::index = int

Definition at line 22 of file tokenizer.h.

Member Data Documentation

n

size_t llm_symbol::n

Definition at line 26 of file tokenizer.h.

next

index llm_symbol::next

Definition at line 24 of file tokenizer.h.

Referenced by Tokenizer::add_bigram_to_queue_refactored().

prev

index llm_symbol::prev

Definition at line 23 of file tokenizer.h.

text

const char* llm_symbol::text

Definition at line 25 of file tokenizer.h.

---

The documentation for this struct was generated from the following file:

• tokenizer.h