microtorch::llama3::Tokenizer::Symbol Struct Reference

Collaboration diagram for microtorch::llama3::Tokenizer::Symbol:

Public Attributes
int	prev
int	next
size_t	start
size_t	n

Detailed Description

Definition at line 348 of file llama3_tokenizer.h.

Member Data Documentation

n

size_t microtorch::llama3::Tokenizer::Symbol::n

Definition at line 350 of file llama3_tokenizer.h.

Referenced by microtorch::llama3::Tokenizer::encode_word().

next

int microtorch::llama3::Tokenizer::Symbol::next

Definition at line 349 of file llama3_tokenizer.h.

prev

int microtorch::llama3::Tokenizer::Symbol::prev

Definition at line 349 of file llama3_tokenizer.h.

start

size_t microtorch::llama3::Tokenizer::Symbol::start

Definition at line 350 of file llama3_tokenizer.h.

---

The documentation for this struct was generated from the following file:

• llama3_tokenizer.h