#include <tokenizer.h>
Collaboration diagram for llm_symbol:
Public Types | |
| using | index = int |
Public Attributes | |
| index | prev |
| index | next |
| const char * | text |
| size_t | n |
Detailed Description
Definition at line 21 of file tokenizer.h.
Member Typedef Documentation
◆ index
| using llm_symbol::index = int |
Definition at line 22 of file tokenizer.h.
Member Data Documentation
◆ n
| size_t llm_symbol::n |
Definition at line 26 of file tokenizer.h.
Referenced by Tokenizer::add_bigram_to_queue_refactored(), and Tokenizer::bpe_tokenize_to_ids().
◆ next
| index llm_symbol::next |
Definition at line 24 of file tokenizer.h.
Referenced by Tokenizer::add_bigram_to_queue_refactored().
◆ prev
| index llm_symbol::prev |
Definition at line 23 of file tokenizer.h.
◆ text
| const char* llm_symbol::text |
Definition at line 25 of file tokenizer.h.
Referenced by Tokenizer::add_bigram_to_queue_refactored(), and Tokenizer::bpe_tokenize_to_ids().
The documentation for this struct was generated from the following file:
