#include <tokenizer.h>
Collaboration diagram for llm_bigram_bpe:
Classes | |
| struct | comparator |
Public Types | |
| using | queue_storage = std::vector< llm_bigram_bpe > |
| using | queue = std::priority_queue< llm_bigram_bpe, queue_storage, comparator > |
Public Attributes | |
| llm_symbol::index | left |
| llm_symbol::index | right |
| std::string | text |
| int | rank |
| size_t | size |
Detailed Description
Definition at line 30 of file tokenizer.h.
Member Typedef Documentation
◆ queue
| using llm_bigram_bpe::queue = std::priority_queue<llm_bigram_bpe, queue_storage, comparator> |
Definition at line 42 of file tokenizer.h.
◆ queue_storage
| using llm_bigram_bpe::queue_storage = std::vector<llm_bigram_bpe> |
Definition at line 40 of file tokenizer.h.
Member Data Documentation
◆ left
| llm_symbol::index llm_bigram_bpe::left |
Definition at line 44 of file tokenizer.h.
Referenced by llm_bigram_bpe::comparator::operator()().
◆ rank
| int llm_bigram_bpe::rank |
Definition at line 47 of file tokenizer.h.
Referenced by llm_bigram_bpe::comparator::operator()().
◆ right
| llm_symbol::index llm_bigram_bpe::right |
Definition at line 45 of file tokenizer.h.
◆ size
| size_t llm_bigram_bpe::size |
Definition at line 48 of file tokenizer.h.
◆ text
| std::string llm_bigram_bpe::text |
Definition at line 46 of file tokenizer.h.
The documentation for this struct was generated from the following file:
