microtorch::bpe::GgufTokenizer Struct Reference

#include <bpe_tokenizer.h>

Collaboration diagram for microtorch::bpe::GgufTokenizer:

Public Attributes
std::string	model
std::vector< std::string >	tokens
std::vector< std::string >	merges
std::string	pre
std::vector< int32_t >	token_type

Detailed Description

Definition at line 154 of file bpe_tokenizer.h.

Member Data Documentation

merges

std::vector<std::string> microtorch::bpe::GgufTokenizer::merges

Definition at line 156 of file bpe_tokenizer.h.

Referenced by microtorch::bpe::load_gguf(), and microtorch::llama3::load_gguf().

model

std::string microtorch::bpe::GgufTokenizer::model

Definition at line 155 of file bpe_tokenizer.h.

Referenced by microtorch::bpe::load_gguf(), and microtorch::llama3::load_gguf().

pre

std::string microtorch::bpe::GgufTokenizer::pre

Definition at line 157 of file bpe_tokenizer.h.

Referenced by microtorch::llama3::load_gguf().

token_type

std::vector<int32_t> microtorch::bpe::GgufTokenizer::token_type

Definition at line 158 of file bpe_tokenizer.h.

Referenced by microtorch::llama3::load_gguf().

tokens

std::vector<std::string> microtorch::bpe::GgufTokenizer::tokens

Definition at line 156 of file bpe_tokenizer.h.

Referenced by microtorch::bpe::load_gguf(), and microtorch::llama3::load_gguf().

---

The documentation for this struct was generated from the following file:

• bpe_tokenizer.h