ember::ChatTokenizer Class Reference

#include <chat_tokenizer.h>

Inheritance diagram for ember::ChatTokenizer:

Collaboration diagram for ember::ChatTokenizer:

Public Member Functions
virtual	~ChatTokenizer ()=default
virtual std::vector< int >	encode (const std::string &text) const =0
virtual std::string	decode (const std::vector< int > &ids) const =0
virtual size_t	size () const =0
virtual const char *	kind () const =0
const ChatFormat &	format () const

Protected Attributes
ChatFormat	format_

Detailed Description

Definition at line 127 of file chat_tokenizer.h.

Constructor & Destructor Documentation

~ChatTokenizer()

virtual ember::ChatTokenizer::~ChatTokenizer ( )

virtualdefault

Member Function Documentation

decode()

virtual std::string ember::ChatTokenizer::decode ( const std::vector< int > &  ids ) const

pure virtual

Implemented in ember::ChatTokenizerOf< T >.

encode()

virtual std::vector< int > ember::ChatTokenizer::encode ( const std::string &  text ) const

pure virtual

Implemented in ember::ChatTokenizerOf< T >.

format()

const ChatFormat & ember::ChatTokenizer::format ( ) const

inline

Definition at line 134 of file chat_tokenizer.h.

{ return format_; }

References format_.

Referenced by ember::ChatTokenizerOf< T >::ChatTokenizerOf().

kind()

virtual const char * ember::ChatTokenizer::kind ( ) const

pure virtual

Implemented in ember::ChatTokenizerOf< T >.

size()

virtual size_t ember::ChatTokenizer::size ( ) const

pure virtual

Implemented in ember::ChatTokenizerOf< T >.

Member Data Documentation

format_

ChatFormat ember::ChatTokenizer::format_

protected

Definition at line 137 of file chat_tokenizer.h.

Referenced by ember::ChatTokenizerOf< T >::ChatTokenizerOf(), and format().

---

The documentation for this class was generated from the following file:

• chat_tokenizer.h