llm_bigram_bpe::comparator Struct Reference

#include <tokenizer.h>

Collaboration diagram for llm_bigram_bpe::comparator:

Public Member Functions
bool	operator() (const llm_bigram_bpe &l, const llm_bigram_bpe &r) const

Detailed Description

Definition at line 32 of file tokenizer.h.

Member Function Documentation

operator()()

bool llm_bigram_bpe::comparator::operator() ( const llm_bigram_bpe &  l, const llm_bigram_bpe &  r  ) const

inline

Definition at line 33 of file tokenizer.h.

                                                                                  {
            // Prioritize lower rank (higher priority in BPE merges)
            // If ranks are equal, prioritize the one starting earlier (lower left index)
            return l.rank > r.rank || (l.rank == r.rank && l.left > r.left);
        }

References llm_bigram_bpe::left, and llm_bigram_bpe::rank.

---

The documentation for this struct was generated from the following file:

• tokenizer.h