rope Namespace Reference

Constants for Rotary Position Embedding (RoPE) More...

Variables
constexpr float	ROPE_THETA = 10000.0f
constexpr int	MAX_SEQUENCE_LENGTH = 2048

Detailed Description

Constants for Rotary Position Embedding (RoPE)

RoPE is used to encode positional information in the transformer's attention mechanism.

Variable Documentation

MAX_SEQUENCE_LENGTH

constexpr int rope::MAX_SEQUENCE_LENGTH = 2048

constexpr

Maximum supported sequence length

Definition at line 57 of file model_constants.h.

Referenced by TinyLlamaModel::initialize_rope_freqs().

ROPE_THETA

constexpr float rope::ROPE_THETA = 10000.0f

constexpr

Base value for frequency computation in RoPE

Definition at line 56 of file model_constants.h.

Referenced by TinyLlamaModel::initialize_rope_freqs().