attention Namespace Reference

Constants for attention mechanism calculations. More...

Variables
constexpr float	ATTENTION_SCALE_DEFAULT = 0.125f
constexpr float	ATTENTION_SCALE_BASE = 1.0f
constexpr float	MIN_SCALE = 1e-4f
constexpr float	MAX_SCALE = 1e4f
constexpr int	KV_REPETITION_FACTOR_DEFAULT = 1

Detailed Description

Constants for attention mechanism calculations.

These constants are used in the scaled dot-product attention mechanism of the transformer architecture.

Variable Documentation

ATTENTION_SCALE_BASE

constexpr float attention::ATTENTION_SCALE_BASE = 1.0f

constexpr

Definition at line 39 of file model_constants.h.

Referenced by calculate_attention_scores().

ATTENTION_SCALE_DEFAULT

constexpr float attention::ATTENTION_SCALE_DEFAULT = 0.125f

constexpr

Definition at line 38 of file model_constants.h.

KV_REPETITION_FACTOR_DEFAULT

constexpr int attention::KV_REPETITION_FACTOR_DEFAULT = 1

constexpr

Definition at line 45 of file model_constants.h.

MAX_SCALE

constexpr float attention::MAX_SCALE = 1e4f

constexpr

Maximum allowed attention scale to prevent overflow

Definition at line 42 of file model_constants.h.

Referenced by calculate_attention_scores().

MIN_SCALE

constexpr float attention::MIN_SCALE = 1e-4f

constexpr

Minimum allowed attention scale to prevent underflow

Definition at line 41 of file model_constants.h.

Referenced by calculate_attention_scores().