Constants used throughout the TinyLlama model implementation. More...
#include <cstdint>
Include dependency graph for model_constants.h:
This graph shows which files directly or indirectly include this file:
Go to the source code of this file.
Namespaces | |
| namespace | bfloat16 |
| Constants for BFloat16 number format handling. | |
| namespace | attention |
| Constants for attention mechanism calculations. | |
| namespace | rope |
| Constants for Rotary Position Embedding (RoPE) | |
| namespace | numeric |
| Constants for ensuring numeric stability. | |
Variables | |
| constexpr uint16_t | bfloat16::EXPONENT_MASK = 0x7F80 |
| constexpr uint16_t | bfloat16::MANTISSA_MASK = 0x007F |
| constexpr uint16_t | bfloat16::SIGN_BIT = 0x8000 |
| constexpr uint16_t | bfloat16::ZERO = 0x0000 |
| constexpr uint16_t | bfloat16::NEG_ZERO = 0x8000 |
| constexpr int | bfloat16::SHIFT_BITS = 16 |
| constexpr float | attention::ATTENTION_SCALE_DEFAULT = 0.125f |
| constexpr float | attention::ATTENTION_SCALE_BASE = 1.0f |
| constexpr float | attention::MIN_SCALE = 1e-4f |
| constexpr float | attention::MAX_SCALE = 1e4f |
| constexpr int | attention::KV_REPETITION_FACTOR_DEFAULT = 1 |
| constexpr float | rope::ROPE_THETA = 10000.0f |
| constexpr int | rope::MAX_SEQUENCE_LENGTH = 2048 |
| constexpr float | numeric::MIN_NORM_EPS = 1e-5f |
| constexpr float | numeric::DEFAULT_EPS = 1e-6f |
| constexpr float | numeric::MAX_LOGIT_THRESHOLD = 100.0f |
Detailed Description
Constants used throughout the TinyLlama model implementation.
This file defines various constants used in different components of the model, including BFloat16 handling, attention mechanisms, rotary position embeddings, and numeric stability thresholds.
Definition in file model_constants.h.
