Key-Value cache for a single transformer layer. More...
#include <model.h>
Collaboration diagram for KVCacheLayer:
Public Attributes | |
| std::vector< float > | k |
| std::vector< float > | v |
Detailed Description
Key-Value cache for a single transformer layer.
Stores the key and value tensors for attention mechanism, with optional CUDA support for GPU acceleration.
Member Data Documentation
◆ k
| std::vector<float> KVCacheLayer::k |
Definition at line 131 of file model.h.
Referenced by attention_batch_cpu(), attention_batch_cpu_sequence_aware(), TinyLlamaModel::forward(), update_kv_cache_batch_cpu(), and update_kv_cache_batch_cpu_sequence_aware().
◆ v
| std::vector<float> KVCacheLayer::v |
Definition at line 132 of file model.h.
Referenced by attention_batch_cpu(), attention_batch_cpu_sequence_aware(), TinyLlamaModel::forward(), update_kv_cache_batch_cpu(), and update_kv_cache_batch_cpu_sequence_aware().
The documentation for this struct was generated from the following file:
