Struct FTryllContextOverrides¶

ClassList > FTryllContextOverrides

#include <TryllContextOverrides.h>

Public Attributes¶

Type	Name
ETryllKvCacheType	KvCacheType = `ETryllKvCacheType::Inherit`
int32	NBatch = `0`
int32	NOutputsMax = `0`
int32	NUbatch = `0`
bool	OffloadKqv = `0`
bool	bOverrideNBatch = `false`
bool	bOverrideNOutputsMax = `false`
bool	bOverrideNUbatch = `false`
bool	bOverrideOffloadKqv = `false`

Detailed Description¶

Backend context construction. Null fields inherit ModelVariant then server config, except n_outputs_max which CreateContext defaults to 1. Structural at the field: these are baked into llama_context_params at context creation, so a change means a new context and a lost KV cache.

Public Attributes Documentation¶

variable KvCacheType¶

ETryllKvCacheType FTryllContextOverrides::KvCacheType;

Number format of the context memory (KV cache). Smaller types save memory at some quality cost. Inherit = the model's setting, else q8_0.

variable NBatch¶

int32 FTryllContextOverrides::NBatch;

variable NOutputsMax¶

int32 FTryllContextOverrides::NOutputsMax;

variable NUbatch¶

int32 FTryllContextOverrides::NUbatch;

variable OffloadKqv¶

bool FTryllContextOverrides::OffloadKqv;

variable bOverrideNBatch¶

bool FTryllContextOverrides::bOverrideNBatch;

Most tokens sent to the model in one call while reading the prompt. Unset = the model's setting.

variable bOverrideNOutputsMax¶

bool FTryllContextOverrides::bOverrideNOutputsMax;

Leave unset. Advanced: how many positions return scores per step.

variable bOverrideNUbatch¶

bool FTryllContextOverrides::bOverrideNUbatch;

Tokens processed per GPU step; sizes the working buffers. Unset = the model's setting.

variable bOverrideOffloadKqv¶

bool FTryllContextOverrides::bOverrideOffloadKqv;

Keep the context memory on the GPU. Can only turn it off, not on for a model placed in RAM. Unset = server and model placement decide.

The documentation for this class was generated from the following file C:/_tryll/_monorepo3/tryll/clients/unreal/Source/TryllClient/Public/Generated/Nodes/TryllContextOverrides.h