GGUF file format

GGUF is a binary file format for storing models for inference with ggml and executors based on ggml. It is designed for fast loading and saving, ease of reading, and single-file deployment. GGUF is the successor to the earlier GGML, GGMF, and GGJT formats. The key improvement over GGJT is the use of a typed key-value structure for metadata, rather than a fixed list of untyped hyperparameters. This allows new metadata to be added without breaking compatibility with existing models.

Design goals

Single-file deployment — models can be distributed and loaded without external files.
Extensibility — new metadata can be added without breaking existing readers.
mmap compatibility — tensors are aligned so models can be loaded with mmap.
Full information — everything needed to load the model is embedded in the file itself.

File structure

A GGUF file is laid out sequentially as follows:

struct gguf_file_t {
    // The header of the file.
    gguf_header_t header;

    // Tensor infos, which can be used to locate the tensor data.
    gguf_tensor_info_t tensor_infos[header.tensor_count];

    // Padding to the nearest multiple of ALIGNMENT.
    uint8_t _padding[];

    // Tensor data (arbitrary binary weights).
    uint8_t tensor_data[];
};

The header appears at the start of every GGUF file:

struct gguf_header_t {
    // Magic number: must be 0x47 0x47 0x55 0x46 ("GGUF").
    uint32_t magic;
    // Format version. Current version is 3.
    uint32_t version;
    // Number of tensors in the file.
    uint64_t tensor_count;
    // Number of metadata key-value pairs.
    uint64_t metadata_kv_count;
    // The metadata key-value pairs.
    gguf_metadata_kv_t metadata_kv[metadata_kv_count];
};

Models are little-endian by default. Big-endian support was added in format version 3. If no additional information is provided, assume the model is little-endian.

Tensor info

Each tensor is described by a gguf_tensor_info_t entry. The actual data starts after all tensor info entries, padded to the alignment boundary:

struct gguf_tensor_info_t {
    // Tensor name, at most 64 bytes.
    gguf_string_t name;
    // Number of dimensions (currently at most 4).
    uint32_t n_dimensions;
    // Size along each dimension.
    uint64_t dimensions[n_dimensions];
    // Element data type.
    ggml_type type;
    // Byte offset of this tensor's data within the tensor_data blob.
    // Must be a multiple of ALIGNMENT.
    uint64_t offset;
};

Alignment

The global alignment is set by the general.alignment metadata key (default: 32). Padding bytes (0x00) are inserted to align tensor data:

uint64_t align_offset(uint64_t offset) {
    return offset + (ALIGNMENT - (offset % ALIGNMENT)) % ALIGNMENT;
}

Metadata types

The gguf_type enum describes every value type that can appear in a GGUF key-value pair:

enum gguf_type {
    GGUF_TYPE_UINT8   = 0,
    GGUF_TYPE_INT8    = 1,
    GGUF_TYPE_UINT16  = 2,
    GGUF_TYPE_INT16   = 3,
    GGUF_TYPE_UINT32  = 4,
    GGUF_TYPE_INT32   = 5,
    GGUF_TYPE_FLOAT32 = 6,
    GGUF_TYPE_BOOL    = 7,   // stored as int8_t; 0 = false, 1 = true
    GGUF_TYPE_STRING  = 8,   // uint64_t length + UTF-8 bytes, no null terminator
    GGUF_TYPE_ARRAY   = 9,   // type + uint64_t count + elements
    GGUF_TYPE_UINT64  = 10,
    GGUF_TYPE_INT64   = 11,
    GGUF_TYPE_FLOAT64 = 12,
    GGUF_TYPE_COUNT,
};

All enums are stored as int32_t. Strings are serialized as a uint64_t length followed by the UTF-8 bytes without a null terminator.

Key-value pairs

Each metadata entry is a gguf_metadata_kv_t:

struct gguf_metadata_kv_t {
    // Key: valid ASCII, hierarchical lower_snake_case segments separated by '.',
    // at most 65535 bytes.
    gguf_string_t key;
    gguf_metadata_value_type value_type;
    gguf_metadata_value_t value;
};

Keys follow the convention namespace.property (e.g. general.architecture, llama.context_length). Community-defined keys should be prefixed with the community name (e.g. rustformers.my_key).

Tensor element types

The ggml_type enum covers all supported tensor element types, including floating-point and quantized formats:

enum ggml_type: uint32_t {
    GGML_TYPE_F32     = 0,
    GGML_TYPE_F16     = 1,
    GGML_TYPE_Q4_0    = 2,
    GGML_TYPE_Q4_1    = 3,
    GGML_TYPE_Q5_0    = 6,
    GGML_TYPE_Q5_1    = 7,
    GGML_TYPE_Q8_0    = 8,
    GGML_TYPE_Q8_1    = 9,
    GGML_TYPE_Q2_K    = 10,
    GGML_TYPE_Q3_K    = 11,
    GGML_TYPE_Q4_K    = 12,
    GGML_TYPE_Q5_K    = 13,
    GGML_TYPE_Q6_K    = 14,
    GGML_TYPE_Q8_K    = 15,
    GGML_TYPE_IQ2_XXS = 16,
    GGML_TYPE_IQ2_XS  = 17,
    GGML_TYPE_IQ3_XXS = 18,
    GGML_TYPE_IQ1_S   = 19,
    GGML_TYPE_IQ4_NL  = 20,
    GGML_TYPE_IQ3_S   = 21,
    GGML_TYPE_IQ2_S   = 22,
    GGML_TYPE_IQ4_XS  = 23,
    GGML_TYPE_I8      = 24,
    GGML_TYPE_I16     = 25,
    GGML_TYPE_I32     = 26,
    GGML_TYPE_I64     = 27,
    GGML_TYPE_F64     = 28,
    GGML_TYPE_IQ1_M   = 29,
    GGML_TYPE_BF16    = 30,
    GGML_TYPE_TQ1_0   = 34,
    GGML_TYPE_TQ2_0   = 35,
    GGML_TYPE_MXFP4   = 39,
    GGML_TYPE_COUNT   = 40,
};

C API

Initializing a context

// Open an empty GGUF context (for building a new file).
struct gguf_context * gguf_init_empty(void);

// Load a GGUF file from disk.
// Set params.no_alloc = false and params.ctx to a ggml_context to also load tensor data.
struct gguf_context * gguf_init_from_file(
    const char * fname,
    struct gguf_init_params params
);

void gguf_free(struct gguf_context * ctx);

Writing files

There are three ways to write a GGUF file:

Single pass
Metadata then data
Data then metadata

Write everything in one call:

// Write the entire context to a binary file.
// Pass only_meta = false to include tensor data.
bool gguf_write_to_file(
    const struct gguf_context * ctx,
    const char * fname,
    bool only_meta
);

Write metadata first, then append tensor data:

gguf_write_to_file(ctx, fname, /*only_meta =*/ true);
FILE * f = fopen(fname, "ab");
fwrite(tensor_data, ...); // append tensor data
fclose(f);

Reserve space for metadata, write data, then fill in the header:

FILE * f = fopen(fname, "wb");
const size_t size_meta = gguf_get_meta_size(ctx);
fseek(f, size_meta, SEEK_SET);
fwrite(tensor_data, ...);      // write tensor data first
void * data = malloc(size_meta);
gguf_get_meta_data(ctx, data); // serialise header into buffer
rewind(f);
fwrite(data, 1, size_meta, f); // write header
free(data);
fclose(f);

Reading key-value metadata

// Number of KV pairs.
int64_t gguf_get_n_kv(const struct gguf_context * ctx);

// Find a key by name; returns -1 if not found.
int64_t gguf_find_key(const struct gguf_context * ctx, const char * key);

// Get the string key for a given key_id.
const char * gguf_get_key(const struct gguf_context * ctx, int64_t key_id);

// Get the type of a KV pair.
enum gguf_type gguf_get_kv_type(const struct gguf_context * ctx, int64_t key_id);

// Type-specific value accessors (will abort if the wrong type is used).
uint8_t      gguf_get_val_u8  (const struct gguf_context * ctx, int64_t key_id);
int8_t       gguf_get_val_i8  (const struct gguf_context * ctx, int64_t key_id);
uint16_t     gguf_get_val_u16 (const struct gguf_context * ctx, int64_t key_id);
int16_t      gguf_get_val_i16 (const struct gguf_context * ctx, int64_t key_id);
uint32_t     gguf_get_val_u32 (const struct gguf_context * ctx, int64_t key_id);
int32_t      gguf_get_val_i32 (const struct gguf_context * ctx, int64_t key_id);
float        gguf_get_val_f32 (const struct gguf_context * ctx, int64_t key_id);
uint64_t     gguf_get_val_u64 (const struct gguf_context * ctx, int64_t key_id);
int64_t      gguf_get_val_i64 (const struct gguf_context * ctx, int64_t key_id);
double       gguf_get_val_f64 (const struct gguf_context * ctx, int64_t key_id);
bool         gguf_get_val_bool(const struct gguf_context * ctx, int64_t key_id);
const char * gguf_get_val_str (const struct gguf_context * ctx, int64_t key_id);

Writing key-value metadata

// Add or overwrite a KV pair. The new pair is always appended.
void gguf_set_val_u8  (struct gguf_context * ctx, const char * key, uint8_t      val);
void gguf_set_val_i8  (struct gguf_context * ctx, const char * key, int8_t       val);
void gguf_set_val_u16 (struct gguf_context * ctx, const char * key, uint16_t     val);
void gguf_set_val_i16 (struct gguf_context * ctx, const char * key, int16_t      val);
void gguf_set_val_u32 (struct gguf_context * ctx, const char * key, uint32_t     val);
void gguf_set_val_i32 (struct gguf_context * ctx, const char * key, int32_t      val);
void gguf_set_val_f32 (struct gguf_context * ctx, const char * key, float        val);
void gguf_set_val_u64 (struct gguf_context * ctx, const char * key, uint64_t     val);
void gguf_set_val_i64 (struct gguf_context * ctx, const char * key, int64_t      val);
void gguf_set_val_f64 (struct gguf_context * ctx, const char * key, double       val);
void gguf_set_val_bool(struct gguf_context * ctx, const char * key, bool         val);
void gguf_set_val_str (struct gguf_context * ctx, const char * key, const char * val);

// Array variants.
void gguf_set_arr_data(struct gguf_context * ctx, const char * key,
                       enum gguf_type type, const void * data, size_t n);
void gguf_set_arr_str (struct gguf_context * ctx, const char * key,
                       const char ** data, size_t n);

// Remove a key (returns its former id, or -1 if not found).
int64_t gguf_remove_key(struct gguf_context * ctx, const char * key);

Working with tensors

// Query tensor count and look up tensors by name or index.
int64_t        gguf_get_n_tensors    (const struct gguf_context * ctx);
int64_t        gguf_find_tensor      (const struct gguf_context * ctx, const char * name);
size_t         gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
const char *   gguf_get_tensor_name  (const struct gguf_context * ctx, int64_t tensor_id);
enum ggml_type gguf_get_tensor_type  (const struct gguf_context * ctx, int64_t tensor_id);
size_t         gguf_get_tensor_size  (const struct gguf_context * ctx, int64_t tensor_id);

// Add a tensor (name must be unique).
void gguf_add_tensor(struct gguf_context * ctx, const struct ggml_tensor * tensor);

// Update a tensor's type and data.
void gguf_set_tensor_type(struct gguf_context * ctx, const char * name, enum ggml_type type);
void gguf_set_tensor_data(struct gguf_context * ctx, const char * name, const void * data);

Standardized metadata keys

Required keys

Key	Type	Description
`general.architecture`	`string`	Architecture identifier, e.g. `llama`, `gpt2`, `falcon`. Lowercase `[a-z0-9]+` only.
`general.quantization_version`	`uint32`	Required when any tensors are quantized.
`general.alignment`	`uint32`	Global alignment in bytes (must be a multiple of 8). Defaults to `32`.

General metadata

Key	Type	Description
`general.name`	`string`	Human-readable model name.
`general.author`	`string`	Author of the model.
`general.version`	`string`	Model version string.
`general.description`	`string`	Free-form description.
`general.license`	`string`	SPDX license expression, e.g. `MIT OR Apache-2.0`.
`general.tags`	`string[]`	Search terms.
`general.languages`	`string[]`	ISO 639 two-letter language codes.
`general.file_type`	`uint32`	Enumerated type of the majority of tensors.

LLM hyperparameters

For LLM architectures, replace [llm] with the architecture name (e.g. llama, gpt2):

Key	Type	Description
`[llm].context_length`	`uint64`	Maximum context length in tokens.
`[llm].embedding_length`	`uint64`	Embedding dimension (`n_embd`).
`[llm].block_count`	`uint64`	Number of transformer blocks.
`[llm].feed_forward_length`	`uint64`	Feed-forward layer size (`n_ff`).
`[llm].attention.head_count`	`uint64`	Number of attention heads.
`[llm].attention.head_count_kv`	`uint64`	KV heads for grouped-query attention.
`[llm].rope.dimension_count`	`uint64`	Rotary embedding dimensions.
`[llm].rope.freq_base`	`float32`	Base frequency for RoPE.

Tokenizer

Key	Type	Description
`tokenizer.ggml.model`	`string`	Tokenizer type: `llama`, `gpt2`, `replit`, `rwkv`.
`tokenizer.ggml.tokens`	`string[]`	Token list indexed by token ID.
`tokenizer.ggml.scores`	`float32[]`	Per-token scores/probabilities.
`tokenizer.ggml.merges`	`string[]`	BPE merge rules.
`tokenizer.ggml.bos_token_id`	`uint32`	Beginning-of-sequence token ID.
`tokenizer.ggml.eos_token_id`	`uint32`	End-of-sequence token ID.
`tokenizer.chat_template`	`string`	Jinja template for prompt formatting.

Naming convention

GGUF filenames follow this structure:

<BaseName>-<SizeLabel>-<FineTune>-<Version>-<Encoding>-<Type>-<Shard>.gguf

All components are separated by -. Components other than BaseName, SizeLabel, and Version are optional.

Component	Description	Example
`BaseName`	Model architecture or family name	`Llama-3`, `Mixtral`
`SizeLabel`	Parameter count with scale prefix (`K`, `M`, `B`, `T`)	`8B`, `8x7B`, `3.8B`
`FineTune`	Fine-tuning goal	`Instruct`, `Chat`
`Version`	Format `v<Major>.<Minor>` (default `v1.0`)	`v0.1`, `v2.0`
`Encoding`	Weight quantization scheme	`F16`, `Q4_0`, `Q5_K`
`Type`	File purpose: `LoRA` or `vocab`; omit for standard model files	`LoRA`
`Shard`	`<NNNNN>-of-<TOTAL>`, 5-digit zero-padded	`00001-of-00003`

At minimum, a filename should include BaseName, SizeLabel, and Version so that it can be validated unambiguously.

Examples

Filename	BaseName	SizeLabel	Version	Encoding	Shard
`Mixtral-8x7B-v0.1-KQ2.gguf`	`Mixtral`	`8x7B`	`v0.1`	`KQ2`	—
`Hermes-2-Pro-Llama-3-8B-F16.gguf`	`Hermes-2-Pro-Llama-3`	`8B`	`v1.0`	`F16`	—
`Grok-100B-v1.0-Q4_0-00003-of-00009.gguf`	`Grok`	`100B`	`v1.0`	`Q4_0`	`00003-of-00009`

Validation regex

You can validate a filename with the following regular expression:

^(?<BaseName>[A-Za-z0-9\s]*(?:(?:-(?:(?:[A-Za-z\s][A-Za-z0-9\s]*)|(?:[0-9\s]*)))*))\-(?:(?<SizeLabel>(?:\d+x)?(?:\d+\.)?\d+[A-Za-z](?:-[A-Za-z]+(\d+\.)?\d+[A-Za-z]+)?)(?:-(?<FineTune>[A-Za-z0-9\s-]+))?)?-(?:(?<Version>v\d+(?:\.\d+)*))(?:-(?<Encoding>(?!LoRA|vocab)[\w_]+))?(?:-(?<Type>LoRA|vocab))?(?:-(?<Shard>\d{5}-of-\d{5}))?\.gguf$

Standardized tensor names

Models using the transformer architecture should use these tensor name conventions: Base layers — AA.weight / AA.bias where AA is:

Name	Layer
`token_embd`	Token embedding
`pos_embd`	Position embedding
`output_norm`	Output normalization
`output`	Output projection

Attention and feed-forward blocks — blk.N.BB.weight / blk.N.BB.bias where N is the block index and BB is:

Name	Layer
`attn_norm`	Attention normalization
`attn_q`	Query projection
`attn_k`	Key projection
`attn_v`	Value projection
`attn_qkv`	Fused QKV projection
`attn_output`	Attention output
`ffn_norm`	Feed-forward normalization
`ffn_up`	FFN up-projection
`ffn_gate`	FFN gate
`ffn_down`	FFN down-projection

Version history

Version	Changes
v1	Initial version.
v2	Most countable fields changed from `uint32` to `uint64` for larger model support.
v3	Added big-endian support.

Get Started

Core Concepts

Backends

Training

File Formats

Examples

GGUF file format

Design goals

File structure

Header

Tensor info

Alignment

Metadata types

Key-value pairs

Tensor element types

C API

Initializing a context

Writing files

Reading key-value metadata

Writing key-value metadata

Working with tensors

Standardized metadata keys

Required keys

General metadata

LLM hyperparameters

Tokenizer

Naming convention

Examples

Validation regex

Standardized tensor names

Version history

Get Started

Core Concepts

Backends

Training

File Formats

Examples

​Design goals

​File structure

​Header

​Tensor info

​Alignment

​Metadata types

​Key-value pairs

​Tensor element types

​C API

​Initializing a context

​Writing files

​Reading key-value metadata

​Writing key-value metadata

​Working with tensors

​Standardized metadata keys

​Required keys

​General metadata

​LLM hyperparameters

​Tokenizer

​Naming convention

​Examples

​Validation regex

​Standardized tensor names

​Version history

Design goals

File structure

Header

Tensor info

Alignment

Metadata types

Key-value pairs

Tensor element types

C API

Initializing a context

Writing files

Reading key-value metadata

Writing key-value metadata

Working with tensors

Standardized metadata keys

Required keys

General metadata

LLM hyperparameters

Tokenizer

Naming convention

Examples

Validation regex

Standardized tensor names

Version history