WOLFRAM

Wolfram Language & System Documentation Center

GGUF (.gguf)

  • Import fully supports the GGUF standard.

Background & Context

    • Open format designed for the fast loading and saving of large language models.
    • Stores models and various metadata.
    • GGUF is an acronym for GPT-Generated Unified Format.
    • Binary file format successor of GGML.
    • Released in 2023 by Georgi Gerganov.

Import

  • Import["file.gguf"] imports a GGUF file, returning a NetExternalObject.
  • Import["file.gguf",elem] imports the specified elements.
  • The import format can be specified with Import["file","GGUF"] or Import["file",{"GGUF",elem,…}].
  • See the following reference pages for full general information:
  • Importimport from a file
    CloudImportimport from a cloud object
    ImportStringimport from a string
    ImportByteArrayimport from a byte array

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Import elements include:
  • "NetExternalObject"NetExternalObject representation of the net

Options

  • Import options:
  • "ContextWindowSize"Automaticsize of the model's context window
    "Output""Text"output type of the model
  • Options "Output" can be set to the following:
  • "Embeddings"import the file as a text-embedding model
    "Text"import the file as a text-generation model
  • Not all models support both output types.
  • Option "ContextWindowSize" can be set to a positive integer or to Automatic, which selects the model's default.

Examples

open all close all

Basic Examples  (2)

Import a net in GGUF format (model credit: Olusegun Odewole, https://huggingface.co/segestic/Tinystories-gpt-0.1-3m ):

Wolfram Language code: net = Import["ExampleData/gguf_example.gguf"]

Show the Import elements available in this file:

Wolfram Language code: Import["ExampleData/gguf_example.gguf", "Elements"]

Run the model until a termination token is reached:

Wolfram Language code: net["There was", Method -> {"StopTokens" -> {"."}}]

Import a "GGUF" file as an embedding model:

Wolfram Language code: net = Import["ExampleData/gguf_example.gguf", "Output" -> "Embeddings"]

Generate an embedding vector for each token in a string:

Wolfram Language code: net["There was a huge oven"]//Short

Import Options  (2)

"ContextWindowSize"  (1)

Import a "GGUF" file:

Wolfram Language code: original = Import["ExampleData/gguf_example.gguf"]

Check the default context window size:

Wolfram Language code: Information[original, "ContextWindowSize"]

Specify a different maximum value:

Wolfram Language code: modified = Import["ExampleData/gguf_example.gguf", "ContextWindowSize" -> 1024]

Inspect the new value in the model information:

Wolfram Language code: Information[modified, "ContextWindowSize"]

Retrieve the original training-time context window size:

Wolfram Language code: Information[modified, "TrainingContextWindowSize"]

"Output"  (1)

By default, models are imported as text generators:

Wolfram Language code: net = Import["ExampleData/gguf_example.gguf"]

Import the model as a text embedder instead:

Wolfram Language code: Import["ExampleData/gguf_example.gguf", "Output" -> "Embeddings"]