WOLFRAM

Wolfram Language & System Documentation Center

LLMEvaluator

is an option for functions such as LLMSynthesize that specifies the LLM configuration.

Details
Details and Options Details and Options
Examples  
Basic Examples  
Scope  
Applications  
See Also
Related Guides
History
Cite this Page

LLMEvaluator

This functionality requires LLM access »

LLMEvaluator

is an option for functions such as LLMSynthesize that specifies the LLM configuration.

Details

  • LLMEvaluator is used to specify the properties of an LLM task including model, special prompts, sampling strategy, etc.
  • Supported settings for LLMEvaluator include:
  • "name"a valid configuration name
    propvala property specification
    <|prop1val1,…|>several property specifications
    LLMConfiguration[…]a valid LLMConfiguration
  • Unspecified properties are set to the values specified by $LLMEvaluator.
  • Valid configuration names includes:
  • "WolframAIAccess"use an LLM model via Wolfram
    "WolframAIAssistant"use the prompts and tools of the Wolfram AI Assistant
    "AgentOne"use the prompts and tools of the Agent One API
  • Supported properties include:
  • "MaxTokens"maximum amount of tokens to generate
    "Model"base model
    "PromptDelimiter"string to insert between prompts
    "Prompts"initial prompts or LLMPromptGenerator objects
    "Reasoning"reasoning settings
    "StopTokens"tokens on which to stop generation
    "Temperature"sampling temperature
    "ToolMethod"method to use for tool calling
    "Tools"list of LLMTool objects to make available
    "TopProbabilities"sampling classes cutoff
    "TotalProbabilityCutoff"sampling probability cutoff (nucleus sampling)
  • Valid property settings are the same as LLMConfiguration.

Examples

open all close all

Basic Examples  (3)

Run an LLM with the Wolfram computationally augmented generation (CAG):

Wolfram Language code: LLMSynthesize["what is the tallest mountain?", LLMEvaluator -> "AgentOne"]

Run an LLM with an evaluator that specifies a prompt:

Wolfram Language code: LLMSynthesize["What is the tallest mountain?", LLMEvaluator -> <|"Prompts" -> LLMPrompt["ELI5"]|>]

Start a chat conversation with an evaluator that specifies a prompt:

Wolfram Language code: ChatEvaluate[ChatObject[LLMEvaluator -> <|"Prompts" -> LLMPrompt["ELI5"]|>], "What's the tallest mountain?"]

Scope  (10)

Specify a token limit to the LLM-generated text:

Wolfram Language code: LLMSynthesize["what is the tallest mountain ?", LLMEvaluator -> <|"MaxTokens" -> 10|>]

Specify the service and the model to use for the generation:

Wolfram Language code: LLMSynthesize["What's the plural of mouse?", LLMEvaluator -> <|"Model" -> <|"Service" -> "Anthropic", "Name" -> "claude-haiku-4-5" |>|>]

Specify several prompts and how to join them together before submitting them to the LLM:

Wolfram Language code: LLMSynthesize["What's the plural of mouse?", "PromptText" , LLMEvaluator -> <|"Prompts" -> {LLMPrompt["ELI5"], "Always answer with a single sentence"}, "PromptDelimiter" -> " ----- " |>]

Specify that the sampling should be performed at zero temperature:

Wolfram Language code: Table[LLMSynthesize["What is the plural of mouse? One sentence.", LLMEvaluator -> <|"Temperature" -> 0|>], 6]

Specify the maximum cumulative probability before cutting off the distribution (aka nucleus sampling):

Wolfram Language code: LLMSynthesize["What's the plural of mouse?", LLMEvaluator -> <|"TotalProbabilityCutoff" -> .2|>]

Specify the number of top-probability tokens to sample from:

Wolfram Language code: LLMSynthesize["What's the plural of mouse?", LLMEvaluator -> <|"TopProbabilities" -> 50, "Model" -> <|"Service" -> "Anthropic", "Name" -> "claude-haiku-4-5" |>|>]

Specify one or more alternative strings that will stop the LLM generation process:

Wolfram Language code: LLMSynthesize["What's the plural of mouse?", LLMEvaluator -> <|"StopTokens" -> {"mouse"}|>]

Specify a tool that the LLM can call if needed:

Wolfram Language code: ChatEvaluate[ChatObject[LLMEvaluator -> <|"Tools" -> {LLMTool[{"altimeter", "gives the altitude at a location"}, {"where" -> "Location"}, GeoElevationData[#where]&], LLMTool[{"thermometer", "gives the current temperature at a location"}, {"where" -> "Location"}, AirTemperatureData[#where]&]}|>], "what are Rome's altitude and the current temperature?"]

Specify that tool calls should attempt to use the native API mechanism:

Wolfram Language code: ChatEvaluate[ChatObject[LLMEvaluator -> <| "ToolMethod" -> "Service", "Tools" -> {LLMTool[{"altimeter", "gives the altitude at a location"}, {"where" -> "Location"}, GeoElevationData[#where]&], LLMTool[{"thermometer", "gives the current temperature at a location"}, {"where" -> "Location"}, AirTemperatureData[#where]&]}|>], "what are Rome's altitude and the current temperature?"]

Specify multiple properties of a configuration:

Wolfram Language code: LLMSynthesize["Which is bigger, Chicago or Boston? Use the tool to check.", LLMEvaluator -> <|"Prompts" -> LLMPrompt["ELI5"], "Tools" -> LLMTool[...]|>]

Applications  (1)

Use the Wolfram CAG to get information about the language:

Wolfram Language code: text = LLMSynthesize["How do I plot a function?", LLMEvaluator -> "WolframAIAssistant"]

Import the string as markdown to render it:

Wolfram Language code: ImportString[text, {"Markdown", "NotebookObject"}]
Wolfram Research (2023), LLMEvaluator, Wolfram Language function, https://reference.wolfram.com/language/ref/LLMEvaluator.html (updated 2026).

Text

Wolfram Research (2023), LLMEvaluator, Wolfram Language function, https://reference.wolfram.com/language/ref/LLMEvaluator.html (updated 2026).

CMS

Wolfram Language. 2023. "LLMEvaluator." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2026. https://reference.wolfram.com/language/ref/LLMEvaluator.html.

APA

Wolfram Language. (2023). LLMEvaluator. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/LLMEvaluator.html

BibTeX

@misc{reference.wolfram_2026_llmevaluator, author="Wolfram Research", title="{LLMEvaluator}", year="2026", howpublished="\url{https://reference.wolfram.com/language/ref/LLMEvaluator.html}", note=[Accessed: 13-June-2026]}

BibLaTeX

@online{reference.wolfram_2026_llmevaluator, organization={Wolfram Research}, title={LLMEvaluator}, year={2026}, url={https://reference.wolfram.com/language/ref/LLMEvaluator.html}, note=[Accessed: 13-June-2026]}