Wolfram Language & System Documentation Center

LLMGraphSubmit

evaluates an LLMGraph asynchronously on the input input.

LLMGraphSubmit[LLMGraph[…],input,target]

specifies the outputs to compute.

Details and Options

LLMGraphSubmit is used to schedule an asynchronous LLMGraph evaluation with granular access to the details of the computation.
LLMGraphSubmit requires external service authentication, billing and internet connectivity.
Possible values for input are:
val value for the single graph input

<|name₁val₁, …|> values for each of the nodes or inputs name_i
When specifying the nodes to compute, possible values for target are:
Automatic the output nodes (default)

All all the nodes in a graph

{name_i,…} a subset of the graph nodes
When the computation of a node is required, all its dependencies are computed first.
The following options can be specified:

Authentication	Inherited	authentication scheme
HandlerFunctions		how to handle generated events
HandlerFunctionsKeys	Automatic	parameters to supply to handler functions
LLMEvaluator	$LLMEvaluator	LLM configuration to use

During the asynchronous execution of LLMGraphSubmit, events can be generated.
Events triggered by the graph processing:

	"NodeCanceled"	a node computation has been canceled
	"NodeFailed"	a node computation has failed
	"NodeSubmitted"	a node has been submitted for computation
	"NodeSynthesized"	a node result has been computed
	"ResultGenerated"	all the graph output nodes have been computed

Events triggered by the task framework:

	"FailureOccurred"	failure is generated during the computation
	"TaskFinished"	task is completely finished
	"TaskRemoved"	task is being removed
	"TaskStarted"	task is started
	"TaskStatusChanged"	task status changed

HandlerFunctionsf uses f for all the events.
With the specification HandlerFunctions-><|…,"event_i"->f_i,…|>, f_i[assoc] is evaluated whenever event_i is generated. The elements of assoc have keys specified by the setting for HandlerFunctionsKeys.
Possible keys specified by HandlerFunctionsKeys include:

	"EventName"	the name of the event being handled
	"Failure"	failure object generated if execution failed
	"CurrentNode"	the node that prompted the event
	"NodeResult"	the result of "CurrentNode"
	"LLMGraph"	current evaluation graph
	"GraphResults"	all the node outputs
	"Task"	the task object generated by LLMGraphSubmit
	"TaskStatus"	the status of the task
	{key₁,…}	a list of keys
	All	all keys
	Automatic	figures out the keys from HandlerFunctions

Values that have not yet been received are given as Missing["NotAvailable"].

Examples

open all close all

Basic Examples (3)

Submit a basic graph:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[<|"Poet1" -> "write a short poem about summer", "Poet2" -> "write a haiku about winter", "Judge" -> "Choose the best composition of your among these:

1) Poem1: `Poet1`

2) Poem2: `Poet2` 

and copy it:"|>], HandlerFunctions -> <|"TaskFinished" -> Function[bestpoem = #GraphResults]|>]

Inspect the output node result:

Wolfram Language code: bestpoem

Submit a graph providing an evaluation input:

Wolfram Language code:

task = LLMGraphSubmit[LLMGraph[<|"haiku" -> "generate a haiku about `Topic`.", "complete" -> "add an extra stanza to the hokku `haiku` to make it a renga."|>], <|"Topic" -> "spring"|>, HandlerFunctions -> <|"TaskFinished" -> Function[renga = #GraphResults;computed = #LLMGraph]|>]

Inspect the output node result:

Wolfram Language code: renga

Inspect the generated LLMGraph, including all computation results:

Wolfram Language code: computed

Retrieve dynamically generated results for each node:

Wolfram Language code:

results = <||>;
LLMGraphSubmit[LLMGraph[<|"haiku" -> "generate a haiku about `Topic`.", "complete" -> "add an extra stanza to the hokku `haiku` to make it a renga."|>], 
	<|"Topic" -> "autumn"|>, HandlerFunctions -> <|"NodeSynthesized" -> Function[AssociateTo[results, #CurrentNode -> #NodeResult]]|>];

Show all the generation steps:

Wolfram Language code: results

Scope (8)

Evaluation (8)

Node Initialization (5)

Provide the input node values to start the graph evaluation:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"Arg1" -> 1, "Arg2" -> 2|>, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|> ]

Wolfram Language code: results

If a single input is required, it is not necessary to use the Association:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "MiddleNode" -> Association["Input" -> {"Arg1"}, 
      "EvaluationFunction" -> ("intermediate result" & ), "Name" -> "MiddleNo ... tomatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> Automatic, "StopTokens" -> Automatic, "ToolMethod" -> Automatic, 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.3"]], "Authentication" -> Automatic, 
  "LLMPacletVersion" -> "2.3"]], 1, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results:

Wolfram Language code: results

This is equivalent to:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "MiddleNode" -> Association["Input" -> {"Arg1"}, 
      "EvaluationFunction" -> ("intermediate result" & ), "Name" -> "MiddleNo ... tomatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> Automatic, "StopTokens" -> Automatic, "ToolMethod" -> Automatic, 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.3"]], "Authentication" -> Automatic, 
  "LLMPacletVersion" -> "2.3"]], <|"Arg1" -> 1|>, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

If no input is required, the graph can be evaluated on an empty Association:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["OutputNode1" -> Association["Input" -> {}, 
      "EvaluationFunction" -> ("Output node 1 result" & ), "Name" -> "OutputNode1", 
      "PendingDependencies" -> {}], "MiddleNode" -> Association["Input" -> ... tomatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> Automatic, "StopTokens" -> Automatic, "ToolMethod" -> Automatic, 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.3"]], "Authentication" -> Automatic, 
  "LLMPacletVersion" -> "2.3"]], <||>, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results:

Wolfram Language code: results

Submit an LLMGraph for partial evaluation, providing only a subset of the input nodes' values:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... tomatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> Automatic, "StopTokens" -> Automatic, "ToolMethod" -> Automatic, 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.3"]], "Authentication" -> Automatic, 
  "LLMPacletVersion" -> "2.3"]], <|"Arg2" -> 1|>]

Bypass the node function evaluation, providing a result for the intermediate nodes:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"OutputNode1" -> "custom result", "Arg2" -> 2|>, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results:

Wolfram Language code: results

Evaluation Properties (3)

Evaluate graph output nodes' results:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"Arg1" -> 1, "Arg2" -> 2|>, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results:

Wolfram Language code: results

This is equivalent to:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"Arg1" -> 1, "Arg2" -> 2|>, Automatic, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Request all nodes' results from an LLMGraph evaluation in the "GraphResult":

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"Arg1" -> 1, "Arg2" -> 2|>, All, HandlerFunctions -> <|"TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results, including input and intermediate nodes:

Wolfram Language code: results

Request a subset of an LLMGraph node results in the "GraphResult":

Wolfram Language code:

timeline = {};TaskWait@LLMGraphSubmit[LLMGraph[Association["Nodes" -> Association["Arg1" -> Association["Name" -> "Arg1", 
      "PendingDependencies" -> {}], "Arg2" -> Association["Name" -> "Arg2", 
      "PendingDependencies" -> {}], "OutputNode1" -> Association["Input" -> {"Arg1"},  ... c, "TotalProbabilityCutoff" -> Automatic, "Prompts" -> Automatic, 
     "PromptDelimiter" -> "\n\n", "StopTokens" -> Automatic, "ToolMethod" -> "Service", 
     "Tools" -> Automatic, "LLMPacletVersion" -> "2.2.3"]], "LLMPacletVersion" -> "2.2.3"]], <|"Arg1" -> 1, "Arg2" -> 2|>, {"OutputNode1"}, HandlerFunctions -> <|"NodeSubmitted" -> Function[AppendTo[timeline, #CurrentNode  -> "submission"]], "NodeSynthesized" -> Function[AppendTo[timeline, #CurrentNode  -> "has run"]], "TaskFinished" -> Function[results = #GraphResults]|>]

Inspect the results:

Wolfram Language code: results

Only required ancestors up to the closest known node result were lazily added to the evaluation stack and run:

Wolfram Language code: timeline

Options (14)

Authentication (2)

Specify the Authentication option values of the LLMGraph[…] are to be used:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[<|"LLMCall1" -> "Hello preferences' service!", "LLMCall2" -> LLMFunction["Hello OpenAI!", LLMEvaluator -> LLMConfiguration[<|"Model" -> {"OpenAI", Automatic}|>]]|>
	], 
	Authentication -> Inherited
	]

Use an explicit Authentication setting to override what is stored in the LLMGraph[…]:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[<|"LLMCall1" -> "Hello service!", "LLMCall2" -> "Hello service!"|>
	], 
	Authentication -> <|"APIKey" -> "1234abcd"|>
	]

HandlerFunctions (6)

LLMGraph Evaluation Events (5)

Check for node function submission and return:

Wolfram Language code:

timeline = {};TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> "Hi"|>], HandlerFunctions -> <|
	"NodeSubmitted" -> Function[AppendTo[timeline, #CurrentNode  -> "Evaluation started."]], 
	"NodeSynthesized" -> Function[AppendTo[timeline, #CurrentNode  -> #NodeResult]]
	 |>
	]

Wolfram Language code: timeline

Check for node canceled by the "TestFunction" result:

Wolfram Language code:

timeline = {};TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> <|"LLMFunction" -> "Hi", "TestFunction" -> Function@TrueQ[#runCall]|>|>], <|"runCall" -> False|>, HandlerFunctions -> <|
	"NodeCanceled" -> Function[AppendTo[timeline, #CurrentNode  -> "Evaluation canceled"]]
	 |>
	]

Wolfram Language code: timeline

Check for node function failure: node failure prevents children nodes submission:

Wolfram Language code:

timeline = {};TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> "Hi"|>], LLMEvaluator -> LLMConfiguration["Model" -> {"OpenAI", "WrongModelName"}], HandlerFunctions -> <|
	"NodeFailed" -> Function[AppendTo[timeline, #CurrentNode  -> #Failure]]
	 |>
	]

Wolfram Language code: timeline

Save the evaluation results when the graph computation has reached the end:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> "Hi"|>], HandlerFunctions -> <|
	"ResultGenerated" -> Function[res = #GraphResults]
	 |>
	]

Wolfram Language code: res

Save partial evaluation results even when the graph computation could not be completed:

Wolfram Language code:

TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> "Hi", "ExtraStep" -> Function[ToLowerCase[#LLMCall]]|>], <||>, All, 
	LLMEvaluator -> LLMConfiguration["Model" -> {"OpenAI", "AWrongModelName"}], 
	HandlerFunctions -> <|
	"TaskFinished" -> Function[results = #GraphResults]
	 |>
	]

Wolfram Language code: results

Task Framework Events (1)

Attach a handler function to each step of the task life cycle:

Wolfram Language code:

timeline = {};TaskWait@LLMGraphSubmit[LLMGraph[<|"LLMCall" -> "Hi"|>], HandlerFunctions -> <|
	"TaskStatusChanged" -> Function[AppendTo[timeline, "Task status changed"]], 
	"TaskStarted" -> Function[AppendTo[timeline, {#EventName, #TaskStatus}]], 
	"TaskFinished" -> Function[AppendTo[timeline, {#EventName, #TaskStatus}]], 
	"TaskRemoved" -> Function[AppendTo[timeline, {#EventName, #TaskStatus}]], 
	"FailureOccurred" -> Function[AppendTo[timeline, {#EventName, #TaskStatus}]]
	 |>
	]

Wolfram Language code: timeline

HandlerFunctionsKeys (4)

Specify that the handler functions argument must contain only the current "TaskStatus" field:

Wolfram Language code:

status = {};
task = LLMGraphSubmit[
	LLMGraph[<|"LLMCall" -> "Hi"|>], 
	HandlerFunctions -> <|
	"TaskStatusChanged" -> Function[AppendTo[status, #]]
	 |>, 
	HandlerFunctionsKeys -> "TaskStatus"
	]
TaskWait@task;
status

Specify a list of keys that are provided to the handler functions:

Wolfram Language code:

chekpoints = {};
task = LLMGraphSubmit[
	LLMGraph[<|"LLMCall" -> "Hi"|>], 
	HandlerFunctions -> <|
	"TaskStatusChanged" -> Function[AppendTo[chekpoints, #]]
	 |>, 
	HandlerFunctionsKeys -> {"TaskStatus", "LLMGraph"}
	]
TaskWait@task;
chekpoints

Include all available keys in the Association provided to the handler functions:

Wolfram Language code:

task = LLMGraphSubmit[
	LLMGraph[<|"LLMCall" -> "Hi"|>], 
	HandlerFunctions -> <|
	"TaskFinished" -> Function[values = #]
	 |>, 
	HandlerFunctionsKeys -> All
	]
TaskWait@task;
values

Automatically find the keys using the function slots:

Wolfram Language code:

task = LLMGraphSubmit[
	LLMGraph[<|"LLMCall" -> "Hi"|>], 
	HandlerFunctions -> <|
	"TaskFinished" -> Function[res = #GraphResults; graph = #LLMGraph]
	 |>, 
	HandlerFunctionsKeys -> Automatic
	]
TaskWait@task;
{res, graph}

LLMEvaluator (2)

With the default value LLMEvaluatorInherited, LLMGraphSubmit is using the LLMGraph evaluator:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[<|"LLMCall1" -> "Hello preferences' service!", "LLMCall2" -> LLMFunction["Hello OpenAI!", LLMEvaluator -> LLMConfiguration[<|"Model" -> {"OpenAI", Automatic}|>]]|>
	], 
	LLMEvaluator -> Inherited
	]

Use LLMEvaluatorconfig to override the LLMGraph[…] evaluator:

Wolfram Language code:

LLMGraphSubmit[LLMGraph[<|"LLMCall1" -> "Hello service!", "LLMCall2" -> LLMFunction["Hello OpenAI!", LLMEvaluator -> LLMConfiguration[<|"Model" -> {"OpenAI", Automatic}|>]]|>
	], 
	LLMEvaluator -> LLMConfiguration[<|"Model" -> {"Anthropic", Automatic}|>]
	]

Top

More Learning

Tech Support

Wolfram Solutions

Wolfram Solutions For Education

Get Started

Grow Your Skills

Work with Us

Educational Programs for Adults

Educational Programs for Youth

Read

LLMGraphSubmit

Details and Options

Examples

Basic Examples (3)

Scope (8)

Evaluation (8)

Node Initialization (5)

Evaluation Properties (3)

Options (14)

Authentication (2)

HandlerFunctions (6)

LLMGraph Evaluation Events (5)

Task Framework Events (1)

HandlerFunctionsKeys (4)

LLMEvaluator (2)

Text

CMS

APA

BibTeX

BibLaTeX

LLMGraphSubmit

Details and Options

Examples

Basic Examples (3)

Scope (8)

Evaluation (8)

Node Initialization (5)

Evaluation Properties (3)

Options (14)

Authentication (2)

HandlerFunctions (6)

LLMGraph Evaluation Events (5)

Task Framework Events (1)

HandlerFunctionsKeys (4)

LLMEvaluator (2)

See Also

Related Guides

History

Text

CMS

APA

BibTeX

BibLaTeX