NetStateObject
NetStateObject[net]
creates an object derived from net that represents a neural net with additional stored state information that is updated when the net is applied to data.
NetStateObject[net,seed]
creates an object in which additional stored state information is initialized using seed.
Details
- NetStateObject[…][data] updates stored state information in the NetStateObject.
- State information is associated with the state ports of recurrent net layers such as LongShortTermMemoryLayer.
- NetStateObject will not store the state of layers whose state ports are initialized from other layers in a NetGraph.
- When a seed is not provided, initial values for recurrent states will consist of arrays of zeros.
- The current value of the stored states is given by NetExtract[NetStateObject[…],"States"].
Examples
open all close allBasic Examples (1)
Create a NetStateObject from an initialized recurrent net:
sobj = NetStateObject@NetInitialize@BasicRecurrentLayer[1, "Input" -> {"Varying", 1}]Evaluate the state object on some data:
sobj[{{1}, {3}, {2}, {2}}]Due to the presence of the stored states, the behavior of the state object can change between evaluations, even on the same input:
sobj[{{1}}]sobj[{{1}}]Scope (1)
Applications (1)
Training an English character-level language model. First, create 300,000 training examples of 25 characters each from two novels:
corpus = StringJoin[ResourceData["The American"], " ", ResourceData["Don Quixote - English"]];sampleCorpus[corpus_, len_, num_] := Module[
{positions = RandomInteger[{len + 1, StringLength[corpus] - 1}, num]},
Thread[StringTake[corpus, {# - len, #}& /@ positions] -> StringPart[corpus, positions + 1]]];
trainingData = sampleCorpus[corpus, 25, 300000];The data is of the form of a classification problem: given a sequence of characters, predict the next one. A sample of the data:
RandomSample[trainingData, 5]//Map[InputForm]//TableFormObtain the list of all characters in the text:
characters = Union[Characters[corpus]]Define a net that takes in a string of characters and returns a prediction for the next character:
net = NetChain[{
UnitVectorLayer[],
GatedRecurrentLayer[128],
GatedRecurrentLayer[128],
SequenceLastLayer[],
LinearLayer[Length[characters]],
SoftmaxLayer[]},
"Input" -> NetEncoder[{"Characters", characters}],
"Output" -> NetDecoder[{"Class", characters}]
];Train the net. This can take up to an hour on a CPU; it is recommended to set option TargetDevice to "GPU" if a supported GPU is available:
trainedNet = NetTrain[net, trainingData, ValidationSet -> Scaled[0.1], BatchSize -> 64, TargetDevice -> "GPU"]Predict the next character, given a sequence of characters:
trainedNet["Why are w"]Generate 100 characters of text, given starting text:
generate[net_, start_, len_] :=
StringJoin@NestList[NetStateObject[net], start, len]generate[trainedNet, "Why are w", 100]One can get more interesting text by sampling from the probability distribution of predictions:
generateSample[net_, start_, len_] := Block[{sobj = NetStateObject[net]},
StringJoin@NestList[sobj[#, {"RandomSample", "Temperature" -> 0.5}]&, start, len]]generateSample[trainedNet, "Why are w", 100]Properties & Relations (3)
If the initial value of a recurrent layer's state is provided by a connection in a NetGraph, that state will not be stored by NetStateObject.
Create a graph that uses a connection to provide the initial value of the state of a BasicRecurrentLayer:
graph = NetInitialize@NetGraph[
{BasicRecurrentLayer[2, "Input" -> {"Varying", 2}]},
{NetPort["State"] -> NetPort[1, "State"]}]This graph cannot be used inside a NetStateObject, as there are no states left to store:
NetStateObject[graph]
The current value of the stored states can be obtained using NetExtract.
First create a NetStateObject:
net = NetInitialize@BasicRecurrentLayer[2, "Input" -> {"Varying", 2}];
sobj = NetStateObject[net]Apply the object to some data:
sobj[{{1, 2}, {2, 0}, {0, 1}}]Extract the current value of the states:
NetExtract[sobj, "States"]For recurrent nets, using a NetStateObject is equivalent to manually keeping track of the recurrent states via NetPort[All,"States"].
To see this, create a classifier that predicts the next element of a sequence:
chain = NetChain[{EmbeddingLayer[3], BasicRecurrentLayer[3], SequenceLastLayer[], SoftmaxLayer[]}, "Output" -> NetDecoder["Class"]]Train the classifier on a set of input sequences:
trained = NetTrain[chain, {{1, 2} -> 3, {2, 3} -> 1, {3, 1} -> 2}]Create a state object and use it to efficiently generate a maximum-likelihood sequence, starting from a single 1:
sobj = NetStateObject[trained]NestList[sobj[{#}]&, 1, 8]Generate from the trained net using NetPort[All,"States"] to set and get the recurrent states, which yields the same result:
state = None;
NestList[Function[
{output, state} = Values@trained[<|"Input" -> {#}, NetPort[All, "States"] -> state|>, {NetPort@"Output", NetPort[All, "States"]}];
output], 1, 8]Text
Wolfram Research (2018), NetStateObject, Wolfram Language function, https://reference.wolfram.com/language/ref/NetStateObject.html.
CMS
Wolfram Language. 2018. "NetStateObject." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/NetStateObject.html.
APA
Wolfram Language. (2018). NetStateObject. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/NetStateObject.html