NetBidirectionalOperator
represents a net that applies net to a sequence and to its reverse, concatenating both results into one output sequence.
NetBidirectionalOperator[{fnet,bnet}]
uses fnet on the normal input and bnet on the reversed input.
NetBidirectionalOperator[nets,agg]
aggregates the two output sequences using the specified aggregation function.
Details and Options
- In NetBidirectionalOperator[net], net should be a net that takes a sequence and produces a sequence. net is often one of the recurrent layers BasicRecurrentLayer, GatedRecurrentLayer, LongShortTermMemoryLayer or NetFoldOperator.
- NetBidirectionalOperator[net] creates two copies of net that are trained independently, and is equivalent to NetBidirectionalOperator[{net,net}].
- In NetBidirectionalOperator[…,agg], valid settings for agg include:
-
Catenate catenate the forward and reversed outputs (default) Total add the forward and reversed outputs Mean take the mean of the forward and reversed outputs - The following training parameter can be included:
-
LearningRateMultipliers Automatic learning rate multipliers for trainable arrays in the net - NetBidirectionalOperator[net,Catenate] will concatenate each element of the forward and reversed outputs, rather than concatenating the sequences themselves.
- If net produces a sequence of vectors of length d, NetBidirectionalOperator[net,Catenate] will produce a sequence of vectors of length 2×d. For the aggregation functions Total and Mean, the vectors will be of length d.
- For the reversed application of net, the input is first reversed, then net is applied normally, then the output is again reversed if net produces a sequence of vectors, before combination with the forward application of net.
- NetExtract allows access to the forward and reverse nets via "ForwardNet" and "BackwardNet".
- The states of the forward and reverse nets are accessible in NetBidirectionalOperator[…] with the same names prefixed respectively by "Forward…" and "Backward…".
- Options[NetBidirectionalOperator] gives the list of default options to construct the operator. Options[NetBidirectionalOperator[…]] gives the list of default options to evaluate the operator on some data.
- Information[NetBidirectionalOperator[…]] gives a report about the operator.
- Information[NetBidirectionalOperator[…],prop] gives the value of the property prop of NetBidirectionalOperator[…]. Possible properties are the same as for NetGraph.
Examples
open all close allBasic Examples (4)
Create a bidirectional operator that contains two LSTM layers:
net = LongShortTermMemoryLayer[3, "Input" -> {"Varying", 2}];
bidir = NetInitialize@NetBidirectionalOperator[net]Apply the operator to an input:
bidir[{{1, 2}, {0, 1}, {0, 2}}]Create a bidirectional operator using different nets for the forward and backward passes:
forward = BasicRecurrentLayer[2, "Input" -> {"Varying", 2}];
backward = BasicRecurrentLayer[4, "Input" -> {"Varying", 2}];
bidir = NetInitialize@NetBidirectionalOperator[{forward, backward}]Apply the operator to an input:
bidir[{{1, 2}, {0, 1}, {0, 2}}]Create a bidirectional operator with a specific aggregation function:
net = LongShortTermMemoryLayer[3, "Input" -> {"Varying", 2}];
bidir = NetInitialize@NetBidirectionalOperator[net, Total]Apply the operator to an input:
bidir[{{1, 2}, {0, 1}, {0, 2}}]Create an operator that yields the final elements of two sequences processed by a forward and a backward recurrent layer:
rnn = LongShortTermMemoryLayer[4, "Input" -> {"Varying", 2}]bidir = NetInitialize@NetBidirectionalOperator[NetChain[{rnn, SequenceLastLayer[]}], Catenate]Apply the operator to an input:
bidir[{{1, 2}, {0, 1}, {0, 2}}]Scope (2)
Create a bidirectional recurrent network where initial states in both directions are exposed:
bidir = NetInitialize@NetGraph[{NetBidirectionalOperator[GatedRecurrentLayer[2]]}, {NetPort["ForwardState"] -> NetPort[1, "ForwardState"], NetPort["BackwardState"] -> NetPort[1, "BackwardState"]}, "Input" -> {"Varying", 3}]bidir[<|"Input" -> RandomReal[1, {5, 3}], "ForwardState" -> {0, 1}, "BackwardState" -> {1, 0}|>]Create a bidirectional recurrent network where initial states in both directions can be learned:
bidir = NetInitialize@NetGraph[<|"RNN" -> NetBidirectionalOperator[GatedRecurrentLayer[2]], "InitialForwardState" -> NetArrayLayer[], "InitialBackwardState" -> NetArrayLayer[]|>, {"InitialForwardState" -> NetPort["RNN", "ForwardState"], "InitialBackwardState" -> NetPort["RNN", "BackwardState"]}, "Input" -> {"Varying", 3}]bidir[RandomReal[1, {5, 3}]]Properties & Relations (2)
Consider a network that takes and produces a sequence of vectors:
net = LongShortTermMemoryLayer[10]NetBidirectionalOperator[net,Catenate] is equivalent to the following NetGraph:
NetGraph[<|"ForwardNet" -> net, "Reverse" -> SequenceReverseLayer[], "BackwardNet" -> net, "Reverse2" -> SequenceReverseLayer[], "Catenate" -> CatenateLayer[2]|>, {{"ForwardNet", "Reverse" -> "BackwardNet" -> "Reverse2"} -> "Catenate"}]NetBidirectionalOperator[net,Total] is equivalent to the following NetGraph:
NetGraph[<|"ForwardNet" -> net, "Reverse" -> SequenceReverseLayer[], "BackwardNet" -> net, "Reverse2" -> SequenceReverseLayer[], "Total" -> ThreadingLayer[Plus]|>, {{"ForwardNet", "Reverse" -> "BackwardNet" -> "Reverse2"} -> "Total"}]Consider a network that takes a sequence of vectors and produces a fixed-size vector:
net = NetChain[{LongShortTermMemoryLayer[10], AggregationLayer[Max, 1]}]NetBidirectionalOperator[net,Catenate] is equivalent to the following NetGraph:
NetGraph[<|"ForwardNet" -> net, "Reverse" -> SequenceReverseLayer[], "BackwardNet" -> net, "Catenate" -> CatenateLayer[1]|>, {{"ForwardNet", "Reverse" -> "BackwardNet"} -> "Catenate"}]NetBidirectionalOperator[net,Total] is equivalent to the following NetGraph:
NetGraph[<|"ForwardNet" -> net, "Reverse" -> SequenceReverseLayer[], "BackwardNet" -> net, "Total" -> ThreadingLayer[Plus]|>, {{"ForwardNet", "Reverse" -> "BackwardNet"} -> "Total"}]Possible Issues (1)
For aggregation functions Total and Mean, the forward and backward nets must produce sequences of arrays of the same size:
forward = BasicRecurrentLayer[2, "Input" -> {"Varying", 2}];
backward = BasicRecurrentLayer[4, "Input" -> {"Varying", 2}];
NetBidirectionalOperator[{forward, backward}]NetBidirectionalOperator[{forward, backward}, Total]
Text
Wolfram Research (2018), NetBidirectionalOperator, Wolfram Language function, https://reference.wolfram.com/language/ref/NetBidirectionalOperator.html (updated 2020).
CMS
Wolfram Language. 2018. "NetBidirectionalOperator." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/NetBidirectionalOperator.html.
APA
Wolfram Language. (2018). NetBidirectionalOperator. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/NetBidirectionalOperator.html