EmbeddingLayer
EmbeddingLayer[size,n]
represents a trainable net layer that embeds integers between 1 and n into a continuous vector space of dimension size.
EmbeddingLayer[size]
leaves the n to be inferred from context.
Details and Options
- EmbeddingLayer operates on a single integer or rank-k array of integers, producing a vector or rank-k+1 array, respectively.
- Specifying EmbeddingLayer[size] will produce a net that infers the n when a NetEncoder[…] that produces integers is connected to its input.
- The embedding performed by EmbeddingLayer is learned during training.
- The following optional parameters can be included:
-
LearningRateMultipliers Automatic learning rate multipliers for the weights "Weights" Automatic initial matrix of weights of dimensions n×size - With Automatic settings, the weights are added automatically when NetInitialize or NetTrain is used.
- If weights have been added, EmbeddingLayer[…][input] explicitly computes the output from applying the layer.
- EmbeddingLayer[…][{input1,input2,…}] explicitly computes outputs for each of the inputi.
- When given a NumericArray as input, the output will be a NumericArray.
- NetExtract can be used to extract weights from an EmbeddingLayer object.
- EmbeddingLayer is typically used inside NetChain, NetGraph, etc.
- EmbeddingLayer exposes the following ports for use in NetGraph etc.:
-
"Input" an integer or rank-k array of integers "Output" a vector or rank-k+1 array - EmbeddingLayer[size,n,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:
-
NetEncoder[…] encoder producing an integer or array of integers "Integer" a single integer d a vector of integers of length d {d1,d2,…} an array of integers of dimensions d1×d2×… "Varying" a variable-length vector of integers {"Varying",d2,d3,…} an array whose first dimension is variable and whose remaining dimensions are d2×d3×… - Options[EmbeddingLayer] gives the list of default options to construct the layer. Options[EmbeddingLayer[…]] gives the list of default options to evaluate the layer on some data.
- Information[EmbeddingLayer[…]] gives a report about the layer.
- Information[EmbeddingLayer[…],prop] gives the value of the property prop of EmbeddingLayer[…]. Possible properties are the same as for NetGraph.
Examples
open all close allBasic Examples (2)
Create an EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 4:
EmbeddingLayer[4, 3]Create a randomly initialized EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 2:
embed = NetInitialize@EmbeddingLayer[2, 3]Apply the layer to an integer:
out = embed[2]
Dimensions[out]Apply the layer to a vector of integers:
out = embed[{1, 2, 1}];
Dimensions[out]Scope (4)
Create a randomly initialized EmbeddingLayer that embeds a variable-length sequence of integers from 1 to 10 into a sequence of vectors of length 2 represented by a matrix whose last dimension has size 2:
embed = NetInitialize@EmbeddingLayer[2, 10, "Input" -> "Varying"]Apply the layer to a single sequence of integers:
embed[{1, 2, 3}]//MatrixFormApply the layer to multiple sequences simultaneously, in which the sequence lengths are not identical:
embed[{{1, 2, 3}, {1, 2, 3, 4}, {2}}]//Map[MatrixForm]Create an EmbeddingLayer with an encoded input, where n is inferred from the encoder:
embed = EmbeddingLayer[5, "Input" -> NetEncoder[{"Class", {a, b, c}}]]Randomly initialize and apply the layer:
embed = NetInitialize[embed]embed[{a, b, a, c}]//MatrixFormCreate an EmbeddingLayer whose input is a sequence of integers representing the characters of a string:
encoder = NetEncoder["Characters"];
embed = NetInitialize@EmbeddingLayer[2, "Input" -> encoder]embed["ABABBC"]//MatrixFormCreate an EmbeddingLayer whose input is a sequence of codes representing tokens from a string:
encoder = NetEncoder["Tokens"];
embed = NetInitialize@EmbeddingLayer[2, "Input" -> encoder]embed["how are you today"]//MatrixFormOptions (1)
"Weights" (1)
Create an EmbeddingLayer with weights explicitly specified:
embed = EmbeddingLayer["Weights" -> {{1, 0, 1, 0}, {0, 1, 1, 0}}]Apply the layer to some inputs:
embed[2]embed[1]Applications (2)
In text processing applications, networks that operate on entire words typically use an embedding to collapse the large vocabulary size of human languages into manageable vector sizes.
Create a chain that embeds successive words from a string into a sequence of vectors of size 10 and then processes those vectors, using a recurrent layer to produce a final vector. First create an encoder using common English words:
encoder = NetEncoder["Tokens"]encoder["this is a sentence"]Create and randomly initialize the net, attaching the encoder to its input:
net = NetChain[{EmbeddingLayer[5], BasicRecurrentLayer[4], SequenceLastLayer[]}, "Input" -> encoder]Randomly initialize the net and apply it to some sentences:
net = NetInitialize[net];net["this is a sentence"]net["and another sentence"]Obtain a pre-trained EmbeddingLayer using NetModel:
net = NetModel["GloVe 50-Dimensional Word Vectors Trained on Wikipedia and Gigaword-5 Data"]A NetEncoder has been attached to the input of the net:
NetExtract[net, "Input"]Apply the embedding to a sentence containing two tokens:
net["hello world"]Visualize the embedding of a longer sentence:
net["it always seems impossible until it is done"]//MatrixPlotProperties & Relations (1)
Evaluate an initialized EmbeddingLayer on a list of indices:
embed = NetInitialize@EmbeddingLayer[5, 2];
indices = {1, 2, 1};
embed[indices]This is equivalent to the following:
NetExtract[embed, "Weights"][[indices]]Text
Wolfram Research (2016), EmbeddingLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/EmbeddingLayer.html (updated 2020).
CMS
Wolfram Language. 2016. "EmbeddingLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/EmbeddingLayer.html.
APA
Wolfram Language. (2016). EmbeddingLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/EmbeddingLayer.html