ParametricRampLayer
represents a net layer that computes a leaky ReLU activation with a slope that can be learned.
ParametricRampLayer[levels]
specifies the levels on which each dimension has a specific slope.
Details and Options
- In ParametricRampLayer[levels], the levels can be specified using the following settings:
-
a level a only a1;;a2 levels a1 through a2 {a,b,…} levels a,b,… {} unique slope values accross all dimensions - In ParametricRampLayer[], the level used is 1.
- The following optional parameters can be specified:
-
"Slope" Automatic learnable tensor of slopes LearningRateMultipliers Automatic learning rate multipliers for the slopes - The slope is a coefficient of leakage applied to input negative values. ParametricRampLayer applies the following function elementwise:
- By default, the slope is initialized to 0.1.
- ParametricRampLayer["Slope"value,LearningRateMultipliers->0] is a leaky ReLU with a fixed slope.
- ParametricRampLayer exposes the following ports for use in NetGraph etc.:
-
"Input" an array of arbitrary rank "Output" an array with the same dimensions as the input - When it cannot be inferred from other layers in a larger net, the option "Input"->{n1,n2,…} can be used to fix the input dimensions of ParametricRampLayer.
- Information[ParametricRampLayer[…]] gives a report about the layer.
- Information[ParametricRampLayer[…],prop] gives the value of the property prop of ParametricRampLayer[…]. Possible properties are the same as for NetGraph.
Examples
open all close allBasic Examples (2)
Create a ParametricRampLayer:
ParametricRampLayer[]Create a ParametricRampLayer with a specified slope:
prelu = ParametricRampLayer[{}, "Slope" -> 0.25]Apply the layer to an input vector:
prelu[{{10, 10, 10}, {-10, -10, -10}}]Scope (3)
Initialize a ParametricRampLayer that takes a vector of length 3, having a slope coefficient for each dimension:
act = NetInitialize@ParametricRampLayer["Input" -> 3]Normal@NetExtract[act, "Slope"]Apply the layer to some inputs:
act[{{-3, -2, -1}, {0, 1, 2}}]Learn the slopes on some data:
trained = NetTrain[act, RandomReal[{-1, 1}, {10, 3}] -> RandomReal[{-1, 1}, {10, 3}]]Normal@NetExtract[trained, "Slope"]Initialize a ParametricRampLayer that takes a length-3 vector and uses a unique slope coefficient:
act = NetInitialize@ParametricRampLayer[{}, "Input" -> 3]Normal@NetExtract[act, "Slope"]Initialize a ParametricRampLayer that takes a sequence of length-3 vectors:
act = NetInitialize@ParametricRampLayer[2, "Input" -> {"Varying", 3}]Normal@NetExtract[act, "Slope"]Options (3)
LearningRateMultipliers (1)
Create a leaky ReLU with a fixed slope of 0.1:
leakyRelu = ParametricRampLayer[{}, "Slope" -> 0.1, LearningRateMultipliers -> 0, "Input" -> 3]Train a network with this nonlinearity:
trained = NetTrain[NetChain[{LinearLayer[3], leakyRelu, LinearLayer[{}], LogisticSigmoid}, "Output" -> NetDecoder["Boolean"]], {1 -> False, 2 -> True, 3 -> False, 4 -> True}]The leaky ReLU still has the same slope value after training:
NetExtract[trained, {2, "Slope"}]"Slope" (2)
Create a ParametricRampLayer already initialized with a given slope value:
prelu = ParametricRampLayer["Slope" -> {0.1, 0.2, 0.3}, LearningRateMultipliers -> 0, "Input" -> 3]prelu[{{-1, -1, -1}, {1, 1, 1}}]Create a leaky ReLU with a unique slope, fixing the value of that slope:
prelu = ParametricRampLayer[{}, "Slope" -> 0.1]prelu[{{-1, -1, -1}, {1, 1, 1}}]Text
Wolfram Research (2020), ParametricRampLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/ParametricRampLayer.html.
CMS
Wolfram Language. 2020. "ParametricRampLayer." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/ParametricRampLayer.html.
APA
Wolfram Language. (2020). ParametricRampLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/ParametricRampLayer.html