WOLFRAM

Wolfram Language & System Documentation Center
Unsupervised Learning with Neural Networks

Unsupervised Learning with Neural Networks

Manifold LearningMNIST Digit Autoencoder
Manifold Learning
Create a synthetic two-dimensional manifold:
Wolfram Language code: manifold = Table[AngleVector[{x, 0.9Pi x}] + x / 20 * RandomVariate[NormalDistribution[], 2], {x, 0, 1, 0.001}]; plot = ListPlot[manifold, PlotStyle -> Orange]
Create a net with a "bottleneck" layer in order to learn a parameterization of the manifold:
Wolfram Language code: net = NetChain[{25, Ramp, 1, Ramp, 25, Ramp, 2}, "Input" -> 2]
Create a loss network that computes a loss based on the "reconstruction error" on the input manifold:
Wolfram Language code: lossNet = NetGraph[{net, MeanSquaredLossLayer[]}, {1 -> 2, NetPort["Input"] -> NetPort[2, "Target"]}]
Train the loss network on the manifold and extract the original net from the loss network:
Wolfram Language code: results = NetTrain[lossNet, <|"Input" -> manifold|>, All] trained = results["TrainedNet"]; trained = NetExtract[trained, 1]
Visualize how the net projects arbitrary points onto the manifold:
Wolfram Language code: {{xmin, xmax}, {ymin, ymax}} = CoordinateBounds[manifold, .2]; delta[x_Real, y_Real] := trained[{x, y}] - {x, y}; Show[plot, StreamPlot[delta[x, y], {x, xmin, xmax}, {y, ymin, ymax}]]
Split the net into "encoder" and "decoder" networks (the encoder parameterizes points using a single scalar value, whereas the decoder reconstructs the point from this parameterization):
Wolfram Language code: decoder = Drop[trained, 3] encoder = Take[trained, 3]
Color each point in the original manifold by its parameterization under the encoder:
Wolfram Language code: ListPlot[Style[#, Hue[First[encoder[#]] / 4]]& /@ manifold]
Obtain the range of the parameterization by applying the encoder to the manifold:
Wolfram Language code: {min, max} = MinMax[encoder[manifold]]
Show the reconstruction over this range along with the original manifold:
Wolfram Language code: Show[plot, ListLinePlot[Table[decoder[x], {x, min, max, .01}]]]
MNIST Digit Autoencoder
First, obtain the training data and select images corresponding to digits 0 through 4:
Wolfram Language code: resource = ResourceObject["MNIST"]; trainingData = ResourceData[resource, "TrainingData"]; testData = ResourceData[resource, "TestData"]; trainingSubset = Select[trainingData, Last[#] ≤ 4&]; testSubset = Select[testData, Last[#] ≤ 4&]; RandomSample[trainingSubset, 8] trainingImages = Keys[trainingSubset];
An autoencoder has the same shape for its input and output, and has a bottleneck in the middle of the net to prevent the net simply memorizing the inputs.
Create a net that takes an input with dimensions {1,28,28} and returns an output with the same dimensions {1,28,28}:
Wolfram Language code: evalnet = NetChain[{FlattenLayer[], LinearLayer[60], BatchNormalizationLayer[], ElementwiseLayer["ReLU"], 40, ElementwiseLayer["ReLU"], 60, BatchNormalizationLayer[], ElementwiseLayer["ReLU"], 784, ReshapeLayer[{1, 28, 28}]}, "Input" -> {1, 28, 28}]
The preceding net takes as input 784 real numbers and compresses this to a vector of 40 real numbers (the bottleneck). The net then needs to reconstruct the original image from these 40 real numbers.
Create an encoder that converts images to arrays of dimension {1,28,28}:
Wolfram Language code: enc = NetEncoder[{"Image", {28, 28}, "Grayscale"}]
Attach a loss to the autoencoder net that penalizes the input being different from the output:
Wolfram Language code: net = NetGraph[<|"evalnet" -> evalnet, "loss" -> MeanAbsoluteLossLayer[]|>, {NetPort["Input"] -> "evalnet" -> "loss", NetPort["Input"] -> NetPort["loss", "Target"]}, "Input" -> enc]
Train the network on the reconstruction error:
Wolfram Language code: results = NetTrain[net, <|"Input" -> trainingImages|>, All, ValidationSet -> Scaled[0.1], MaxTrainingRounds -> 100] trained = results["TrainedNet"]
Obtain a subnetwork that performs only reconstruction:
Wolfram Language code: reconstructor = NetReplacePart[NetExtract[trained, "evalnet"], {"Input" -> enc, "Output" -> NetDecoder[enc]}]
Reconstruct some sample images:
Wolfram Language code: reconstructor /@ {[image], [image], [image], [image], [image]}
Obtain a subnetwork that produces the code vector:
Wolfram Language code: encoder = NetTake[reconstructor, {1, 6}]
Compute codes for all of the test images:
Wolfram Language code: testImages = Keys[testSubset]; features = encoder[testImages];
Project the code vectors to three dimensions and visualize them along with the original labels (not seen by the network). The digit classes tend to cluster together:
Wolfram Language code: coords = DimensionReduce[features, 3]; labels = Values[testSubset]; ListPointPlot3D[Table[Extract[coords, Position[labels, i]], {i, 0, 4}], PlotLegends -> PointLegend[96, Range[0, 4]], BoxRatios -> 1, Axes -> None, Boxed -> True, PlotStyle -> Map[ColorData[96], Range[1, 5]], AspectRatio -> 1]
Perform automatic clustering directly on the code vectors and show a sample taken from each cluster:
Wolfram Language code: components = ClusteringComponents[features, 5, 1]; Map[Part[testImages, RandomSample[#, 10]]&, PositionIndex[components]]
Visualize a hierarchical clustering of random representatives from each class:
Wolfram Language code: representatives = Catenate@GroupBy[testSubset, Last -> First, RandomSample[#, 6]&]; ClusteringTree[encoder[representatives] -> Map[ImageCrop, representatives]]