Entropy
Details and Options
- Entropy is a measure of variation (size of information) within the data.
- The data can have the following additional forms and interpretations:
-
{x1,x2,…} list of numeric or symbolic values » » string list of characters in a String string » WeightedData list of values with weights » TimeSeries, TemporalData vector or array of values (the time stamps ignored) » - Entropy[k,{x1, …}] is equivalent to -
piLog[k,pi], where pi is the occurrence fraction of the distinct element xi. » - The following options can be given:
-
SameTest Automatic function to test equality of expressions » - Entropy[data,SameTestf] applies f to pairs of data elements to determine whether they should be considered equivalent.
Examples
open all close allBasic Examples (3)
Entropy[{0, 1, 1, 4, 1, 1}]Entropy["A quick brown fox jumps over the lazy dog"]%//NEntropy[{a, b, b, c, c, c, c}]One of the seven elements occurs once, another twice, and the third one occurs 4 times:
-1 / 7Log[1 / 7] - 2 / 7Log[2 / 7] - 4 / 7Log[4 / 7] == %Scope (4)
Entropy of a random list of zeros and ones:
list = RandomInteger[1, 1000];Counts[list]Entropy[list]%//NEntropy of WeightedData:
data = {0, 1, 1, 4, 1, 1};
weights = {3, 1, 1, 1, 1, 1};Entropy[WeightedData[data, weights]]//SimplifyCompare to the entropy of the data without weights:
Entropy[data]//SimplifyN /@ {%, %%}Calculate the entropy of a TimeSeries:
ts = TemporalData[TimeSeries, {{{5, 4, 5, 6, 3, 3, 3, 6, 6, 6, 3, 3, 3, 6, 6, 6, 3, 6, 4, 4, 6, 6, 4, 5,
3, 5, 3, 4, 5, 6, 5, 3, 6, 3, 6, 5, 5, 4, 3, 5, 3, 5, 4, 5, 3, 3, 4, 6, 4, 5, 6, 5, 3, 4, 6, 4,
5, 3, 3, 6, 4, 4, 3, 4, 6, 3, 3, 4, 3, 3, ... , 4, 5,
4, 3, 3, 3, 3, 3, 6, 4, 4, 5, 5, 3, 3, 4, 4, 5, 3, 4, 3, 4, 6, 3}}, {{1, 174, 1}}, 1,
{"Continuous", 1}, {"Discrete", 1}, 1,
{ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}, ValueDimensions -> 1}}, False,
14.3];ListPlot[ts]Entropy[ts]The entropy depends only on the values:
Entropy[ts["Values"]]%% - %//Simplifylist = {0, 1, 1, 4, 1, 1};Entropy[10, list]//SimplifyEntropy[2, list]//SimplifyEntropy[E, list] == Entropy[list]Options (1)
Applications (2)
Entropy can be used to compare the levels of variations between datasets:
list1 = {0, 1, 1, 0, 0, 1, 1};
list2 = {1, 2, 3, 4, 5, 6, 7};N[Entropy /@ {list1, list2}]Calculate the entropy for a path of a TelegraphProcess with rate 
:
μ = 1.3;
data = RandomFunction[TelegraphProcess[μ], {1, 10 ^ 2}]ListPlot[data]Entropy[data]N[%]Properties & Relations (3)
Entropy does not depend on the data values:
Entropy[{1, 2, 3, 4}]Entropy[{x, y, z, w}]Entropy["star"]Compute the entropy of a list from the definition:
list = RandomInteger[3, 100]probs = Values[Counts[list]] / Length[list]-Total[probs Log[probs]]Entropy[list] - %//FullSimplifyEntropy of a string is equivalent to the entropy of its characters list:
string = "A quick brown fox jumps over the lazy dog";Entropy[string]Characters[string]Entropy[%]%%% - %//SimplifyPossible Issues (1)
Neat Examples (2)
Entropy[ExampleData[{"Text", "Hamlet"}]]//Short%//NEntropy[ExampleData[{"Text", "ShakespearesSonnets"}]]//NCompare the entropy of the text of the UN Declaration of Human Rights in different languages:
unhr = Select[ExampleData["Text"], StringContainsQ[#[[2]], "UNHumanRights"]&];unhr//Shortlabels = Map[Style[StringDrop[#, 13], 9]&, unhr[[All, 2]]];BarChart[Map[N@Entropy[ExampleData[#]]&, unhr], ChartLabels -> Placed[labels, After], BarOrigin -> Left, ColorFunction -> Function[{height}, ColorData["IslandColors"][1 - height]]]Text
Wolfram Research (2008), Entropy, Wolfram Language function, https://reference.wolfram.com/language/ref/Entropy.html.
CMS
Wolfram Language. 2008. "Entropy." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/Entropy.html.
APA
Wolfram Language. (2008). Entropy. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/Entropy.html