Wolfram Language & System Documentation Center

MarkovProcessProperties

gives a summary of properties for the finite state Markov process mproc.

MarkovProcessProperties[mproc,"property"]

gives the specified "property" for the process mproc.

Details

MarkovProcessProperties can be used for finite state Markov processes such as DiscreteMarkovProcess and ContinuousMarkovProcess.
MarkovProcessProperties[mproc,"Properties"] gives a list of available properties.
MarkovProcessProperties[mproc,"property","Description"] gives a description of the property as a string.
Basic properties include:

	"InitialProbabilities"	initial state probability vector
	"TransitionMatrix"	conditional transition probabilities m
	"TransitionRateMatrix"	conditional transition rates q
	"TransitionRateVector"	state transition rates μ
	"HoldingTimeMean"	mean holding time for a state
	"HoldingTimeVariance"	variance of holding time for a state
	"SummaryTable"	summary of properties

For a continuous-time Markov process "TransitionMatrix" gives the transition matrix of the embedded discrete-time Markov process.
The holding time is the time spent in each state before transitioning to a different state. This takes into account self-loops which may cause the process to transition to the same state several times.
Structural properties include:

	"CommunicatingClasses"	sets of states accessible from each other
	"RecurrentClasses"	communicating classes that cannot be left
	"TransientClasses"	communicating classes that can be left
	"AbsorbingClasses"	recurrent classes with a single element
	"PeriodicClasses"	communicating classes with finite period greater than 1
	"Periods"	period for each of the periodic classes
	"Irreducible"	whether the process has a single recurrent class
	"Aperiodic"	whether all classes are aperiodic
	"Primitive"	whether the process is irreducible and aperiodic

The states of a finite Markov process can be grouped into communicating classes where from each state in a class there is a path to every other state in the class.
A communicating class can be transient when there is a path from the class to another class or recurrent when there is not. A special type of recurrent class, called absorbing, consist of a single element.
A state is periodic is if there is a non-zero probability that you return to the state after two or more steps. All the states in a class have the same period.
Transient properties before the process enters a recurrent class:

	"TransientVisitMean"	mean number of visits to each transient state
	"TransientVisitVariance"	variance of number of visits to each transient state
	"TransientTotalVisitMean"	mean total number of transient states visited

A Markov process will eventually enter a recurrent class. The transient properties characterize how many times each transient state is visited or how many different transient states are visited.
Limiting properties include:

	"ReachabilityProbability"	probability of ever reaching a state
	"LimitTransitionMatrix"	Cesaro limit of the transition matrix
	"Reversible"	whether the process is reversible

If a property is not available, this is indicated by Missing["reason"].

Examples

open all close all

Basic Examples (2)

Summary table of properties:

Wolfram Language code:

proc = DiscreteMarkovProcess[3, {{1 / 2, 1 / 2, 0, 0}, {1 / 2, 1 / 2, 0, 0}, {1 / 4, 1 / 4, 1 / 4, 1 / 4}, {0, 0, 0, 1}}];

Wolfram Language code: Graph[proc]

Wolfram Language code: MarkovProcessProperties[proc]

Find the values of a specific property:

Wolfram Language code: proc = ContinuousMarkovProcess[1, {{-a, a}, {0, 0}}];

Wolfram Language code: MarkovProcessProperties[proc, "RecurrentClasses"]

Description of the property:

Wolfram Language code: MarkovProcessProperties[proc, "RecurrentClasses", "Description"]

Scope (5)

Find the communicating classes, highlighted in the graph through different colors:

Wolfram Language code:

proc = DiscreteMarkovProcess[1, {{0, 1 / 3, 0, 2 / 3, 0}, {1 / 2, 0, 0, 0, 1 / 2}, {0, 0, 1 / 2, 1 / 2, 0}, {0, 0, 1 / 2, 1 / 2, 0}, {0, 0, 0, 0, 1}}];

Wolfram Language code: gr = Graph[proc, GraphLayout -> "LayeredDrawing"]

Wolfram Language code: MarkovProcessProperties[proc, "CommunicatingClasses"]

Wolfram Language code: ConnectedComponents[gr]

The process is not irreducible:

Wolfram Language code: MarkovProcessProperties[proc, "Irreducible"]

Wolfram Language code: ConnectedGraphQ[gr]

Find the recurrent classes, represented by square and circular vertices in the graph:

Wolfram Language code: MarkovProcessProperties[proc, "RecurrentClasses"]

Wolfram Language code: Select[ConnectedComponents[gr], Complement[VertexOutComponent[gr, #], #] === {}&]

Find the transient classes, represented by diamond vertices in the graph:

Wolfram Language code: MarkovProcessProperties[proc, "TransientClasses"]

Wolfram Language code: Select[ConnectedComponents[gr], Complement[VertexOutComponent[gr, #], #] =!= {}&]

Find the absorbing classes, represented by square vertices in the graph:

Wolfram Language code: MarkovProcessProperties[proc, "AbsorbingClasses"]

Wolfram Language code: Position[VertexOutDegree[SimpleGraph[gr]], 0]

Define a Markov process with self-loops:

Wolfram Language code: proc1 = DiscreteMarkovProcess[1, {{0, 1 / 3, 0, 2 / 3}, {1, 0, 0, 0}, {0, 0, 1 / 2, 1 / 2}, {0, 0, 1 / 2, 1 / 2}}];

Wolfram Language code: Graph[proc1]

The self-loops make the class aperiodic:

Wolfram Language code: MarkovProcessProperties[proc1, "PeriodicClasses"]

Markov process with no self-loops:

Wolfram Language code: proc2 = DiscreteMarkovProcess[1, {{0, 1 / 3, 0, 2 / 3}, {1, 0, 0, 0}, {0, 0, 0, 1}, {0, 0, 1, 0}}];

Wolfram Language code: Graph[proc2]

Here both classes are periodic:

Wolfram Language code: MarkovProcessProperties[proc2, "PeriodicClasses"]

Summary table of properties for a continuous-time Markov process:

Wolfram Language code: proc = ContinuousMarkovProcess[1, With[{λ = 3, μ = 2}, {{-λ, λ, 0}, {μ, -λ - μ, λ}, {0, μ, -μ}}]];

Wolfram Language code: Graph[proc]

Wolfram Language code: MarkovProcessProperties[proc]

Obtain the value for a specific property for a continuous Markov chain:

Wolfram Language code: proc = ContinuousMarkovProcess[1, With[{λ = 3, μ = 2}, {{-λ, λ, 0}, {μ, -λ - μ, λ}, {0, μ, -μ}}]];

Wolfram Language code: MarkovProcessProperties[proc, "TransitionMatrix"]//MatrixForm

Wolfram Language code: ListLinePlot[RandomFunction[proc, {0, 5}]["Paths"], Ticks -> {Automatic, Range[3]}]

Wolfram Language code:

ListLinePlot[RandomFunction[DiscreteMarkovProcess[1, MarkovProcessProperties[proc, "TransitionMatrix"]], {0, 10}]["Paths"], Ticks -> {Automatic, Range[3]}]

Find the conditional mean number of total transitions starting in state 1 and ending in state 4:

Wolfram Language code: 𝒫 = ContinuousMarkovProcess[{1, 0, 0, 0}, {{-3, 1, 2, 0}, {3, -6, 2, 1}, {4, 2, -9, 3}, {0, 0, 0, 0}}];

Wolfram Language code: MarkovProcessProperties[𝒫, "TransitionCount"][[4]][100] / Probability[x[100] == 4, x𝒫]

Wolfram Language code: N[%]//Quiet

Compare with the results from simulation:

Wolfram Language code:

data = RandomFunction[𝒫, {0, 100}, 4 * 10 ^ 4, Method -> {Automatic, "StoppingFunction" -> (#3 =!= 4&)}]["ValueList"];

Wolfram Language code: N[Mean[Length[#] - 1& /@ data]]

Find the conditional mean number of transitions from state 2 to state 3:

Wolfram Language code: 𝒫 = ContinuousMarkovProcess[{1, 0, 0, 0}, {{-3, 1, 2, 0}, {3, -6, 2, 1}, {4, 2, -9, 3}, {0, 0, 0, 0}}];

Wolfram Language code: MarkovProcessProperties[𝒫, {"TransitionCount", {{2, 3}}}][[4]][100] / Probability[x[100] == 4, x𝒫]

Wolfram Language code: N[%]//Quiet

Compare with the results from simulation:

Wolfram Language code: data = RandomFunction[𝒫, {0, 10 ^ 3}, 10 ^ 4, Method -> {Automatic, "StoppingFunction" -> (#3 =!= 4&)}]["ValueList"];

Wolfram Language code: N[Mean[Count[Partition[data[[#]], 2, 1], {2, 3}]& /@ Range[Length[data]]]]

Applications (2)

A gambler starts with $3 and bets $1 at each step. He wins $1 with a probability of 0.4:

Wolfram Language code:

GamblersRuin[p_, n_] := SparseArray[{{1, 1} -> 1, {n + 1, n + 1} -> 1, {i_, j_} /; 1 < i < n + 1 && j == i + 1 -> p, {i_, j_} /; 1 < i < n + 1 && j == i - 1 -> 1 - p}, {n + 1, n + 1}]

Wolfram Language code: gamblerwealth = DiscreteMarkovProcess[4, GamblersRuin[0.4, 7]];

Find the expected number of times the gambler has units:

Wolfram Language code:

Transpose[{MarkovProcessProperties[gamblerwealth, "TransientClasses"][[1]] - 1, MarkovProcessProperties[gamblerwealth, "TransientVisitMean"]}]//Grid

Verify the answer using simulation:

Wolfram Language code:

data = RandomFunction[gamblerwealth, {0, 10 ^ 3}, 10 ^ 4, Method -> {Automatic, "StoppingFunction" -> Function[{len, pos}, len < 1001 && pos ≠ 1 && pos ≠ 8]}]["ValueList"];

Wolfram Language code: Table[Mean[Count[#, i]& /@ data], {i, 2, 7}]//N

Find the expected time until the gambler wins $7 or goes broke:

Wolfram Language code: Mean[FirstPassageTimeDistribution[gamblerwealth, {1, 8}]]

Total states visited before the gambler wins $7 or goes broke:

Wolfram Language code: MarkovProcessProperties[gamblerwealth, "TransientTotalVisitMean"]

Verify the answer using simulation:

Wolfram Language code: data = DeleteDuplicates /@ RandomFunction[gamblerwealth, {0, 10 ^ 2}, 4 * 10 ^ 3]["ValueList"];

Wolfram Language code: Mean[Length[#] - 1& /@ data]//N

In a game of tennis between two players, suppose the probability of the server winning a point is . There are 17 possible states:

Wolfram Language code:

states = {"GameA", "GameB", "Deux", "AdvA", "AdvB", "40-15", "15-40", "40-0", "30-15", "15-30", "0-40", "30-0", "15-15", "0-30", "15-0", "0-15", "0-0"};

Visualize the random walk graph for :

Wolfram Language code:

proc = With[{p = 3 / 5, q = 2 / 5}, DiscreteMarkovProcess[17, (⁠|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | p | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| p | 0 | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | q | p | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| p | 0 | 0 | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | q | 0 | 0 | p | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| p | 0 | 0 | 0 | 0 | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | q | 0 | 0 | p | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | p | 0 | 0 | 0 | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | q | 0 | 0 | 0 | 0 | p | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | p | q | 0 |⁠)]];

Wolfram Language code:

Graph[proc, VertexSize -> Large, VertexLabels -> Thread[Range[17] -> Map[Placed[Style[#, {Bold}], Center]&, states]], ImageSize -> Large, ImagePadding -> 25]

Find the probability of the server winning the game if :

Wolfram Language code: MarkovProcessProperties[proc, "ReachabilityProbability"][[1]]//N

Wolfram Language code: PDF[proc[Infinity], 1]//N

Find the mean time to absorption, that is, the number of points played:

Wolfram Language code: Total[MarkovProcessProperties[proc, "TransientVisitMean"]]//N

Wolfram Language code: N[Mean[FirstPassageTimeDistribution[proc, {1, 2}]]]

Find the mean number of states visited:

Wolfram Language code: MarkovProcessProperties[proc, "TransientTotalVisitMean"]//N

Find the average number of times the score will be tied at deuce:

Wolfram Language code: MarkovProcessProperties[proc, "TransientVisitMean"][[1]]//N

Verify the answer using simulation:

Wolfram Language code:

Mean[Count[#, 3]& /@ RandomFunction[proc, {0, 10 ^ 3}, 10 ^ 4, Method -> {Automatic, "StoppingFunction" -> Function[{len, pos}, len < 1001 && pos ≠ 1 && pos ≠ 2]}]["ValueList"]]//N

Properties & Relations (1)

The transition matrix of this Markov process is not irreducible:

Wolfram Language code:

proc = DiscreteMarkovProcess[i, {{(1/2), (1/2), 0, 0, 0}, {(1/3), (2/3), 0, 0, 0}, {0, (1/3), (2/3), 0, 0}, {0, 0, 0, 0, 1}, {0, 0, 0, 1, 0}}];

Wolfram Language code: MarkovProcessProperties[proc, "Irreducible"]

Hence the stationary distribution depends on the initial state probabilities:

Wolfram Language code:

StationaryDistribution[DiscreteMarkovProcess[1, {{(1/2), (1/2), 0, 0, 0}, {(1/3), (2/3), 0, 0, 0}, {0, (1/3), (2/3), 0, 0}, {0, 0, 0, 0, 1}, {0, 0, 0, 1, 0}}]]

Wolfram Language code:

StationaryDistribution[DiscreteMarkovProcess[4, {{(1/2), (1/2), 0, 0, 0}, {(1/3), (2/3), 0, 0, 0}, {0, (1/3), (2/3), 0, 0}, {0, 0, 0, 0, 1}, {0, 0, 0, 1, 0}}]]

Possible Issues (1)

Some property values may not be available:

Wolfram Language code: MarkovProcessProperties[DiscreteMarkovProcess[1, {{1 / 3, 2 / 3}, {1 / 2, 1 / 2}}], "TransitionRateVector"]

This property is available only for continuous Markov processes:

Wolfram Language code: MarkovProcessProperties[ContinuousMarkovProcess[1, {{-2 / 3, 2 / 3}, {1 / 2, -1 / 2}}], "TransitionRateVector"]

Top

More Learning

Tech Support

Wolfram Solutions

Wolfram Solutions For Education

Get Started

Grow Your Skills

Work with Us

Educational Programs for Adults

Educational Programs for Youth

Read

MarkovProcessProperties

Details

Examples

Basic Examples (2)

Scope (5)

Applications (2)

Properties & Relations (1)

Possible Issues (1)

Text

CMS

APA

BibTeX

BibLaTeX

MarkovProcessProperties

Details

Examples

Basic Examples (2)

Scope (5)

Applications (2)

Properties & Relations (1)

Possible Issues (1)

See Also

Related Guides

History

Text

CMS

APA

BibTeX

BibLaTeX