Markov Chain Monte Carlo Discrete Continuous

Markov Chain

Control of Networks: Mathematical Background

Jean Walrand , Pravin Varaiya , in High-Performance Communication Networks (Second Edition), 2000

9.1.1 Overview

A Markov chain is a model of the random motion of an object in a discrete set of possible locations. Two versions of this model are of interest to us: discrete time and continuous time. In discrete time, the position of the object–called the state of the Markov chain–is recorded every unit of time, that is, at times 0, 1, 2, and so on. In continuous time, the state is observed at all times t ≥ 0. One can think of the continuous-time model as being a discrete time model where the time unit is infinitesimally small. The state of the Markov chain changes randomly. In discrete time, there is a die at every location. Every time unit, the Markov chain tosses the die at its current location to decide where to jump next. In that way, the law of the future motion of the state depends only on the present location and not on previous locations. This key property that the Markov chain has of "forgetting" its past locations greatly simplifies the analysis.

Engineers use Markov chains to model the progression of the calls that a telephone network carries and of packets that a datagram or virtual circuit network transports. The randomness in these models reflects the uncertainty about when users place calls or send packets and about the length of packets and their destination. The randomness also captures the transmission errors and failures of devices.

The theory of Markov chains tells us how to calculate the fraction of time that the state of the Markov chain spends in the different locations. Network engineers use that theory to estimate the delays and losses of packets in networks or the fraction of time that telephone calls are blocked because all the circuits are busy. The engineers then use these estimates to design and control networks, as we explained in Chapter 8.

In this section, we review the main results of the theory of Markov chains, and we illustrate these results with examples.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780080508030500149

Hardware Accelerator Systems for Artificial Intelligence and Machine Learning

Amitabh Biswal , ... Zakir Hussain , in Advances in Computers, 2021

2.6.8 Markov chain model

Markov chain model is a stochastic model which has Markov property. Markov property is satisfied when current state of the process is enough to predict the future state of the process and the prediction should be as good as making prediction by knowing their history. It is a very easy process to model random process.

Most simple example of Markov chain is a board games played with dice. The next state of the game depends just only the current state of the game and what is the outcome of the dice. It does not depend on how the state of game reached there. Random walk and weather models are another example of Markov model.

Fig. 5 is the simplest example of the weather model. It shows the probability of transition given the current state. If current weather is cloudy then their is 50% chance of getting a rain and 40% chance of being sunny. So the probability of being sunny given that its cloudy is only dependent on the previous state which is cloudy. It does not take into account, which state was before that.

Fig. 5. Markov state diagram.

Formal definition of discrete chain Markov chain can represented as

$\begin{array}{l}\hfill P_r(X_{n+1}=x|X_1=x_1,X_2=x_2,\dots ,X_n=x_n)=P_r(X_{n+1}=x|X_n=x_n)\end{array}$

In the above equation if for n ≥ 0 if value of P _x (X) is similar then this equation is called homogeneous equation and the matrix becomes the transition matrix.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S0065245821000139

GIS Methods and Techniques

Jie Dai , Li An , in Comprehensive Geographic Information Systems, 2018

1.21.3.5 Spatial Markov Chains Models

Traditional Markov chain models aim to predict the status of an object (or a set of objects) or phenomenon at future times. There are several statuses the object(s) or the phenomenon of interest can be subject to, and the probability of changing from one status to another is derived from analyzing empirical observations. The principle is relatively simple: between adjacent time intervals, the changing probability between particular status pairs is constant.

Spatial Markov chain models address the change of spatial units, such as land cover/use change at different times. This method applies the traditional "snapshot" data model and examines the temporal trends of change. What distinguishes spatial Markov chain models from regular ones is the consideration of spatial dependence (e.g., through spatial lag or spatial weights matrix; Anselin, 2003) among nearby units. Despite many strengths of spatial Markov chain models (particularly it offers a simple methodology for exploring spatiotemporal changes), a number of drawbacks are also noteworthy, such as the questionable assumption of stationarity. For more detail, see An and Brown (2008), Iacono et al. (2012), and An et al. (2015).

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780124095489096251

Dynamic Programming

Jean-Michel Réveillac , in Optimization Tools for Logistics, 2015

4.4.1 Property of Markov chains

A Markov chain can have one or a number of properties that give it specific functions, which are often used to manage a concrete case.

4.4.1.1 Absorbing chain

A chain can be absorbing when one of its states, called the absorbing state, is such it is impossible to leave once it has been entered.

The chain is absorbing if and only if one of its states is absorbing and if from a non-absorbing state we can reach an absorbing state.

The non-absorbing states in a chain are said to be transient or transitory states. It is worth noting that this state is not always returned to.

When a state is always returned to, at the end of a mean infinite time, it is said to be null recurrent.

When a state is returned to infinite times, at intervals of finite times, on average, it is said to be positive recurrent.

4.4.1.2 Irreducible chain

A Markov chain is said to be irreducible or ergodic when all its states communicate i.e. when for every couple of states (e_i, e_j), the probability of going from one to the other is strictly positive.

This property is easy to see on a transition graph (see section 4.4.2.3). Each couple of vertices is linked by one or a number of edges and once we have arrived on one of these vertices, we can leave (probability less than 1).

In the event the chosen vertex cannot be left, it is in an absorbing state and the chain is thus not irreducible.

It can thus be deduced that every absorbing Markov chain is not irreducible.

4.4.1.3 Homogeneous chain

A time-Homogeneous Markov chain is a Markov chain whose probability of transitioning is independent of time i.e. if E is a finite space and has countable states, n the moment the chain finds itself on a state, x and y of the states in E, we have:

$\mathbb{P}\left(X_{n+1}=y|X_n=x\right)=\mathbb{P}\left(X_1=y|X_0=x\right)$

4.4.1.4 Regular chain

A Markov chain is regular if we can go from any state towards any other state following a fixed number of steps, the starting state being independent.

Note that a regular chain is irreducible.

4.4.1.5 Reversible chain

A Markov chain of transition matrix M is reversible in relation to a probability π, if for all x, y ∈ E we have:

$\pi \left(x\right)M\left(x,y\right)=\pi \left(y\right)M\left(y,x\right)$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9781785480492500049

Modeling tools to evaluate the performance of wireless multi-hop networks

Hakim Badis , Abderrezak Rachedi , in Modeling and Simulation of Computer Networks and Systems, 2015

3.1.1 Preliminary and definitions

A Markov chain is a very powerful tool used in various fields including physics, economics, engineering, genetics, and more. It is widely used because of its simplicity and flexibility. It's easy to model different systems with an arbitrary number of states and their transition matrix. It is used to model a dynamic system like wireless network that changes its states over time. Markov chains are classified into two kinds: continuous time Markov chains (CTMC) and discrete time Markov chains (DTMC). In DTMC, the state is allowed to change only at the discrete instants; that is not the case for CTMC, where the state can change at any time.

In the computer networks area, the stochastic models are selected to represent uncertainty phenomena of the network (because it depends on many unknown factors) where its behavior varies as time advances.

In this subsection, we focus on two communication systems: Single-Input Single-Output (SISO) and Multiple-Input Multiple-Output (MIMO), both in wireless multi-hop networks. We present stochastic models based on Markov chain to evaluate the network throughput and MAC protocols performance based on IEEE 802.11.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128008874000237

Modeling dependencies

Manfred Gilli , ... Enrico Schumann , in Numerical Methods and Optimization in Finance (Second Edition), 2019

7.2 Markov chains

7.2.1 Concepts

In a Markov chain, the probability for the next state or value of a variate depends on the current state. This approach is particularly useful for time series with conditional probabilities, and Markov Chain Monte Carlo (MCMC) plays a very prominent role in financial simulations.

In its simplest version, there is only a limited set of alternatives, and the probabilities for the next state of a variable only depend on its current state. For example, consider a bond that can be rated either "investment grade" (IG state 1) or "high yield" (HY state 2). With a low probability, the rating will change, but most likely, it will remain the same. In other words, the rating in the next period highly depends on its current rating, because upgrading or downgrading usually happens only with a low probability. For example, $\mathrm{prob}(x_{i+1}=\mathit{IG}|x_i=\mathit{IG})>\mathrm{prob}(x_{i+1}=\mathit{IG}|x_i=\mathit{HY})$ . The conditional probabilities can be collected in a transition matrix Π:

$\mathrm{\Pi }=[p_{x_{i+1}|x_i}]=\left[\begin{array}{cc}\hfill p_{\mathit{IG}|\mathit{IG}}\hfill & \hfill p_{\mathit{HY}|\mathit{IG}}\hfill \\ \hfill p_{\mathit{IG}|\mathit{HY}}\hfill & \hfill p_{\mathit{HY}|\mathit{HY}}\hfill \end{array}\right].$

Given the transition matrix, sampling is rather straightforward. The rows in the transition matrix provide the conditional probabilities for a given current state $x_i$ ; for example, $\mathrm{\Pi }=\left[\begin{array}{cc}\hfill 0.9\hfill & \hfill 0.1\hfill \\ \hfill 0.1\hfill & \hfill 0.9\hfill \end{array}\right]$ means that, for either state, there is a 90% chance that the next state will be the same and a 10% chance that the process will switch. Sampling the subsequent state $x_{i+1}$ can then be done with a roulette wheel principle (see Section 6.5.2).

If there are $n_x$ different states and $p_{x_{i+1}|x_i}=1/n_x$ , there will be no serial dependence. This changes, however, if the probabilities are state dependent. For $\mathrm{\Pi }=\left[\begin{array}{cc}\hfill 0.9\hfill & \hfill 0.1\hfill \\ \hfill 0.0\hfill & \hfill 1.0\hfill \end{array}\right]$ , state 1 is likely to be followed by state 1; once state 2 has been sampled, the process will never return to state 1. In this case, state 2 is called an "absorbing state." Alternatively, a transition matrix $\mathrm{\Pi }=\left[\begin{array}{cc}\hfill 0.01\hfill & \hfill 0.99\hfill \\ \hfill 0.90\hfill & \hfill 0.10\hfill \end{array}\right]$ will produce heavy oscillation between states, though, again, there is a higher chance that the system is in state 2. To get the conditional probabilities for the state for the period after the next, the transition matrix must be multiplied by itself, $p_{x_{i+2}|x_i}={\mathrm{\Pi }}^2$ ; for the tth future period, $p_{x_{i+t}|x_i}={\mathrm{\Pi }}^t$ . Note that the larger t becomes, the closer these probabilities get to the unconditional probabilities for the different states. Note that Π must be a square $n_x\times n_x$ matrix, and the rows must add up to 1.

This principle can be generalized for joint densities $p_{xy}$ with conditional distributions $p_{x|y}$ and $p_{y|x}$ . The simulation procedure is just a variant of the univariate case. An important prerequisite here is that at least one of the variates depends on the lagged value. If, say, $x_{i+1}$ depends on $y_i$ , but $y_{i+1}$ is conditioned on $x_{i+1}$ , one has to bear in mind the sequence for drawing the samples. Also, if x and y can assume a different number of states, $n_x$ and $n_y$ , respectively, then the dimensions for ${\mathrm{\Pi }}_x$ and ${\mathrm{\Pi }}_y$ must be $n_y\times n_x$ and $n_x\times n_y$ , respectively.

Markov chains can also be generalized to continuous distributions; ⁴ the geometric Brownian motion, often used to model stock price processes, would be one example for this: the new price depends on the (realized) previous price plus a (random) price change (see Section 8.3).

7.2.2 The Metropolis algorithm

Metropolis et al. (1953) suggest a Monte Carlo method to simulate states for a Boltzmann distribution in molecular and atomic systems. Assume the current state (or position) is $x_i$ and the energy function is $E(x_i)$ . Then, a random new position is generated within the neighborhood of $x_i$ , $y=x_i+u_{i}s$ , where $u_i$ is vector of uniform samples $u_i\in [-1,+1]$ (same dimensions as $x_i$ ), and s is a scalar. This new position is accepted for certain if it has lower energy than $x_i$ . But also a higher energy state is accepted with a certain probability $\exp (-{\mathrm{\Delta }}_E/(kT))$ , where ${\mathrm{\Delta }}_E=E(y)-E(x_i)$ , T is the temperature, and k is the Boltzmann constant. If kT is large (in proportion to ${\mathrm{\Delta }}_E$ ), chances of acceptance, $x_{i+1}:=y$ , are high, otherwise they are low. ⁵

In terms of the acceptance–rejection method, the idea was to generate a candidate new solution, set the density $f(x_{\text{new}})$ in proportion to the majorizing density, and use this proportion as an acceptance probability. The twist here is that the reference point is the current point's density: the new solution is accepted with probability $p=\min (1,f(x_{\text{new}})/f(x_i))$ . If the new point has a higher density than the current one, it is accepted; if its density is lower in proportion to the current one, the chance of acceptance is also lower. For a d-dimensional variate x, the pseudocode is given in Algorithm 23.

Algorithm 23. Metropolis algorithm.

This algorithm produces a vector of n variates, and the stationary distribution of the $x_i$ s will be (proportional to) f. The parameter s governs the maximum step size in any direction. In principle, one is rather free in picking a value for s, though there are some general principles. When s is small, the current and the suggested value will be very similar—and so will be their densities. Hence, the acceptance ratio will be close to 1, and it should not take many attempts to get a new solution. But because of the similarity in values, there will be strong serial dependence between the $x_i$ s, and there can be clusters in some parts of the probability space while no samples from other relevant areas exist. Choosing a large s has the exact opposite effect: there will be less serial dependence in the $x_i$ s, but it will take more attempts to generate a new variate. Also, increasing the number of dimensions while keeping s fixed, the size of neighborhood will grow, and the overall step size in terms of the Euclidean distance between current and new points will increase. Reducing s with the square root of dimensions will counteract this effect. ⁶ If serial dependence is a problem and independent variates are required, shuffling can help (see Section 6.5.3). In either case, one must ensure that all new candidates are valid solutions and remain within the support. Finally, to avoid problems with unfortunate starting points, it is advisable to allow the process to burn in and discard initial samples.

Like the acceptance–rejection method, the Metropolis algorithm can readily be applied to multivariate distributions with known joint density. For an illustration, consider the bivariate normal distribution.

Fig. 7.2 depicts the 2000 samples generated for a required correlation of 0.75. The left panel uses a step size of s = 0.2, the right one of 1. To show the movements during the process, the $x_1$ is depicted in black, and subsequent points are increasingly brighter. As can be seen, larger d scatters the points more evenly over the probability space, yet it requires more run time.

Figure 7.2. Bivariate normal samples, generated with the Metropolis algorithm. Left panel: metropolisMVNormal(2000,2,0.2,0.75) (s = 0.2). Right panel: metropolisMVNormal(2000,2,1.0,0.75) (s = 1.0).

A generalized version of this is the Metropolis–Hastings algorithm (see Hastings, 1970). The idea is to simulate x by again drawing candidate solutions and accepting them according to probabilistic criterion, yet by using a majorizing distribution. Algorithm 24 provides some details.

Algorithm 24. Metropolis–Hastings.

Hastings (1970) generalized this approach further. Again, a majorizing or candidate-generating density g is required, whereas the actual target density is p. As for the previous example, let $x_i$ be the current solution and y be a new sample in its proximity. The probability of acceptance depends on the Hastings ratio, $r=\frac{p_x(y)}{p_x(x_i)}\frac{g_{y_{i+1}|y_i}(x_i|y)}{g_{y_{i+1}|y_i}(y|x_i)}$ . The main advantage of the Metropolis–Hastings algorithm is that the shape of the target does not need to be known. If the conditional distribution is known but not the joint, the Gibbs sampler can be used (see Geman and Geman, 1984).

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128150658000182

A Research Overview of Tool-Supported Model-based Testing of Requirements-based Designs

Raluca Marinescu , ... Paul Pettersson , in Advances in Computers, 2015

8.1 Markov Chains

A Markov chain [ 87] is simply a sequence of random values in which the next value is in some way dependent on the current value, rather than being completely random. To generate test-cases from models, one needs to represent the system behavior and its constraints in a non-ambiguous way. However, the respective system behavior changes depending on how the system is used, that is, assuming various computational loads, an experienced or novice user, different usage frequencies etc. Modeling notations like Markov chains are appropriate abstractions of such usage profiles that characterize the operational use of a software system.

Markov chains have a simple structure and consequently they provide an intuitive view of the system execution. Markovian models allow test input sequences to be generated from multiple probability distributions, making the notation more general than other existing ones. In addition, the stopping criteria of the test-case generation algorithms based on Markov chain representations could lead to the system's reliability assessment, via the included failure information. One of the obvious drawbacks of the model is its somewhat poor scalability. The transition probabilities in the model are stored in a matrix that becomes hard to comprehend as the states and their transitions accumulate.

In the following, we describe two of the MBT tools that rely on Markov chains, after which we apply one of them on our running example.

MaTeLo Testing Tool. This is a commercial tool that has been developed in an European project named Markov Test Logic (MaTeLo) [88] and it is mainly targeted at deriving test-cases from the intended usage of the software under test, modeled as a Markov chain. The focus of MaTeLo is not on requirements coverage (although this could be considered as a secondary goal) but on covering the ways in which the software is intended to be used, in order to detect the most critical faults that could appear from the respective use. The input to MaTeLo that describes the usage profile can be given as a Message Sequence Chart (MSC), a UML sequence diagram, or a state-chart.

J Usage Model Builder Library (JUMBL). This research prototype [39] provides support for the automated test-case generation from usage models described in TML, the modeling language used in the tool to describe probabilities as constraints, along with simple objective functions that facilitate defining the probability distribution for a usage model, similar to those in Markov models. In comparison to MaTeLo, JUMBL does not provide any usage model editor, instead it allows users to develop usage models as spreadsheets, in text editors, or graphical editors. Another difference from MaTeLo is that it uses three different algorithms to automatically generate test-cases: (i) Postman-based test-generation, (ii) Random-based test-generation, and (iii) Probability-based test-generation, as well as a way of manually generating test-cases in an interactive editor.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S0065245815000297

Stochastic Models

P.A. Ferrari , in International Encyclopedia of the Social & Behavioral Sciences, 2001

3.1 Coupling

Two or more Markov chains can be constructed with the same uniform random variables U _n . This is called a coupling and it is one of the major tools in stochastic processes. Applications of coupling include proofs of convergence of a Markov chain to its invariant measure, comparisons between chains to obtain properties of one of them in function of the other, and simulation of measures that are invariant for Markov chains.

Recent books on Markov chains include Chen (1992), Bianc and Durrett (1995), Fristedt and Gray (1997), Brémaud (1999), Durrett (1999a), Schinazi (1999), Thorisson (2000), Häggström (2000), and Ferrari and Galves (2000).

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767005921

Elements of queuing theory and queuing networks

Vasileios Karyotis Vasileios Karyotis , M.H.R. Khouzani M.H.R. Khouzani , in Malware Diffusion Models for Wireless Complex Networks, 2016

B.3.1 Discrete-time Markov Chains

A Markov chain discrete-time Markov chain - DTMC is a random process that undergoes transitions from one state to another on a state space. It must possess the "memorylessness" property , namely, that the probability distribution of the next state depends only on the current state and not on the sequence of events that preceded it. The material presented in this subsection will be useful for the study of queuing systems treated in the later subsections of Appendix B.3 and Chapters 4 and 5 Chapter 4 Chapter 5 .

More formally, a stochastic process $\{X_n,n=0,1,2,\dots \}$ that takes a finite or countable number of possible values is said to be in state $i$ at time $n$ if $X_n=i$ . We assume that there is a fixed probability $P_{ij}$ that if the process is in state $i$ it will next be in state $j$ ,

(B.2) $Pr\{X_{n+1}=j\mid X_n=i,X_{n-1}=i_{n-1},\dots ,X_1=i_1,X_0=i_0\}=Pr\{X_{n+1}=j\mid X_n=i\}=P_{ij}$

for all states $i_0,i_1,\dots ,i_{n-1},i,j$ and all $n\ge 0$ . Such a stochastic process is known as a (discrete) Markov chain. The Markovian property, expressed by Eq. (B.2), is that the conditional distribution of any future state $X_{n+1}$ given the past states $X_0,X_1,\dots ,X_{n-1}$ and the present state $X_n$ is independent of the past states and depends only on the present state. Thus, a Markov chain is a sequence of random variables $X_1,X_2,X_3,\dots$ with the Markov property. Since the process must make a transition into some state, ${\sum }_{j=0}^{\infty }{P}_{ij}=1,i=0,1,\dots ,P_{ij}\ge 0,i,j\ge 0$ .

A Markov chain is said to be homogeneous in time if the transition probability between two distinct states depends only on the time step difference, i.e. $Pr\{X_{m+n}=j\mid X_m=i\}=P_{ij}^n$ . Here, we focus on homogeneous Markov processes.

Eq. (B.2) defines the so-called one-step transition probabilities , namely probabilities that describe the transition of the chain in one time step. These transitions can be depicted via a state diagram, where a state diagram is a directed graph used to picture the state transitions. The states that the chain transitions to will be called neighboring states. Similarly, the $n$ -step probabilities that the process will be in state $j$ starting at $i$ after $n$ successive transitions ( $n$ successive time steps) are

(B.3) $P_{ij}^n=Pr\{X_{n+m}=j\mid X_m=i\}\text{,}n,i,j\ge 0\text{.}$

In order to compute the $n$ -step transition probabilities, the Chapman-Kolmogorov equations shown in the following can be employed:

(B.4) $P_{ij}^{n+m}=\sum _{k=0}^{\infty }{P}_{ik}^n{P}_{kj}^m\text{,}\text{for all}n,m\ge 0,\text{all}i,j\text{.}$

If $f_{ij}^n$ is the probability that starting in state $i$ , the first transition to $j$ occurs at time $n$ , then state $j$ is said to be recurrent if $f_{jj}=1$ , which means that starting at state $j$ the process will return to state $j$ with probability 1 for some time $n$ ( $f_{jj}={\sum }_{n=1}^{\infty }{f}_{ij}^n$ ). A state is transient if $f_{jj}<1$ . If state $j$ is recurrent and ${\mu }_j={\sum }_{n=1}^{\infty }n{f}_{jj}^n$ denotes the expected number of transitions needed to return to start $j$ (mean recurrence time - expected return time in state $j$ ), then state $j$ is positive recurrent if ${\mu }_j<\infty$ and null recurrent if ${\mu }_j=\infty$ . A state $j$ is said to be accessible from a state $i$ (written $i\to j$ ) if a system that started in state $i$ has a nonzero probability of transitioning into state $j$ at some point ( $Pr\{X_n=j\mid X_0=i\}=P_{ij}^n>0$ for some $n>0$ ). A state $i$ has period $k$ if any return to state $i$ must occur in multiples of $k$ time steps. If $k=1$ , then the state is said to be aperiodic, namely, returns to state $i$ can occur at irregular times. A Markov chain is aperiodic if every state is aperiodic and an irreducible Markov chain only needs one aperiodic state to imply all states are aperiodic. A state $i$ is called absorbing if it is impossible to leave this state.

A state $i$ is said to be ergodic if it is aperiodic and positive recurrent, namely, if it is recurrent, has a period of 1, and it has finite mean recurrence time. If all states in an irreducible Markov chain are ergodic, then the chain is said to be ergodic. It can be shown that a finite state irreducible Markov chain is ergodic if it has an aperiodic state.

If an irreducible aperiodic Markov chain is positive recurrent, then we can define as ${\pi }_j={lim}_{n\to \infty }{P}_{ij}^n$ , where $P_{ij}^n$ is obtained from (B.3) for $n=1$ . Let $S$ denote the set of possible states that the process can make a transition to. If there is a probability distribution over states $\pi$ such that

(B.5) ${\pi }_j=\sum _{i\in S}{\pi }_{i}Pr\{X_{n+1}=j\mid X_n=i\}$

for every state $j$ and every time $n$ then $\pi$ is a stationary distribution of the Markov chain. An irreducible chain has a stationary distribution if and only if all of its states are positive recurrent. In that case, $\pi$ is unique and related to the expected return time $M_j$ by ${\pi }_j=\frac{C}{M_j}$ , where $C$ is a normalizing constant independent of the state. The chain converges to the stationary distribution regardless of the initial state distribution. Such $\pi$ is called the equilibrium distribution of the chain. For a finite state space, a stationary distribution $\pi$ is a (row) vector, whose entries are non-negative and sum to 1, it is unchanged by the operation of transition matrix $P$ and it is defined by $\pi P=\pi$ . In addition, $0\le {\pi }_j\le 1$ and ${\sum }_{j\in S}{\pi }_j=1$ . Then the transition probability distribution can be represented by the transition matrix $P$ , with the $(i,j)$ th element of $P$ equal to ${\pi }_j=Pr\{X_{n+1}=j\mid X_n=i\}$ .

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128027141000232

Multistate Transition Models in Demography

D. Courgeau , in International Encyclopedia of the Social & Behavioral Sciences, 2001

1.2 More Complex Models

The Markov chain assumption is restrictive and constitutes a rough approximation for many demographic processes. For example, in migration analysis one needs to account for duration dependence in the propensity to move. There are also preferred patterns of moves between regions; for instance, return moves back to a region of origin are preferred to moves to other regions. Such features can be taken into account using more complex models.

The first way to analyze non-Markovian processes is to expand the state space so that the process in the new space is Markovian. Such an extension, however, again leads to an increase of the amount of data necessary to estimate the large number of transition intensities beyond what is possible with the usual data sets. A second possibility lies in the use of a semi-Markov process. In such a model the probability of moving at a given time can bear any arbitrary relation to the duration of stay (Ginsberg 1971). Nevertheless, any extension leaves out some property of the studied process and even more complex models are needed. For example, in the case of migration a semi-Markov process is unable to take into account the higher probability of return moves unless one introduces information about previously occupied states.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767021021

tranhaday1969.blogspot.com

Source: https://www.sciencedirect.com/topics/computer-science/markov-chain

Share this post