recurrence in a Markov chain

Let $\{X_{n}\}$ be a stationary (http://planetmath.org/StationaryProcess) Markov chain and $I$ the state space. Given $i,j\in I$ and any non-negative integer $n$ , define a number $F_{ij}^{n}$ as follows:

F_{ij}^{n}:=\begin{cases}0&\text{if }n=0,\\ P(X_{n}=j\mbox{ and }X_{m}\neq j\mbox{ for }0<m<n\mid X_{0}=i)&\text{otherwise% }.\end{cases}

In other words, $F_{ij}^{n}$ is the probability that the process first reaches state $j$ at time $n$ from state $i$ at time $0$ .

From the definition of $F_{ij}^{n}$ , we see that the probability of the process reaching state $j$ within and including time $n$ from state $i$ at time $0$ is given by

\sum_{m=0}^{n}F_{ij}^{m}.

As $n\to\infty$ , we have the limiting probability of the process reaching $j$ eventually from the initial state of $i$ at $0$ , which we denote by $F_{ij}$ :

F_{ij}:=\sum_{m=0}^{\infty}F_{ij}^{m}.

Definitions. A state $i\in I$ is said to be recurrent or persistent if $F_{ii}=1$ , and transient otherwise.

Given a recurrent state $i$ , we can further classify it according to “how soon” the state $i$ returns after its initial appearance. Given $F_{ii}^{n}$ , we can calculate the expected number of steps or transitions required to return to state $i$ by time $n$ . This expectation is given by

\sum_{m=0}^{n}mF_{ii}^{m}.

When $n\to\infty$ , the above expression may or may not approach a limit. It is the expected number of transitions needed to return to state $i$ at all from the beginning. We denote this figure by $\mu_{i}$ :

\mu_{i}:=\sum_{m=0}^{\infty}mF_{ii}^{m}.

Definitions. A recurrent state $i\in I$ is said to be or strongly ergodic if $\mu_{i}<\infty$ , otherwise it is called null or weakly ergodic. If a stronly ergodic state is in addition aperiodic (http://planetmath.org/PeriodicityOfAMarkovChain), then it is said to be an ergodic state.

Title	recurrence in a Markov chain
Canonical name	RecurrenceInAMarkovChain
Date of creation	2013-03-22 16:24:43
Last modified on	2013-03-22 16:24:43
Owner	CWoo (3771)
Last modified by	CWoo (3771)
Numerical id	5
Author	CWoo (3771)
Entry type	Definition
Classification	msc 60J10
Synonym	null recurrent
Synonym	positive recurrent
Synonym	strongly ergodic
Synonym	weakly ergodic
Defines	recurrent state
Defines	persistent state
Defines	transient state
Defines	null state
Defines	positive state
Defines	ergodic state