Definitions

Let be a Markov chain with transition matrix . Define

using the convention . Remark They are different only if .

We can definte the -th passage time to , setting and for

and the excursion time beetween visits

this explain our choice for .

The following lemma should be obvious. Lemma For the random variable is independent of and for all

Proof Since is a stopping time, by the strong Markov property

using the definition of the excurtion time

so the probability

Other useful definitions are:

Def (Recurrent state) We say that the state is recurrent if

a non-recurrent state is called transient.

There is a link beetween recurrent and and return probability.

Lemma For all , . Proof By induction. The base case is true (with the convention ). Assume it holds for .

or equivalently using the excurtion time

using the definition of conditional probability

using the strong Markov property

since is the same event as .

Characterization

Theorem We have the dichotomy:

  1. se , allora è ricorrente e
  1. se , allora è transiente e

Proof The probability that the state is recurrent, i.e. the number of visits is infinite is

where we have used Continuity of mesure and the previous lemma. Since , we also know that

where we have used Fubini’s theorem.

Doing the same computation for a state such that we get

computing the expected value we get

but also

and the theorem is proven.

Corollary Every state is either transient of recurrent.

Class property

We can show that being recurrent or transient is a class property.

Theorem Let be a communicating class. Then either all state in are transient or all are recurrent.

Proof Take any pair and suppose is transient, let’s show that is also transient. Since then such that

for all

then

where we have used the fact that is transient, and the dichotomy. This proves that also is transient.

We have proven that if a state is transient, then the whole class is transient. This means either state are all recurrent, or all transient.

Theorem Every recurrent class is closed.

Proof Let’s show that if is open, then it’s transient. Since is open, then and such that (but not since . This means there exist such that

Theorem Let be an irreducible recurrent markov chain. Then for every initial distribution every state is visited a.s:

Proof Using the w.m.p

since , if we prove that for all we have done. We now use the recurrence hypotesis

choose an arbitrary time

since the first sum is less or equal to (a finite number)

since the second terms add to one, this implies that