Let be a Markov chain on the state space having transition matrix with elements . Let be the function with and . Find a function such that
is a martingale relative to the filtration generated by the process .
Notice that since are measurable functions, then is composed of linear combinations of -measurable functions and hence is -adapted. Furthermore, for any , is finite everywhere, hence is .
Therefore, we only need to check the conditional martingale property, i.e. we want to show .
That is, we want
Therefore, if is to be a martingale, we must have
Since , we can compute the right hand side without too much work.
This explicitly defines the function and verifies that is a martingale.