CHAPTER 12
MARKOV PROCESS MODELS
TRUE/FALSE QUESTIONS
1. A stage in a Markov process always corresponds to a fixed time
2. Although the number of possible states in a Markov process may
3. “How you arrived at where you are now has no bearing on where
you go next.” This, simply put, is the Markovian memoryless
4. In a Markov process, an absorbing state is a special type of
5. All Markov processes exhibit some form of periodicity.
6. In a Markovian transition matrix, each column’s probabilities
7. Markovian transition matrices are necessarily square. That
is, there are exactly the same number of rows as there are columns.
8. Every Markov process has at least one absorbing state.
9. State probabilities for any given stage must sum to 1.
10. The values towards which state probabilities converge are the
11. For Markov processes with absorbing states, steady-state
behavior is independent of the initial process state.
12. If all the rows of a transition matrix are identical, there
13. All Markov processes eventually converge to a steady-state.
14. Once a process reaches steady–state, the state probabilities
15. Markov processes are a powerful decision making tool useful in
explaining the behavior of systems and determining limiting
MULTIPLE CHOICE QUESTIONS
1. All of the following are necessary characteristics of a Markov
process except:
a. a countable number of stages.
b. a countable number of states per stage.
c. at least one absorbing state.
d. the memoryless property.
2. Which of the following is a necessary characteristic of a
Markovian transition matrix?
a. Periodicity.
b. Column numbers sum to 1.
c. Square (number of rows = number of columns).
d. Singularity.
3. Consider the transition matrix:
| .3 .2 .5 |
| .1 .6 .3 |
| .2 .3 .5 |
The steady-state probability of being in state 1 is approximately:
a. .177
b. .231
c. .300
d. .403
4. The transition matrix
| 0 1 0 |
| 0 0 1 |
| 1 0 0 |
represents what type of Markov process?
a. Periodic.
b. Absorbing.
c. Independent.
d. Nonrecurrent.
5. | 0.2 0.8 |
| 0 1 |
This transition matrix represents what type of Markov process?
a. Periodic.
b. Absorbing.
c. Independent.
d. Nonrecurrent.
6. Regarding a transition matrix which possesses an absorbing
state:
a. All row values will not sum to 1.
b. There will be a complementary absorbing state.
c. At least two columns will be identical.
d. That state’s row will consist of a “1” and “0’s”.
7. If a Markov process consists of two absorbing states and two
nonabsorbing states, the limiting probabilities for the nonabsorbing
states will:
a. both equal zero.
b. be 0.5 and 0.5.
c. be identical to the transient state probabilities.
d. depend on the state vector.
8. A state vector is used for determining the:
a. number of stages until steady-state is reached.
b. probability that the process is in a given state.
c. existence of absorbing states.
d. values of transient state probabilities.
9. Steady-state probabilities are independent of the initial
state if:
a. the number of initial states is finite.
b. there are no absorbing states.
c. the number of states and stages are equal.
d. the process generates a fixed number of transient
states.
10. A Markovian system is currently at stage 1. To determine the
state of the system at stage 6, we must have, in addition to the
transition matrix, the state probabilities at:
a. stage 5.
b. stage 1.
c. any stage, up to and including 5.
d. no stage values are needed.
11. The “mean recurrence time” for a state in a Markov process:
a. is the average time it takes to return to that given
state.
b. is the complement of the steady-state value.
c. only applies to processes with absorbing states.
d. depends upon the total number of stages involved.
12. Retired people often return to the workforce. If a retired
woman returns to work at the same place from which she retired —
even if only part time or for a limited term — that signifies that
retirement is not a:
a. transient state.
b. steady-state.
c. periodic state.
d. absorbing state.
13. A firm displeased with its projected steady-state market share
may try to improve the situation by taking steps which hopefully
will:
a. extend the number of stages.
b. alter the transition matrix.
c. better its transient state standing.
d. reduce the number of recurrent states.
14. A gambler has an opportunity to play a coin tossing game in
which he wins his wager with probability .49 and loses his wager
with probability .51. Suppose the gambler’s initial stake is $40 and
the gambler will continue to make $10 bets until his fortune either
reaches $0 or $100 (at which time play will stop). Which of the
following statements is true?
a. Increasing the amount of each wager from $10 to $20 will
increase the expected playing time.
b. Increasing the initial stake to $50 will increase the
expected playing time.
c. Reducing the initial stake to $20 will increase the
expected playing time.
d. Increasing the probability of winning from .49 to 1.0
will increase the expected playing time.
15. In determining steady-state behavior for a process with
absorbing states, the subdivision of the transition matrix yields:
a. an identity submatrix, but no zero submatrix.
b. no identity submatrix, but a zero submatrix.
c. both an identity submatrix and a zero submatrix.
d. neither an identity submatrix nor a zero submatrix.
16. The state vector for stage j of a Markov chain with n states:
a. is a 1 x n matrix.
b. contains transition probabilities for stage j.
c. contains only nonzero values.
d. contains the steady-state probabilities.
17. If we perform the calculations for steady-state probabilities
for a Markov process with periodic behavior, what do we get?
a. Steady-state probabilities.
b. An unsolvable set of equations.
c. The fundamental matrix.
d. The long run percentage of time the process will be in
each state.
18. What is the fundamental matrix for a Markov process with
absorbing states?
a. A matrix composed of the identity submatrix, a zero
submatrix, a submatrix of the transition probabilities
between the non-absorbing states and the absorbing
states, and a submatrix of transition probabilities
between the non-absorbing states.
b. A matrix representing the average number of times the
process visits the non-absorbing states.
c. The inverse of the identity matrix minus the matrix of
the transition probabilities between the non-absorbing
states and the absorbing states.
d. The matrix product of the limiting transition matrix and
the matrix of transition probabilities between the non–
absorbing states.
19. For a Markov process with absorbing states, we define
Π(j) = state vector at stage j
N = fundamental matrix
I = identity matrix
Q = matrix of transition probabilities between non–
absorbing states
R = matrix of transition probabilities between non–
absorbing states and absorbing states
The limiting state probabilities equal:
a. Π(1) * N * R
b. I * R * Q
c. (I – Q)-1
d. Π(1) * N * Q
20. If we add up the values in the n rows of the fundamental
matrix for a Markov process with absorbing states, what is the
result?
a. The rows each add to 1.
b. The limiting probability for each state.
c. A meaningless number.
d. The mean time until absorption for each state.
SHORT ANSWER QUESTIONS
1. What is the steady-state significance, if any, of a zero in
the transition matrix?
2. In a Markovian system, is it possible to have only one
absorbing state?
3. Charles dines out twice a week. On Tuesdays, he always
frequents the same Mexican restaurant; on Thursdays, he randomizes
between Greek, Italian, or Thai (but never Mexican). Is this
transient, periodic, or recurrent behavior?
4. In a Markov process, what determines the duration of a stage?
5. The Department of Motor Vehicles (DMV) has 4 stations for
driver’s license renewal:
– fee payment
– eyesight test
– driving record check
– picture taking.
An applicant may start at any station and go from any station to any
other station. Generally, an applicant will go to the unvisited
station with the shortest line. If we model the stations as
“states,” can we use a Markov chain to model the DMV renewal
process?
6. What is the minimum percentage of transition probabilities
that must be nonzero?
7. Is this an acceptable transition matrix? Explain your answer.
| .3 .3 .4 0 |
| .2 .5 0 .3 |
| .1 .6 .2 .1 |
8. When calculating steady-state probabilities, we multiply the
vector of n unknown values times the transition probability matrix
to produce n equations with n unknowns. Why do we arbitrarily drop
one of these equations?
9. Is this an identity matrix? Explain your answer.
| 0 1 0 |
| 1 0 0 |
| 0 0 1 |
10. Define these Excel functions:
MINVERSE()
INDEX()
MMULT().
FORMULATION/SOLUTION/ANALYSIS QUESTIONS
1. Assume a three-brand market for ordinary, blended scotch
whiskey: Sutty Cark, B & J, and Walkin’ Johnnie (Red). It is a
static, closed market in the short term — i.e., no “leakage”
to/from other brands, and a fixed market size. Customers typically
buy one liter a month. Brand loyalty/disloyalty is shown in the
following transition matrix:
Next Month
SC BJ WJ
SC 0.6 0.3 0.1
This Month BJ 0.05 0.9 0.05
WJ 0.4 0.4 0.2
Last month, BJ has 40% of the market, the remainder being equally
split between SC and WJ.
A. Assuming constant transition probabilities, what share of the
market will Walkin’ Johnnie (Red) have next month?
B. What are the steady state market shares for each of the three
2. Gleason’s Department Store customers who use the store’s in–
house charge card are expected to pay for their purchases in the
month billed (the month following the purchase date). As a matter of
course, only 60% of customers pay on time, while the remaining 40%
defer payment and incur financing charges. The following transition
matrix describes the age of Gleason’s charge accounts.
Account Age Next Month
1 2 3 4 Paid Uncollectible
Account 1 0 .4 0 0 .6 0
Age 2 0 0 .5 0 .5 0
This 3 0 0 0 .6 .4 0
Month 4 0 0 0 0 .2 .8
Paid 0 0 0 0 1 0
Uncollectible 0 0 0 0 0 1
What percentage of this month’s store charge card purchases will
eventually be collected?
3. Every week a charter plane brings a group of high-stakes
gamblers into Las Begas. Half the group stays and begins gambling at
Hot Slots near the Strip, and the other half is housed and begins
gambling at Better Bandits, some distance away. (Both hotel/casinos
are owned by the same corporation.)
Once an hour, dedicated shuttle buses will transport to the other
casino any of the group members who wish to try their luck at the
other casino. The transition probabilities are as follows:
Next Hour
Hot Slots Better Bandits
This Hot Slots .8 .2
Hour Better Bandits .3 .7
A. After three hours, what proportion of these gamblers are in
the Hot Slots Casino?
B. What is the long run average percentage of these gamblers who
will be at Hot Slots Casino?
4. The transition matrix for customer purchases of alkaline
batteries is believed to be as follows:
Next Purchase
Duracell Eveready Other
Current Duracell .43 .35 .22
Purchase Eveready .38 .45 .17
Other .13 .25 .62
A. Based on this transition matrix, what is Duracell’s market
share for the alkaline battery market?
B. Each 1% of the market share of the alkaline battery market is
worth $3.2 million in profit. Suppose that Duracell is contemplating
an advertising campaign which it believes will result in the
transition probabilities for battery purchases to be as follows:
Next Purchase
Duracell Eveready Other
Current Duracell .47 .31 .22
Purchase Eveready .38 .40 .22
Other .22 .25 .53
What is the most that Duracell should be willing to pay for this
campaign?
5. Suppose you play a coin flipping game with a friend in which a
fair coin is used. If the coin comes up heads you win $1 from your
friend, if the coin comes up tails, your friend wins $1 from you.
You have $3 and your friend has $4. You will stop the game when one
of you is broke. Determine the probability that you will win all of
your friend’s money.
6. A simple computer game using a Markov chain starts at the Cave
or the Castle with equal probability and ends with Death (you lose)
or Treasure (you win). The outcome depends entirely on luck. The
transition probabilities are:
Next State
Cave Castle Death Treasure
Cave .4 .3 .2 .1
Current Castle .5 .3 .1 .1
State Death 0 0 1 0
Treasure 0 0 0 1
A. What is the average number of times you would expect to visit
the Cave and the Castle, depending on which state is the starting
state?
B. What is the mean time until absorption for the Cave and the
Castle?
C. What is the likelihood of winning the game?