Corrigendum to: “Discounted Stochastic Games with No Stationary Nash Equilibrium: Two Examples” ∗

Levy (2013) presents examples of discounted stochastic games that do not have stationary equilibria. The second named author has pointed out that one of these examples is incorrect. In addition to describing the details of this error, this note presents a new example by the ﬁrst named author that succeeds in demonstrating that discounted stochastic games with absolutely continuous transitions can fail to have stationary equilibria.


Introduction
The paper "Discounted Stochastic Games with No Stationary Nash Equilibrium: Two Examples" (Levy (2013)) presents two constructions of discounted stochastic games with continuous state spaces that do not possess stationary equilibria. One is in the class of stochastic games with deterministic transitions, while the other is in the class of games in which all transitions are absolutely continuous with respect to a fixed measure.
The construction of the game from the second class is accomplished in three steps. 1. An example presented in Kohlberg and Mertens (1986) has a circular set of Nash equilibria, and a map from the set of equilibria to small perturbations of the game is constructed. 2. An intricate construction concerning realising piece-wise linear functions on the square as outcomes of strategic games is presented. 3. These techniques are combined to define the desired stochastic game.
• A bounded Borel-measurable stage payoff function r : Ω × I → R P .
For a discount factor β ∈ (0, 1), Γ(β) is the associated discounted stochastic game. Throughout (except in the statements of some results) we work with a fixed β.
Definition 2.1. An Absolutely Continuous (A.C.) stochastic game is a stochastic game Γ = Ω, P, I, r, q for which there is a ν ∈ ∆(Ω) such that for all z ∈ Ω and a ∈ I, q(z, a) is absolutely continuous with respect to ν.
A stationary strategy for player ℓ is a Borel-measurable mapping σ ℓ : Ω → ∆(I ℓ ). Let Σ ℓ 0 denote the set of stationary strategies for player ℓ, and let Σ 0 = ℓ∈P Σ ℓ 0 be the set of stationary strategy profiles. Together with the transition function and an initial state z, a stationary strategy profile σ induces a probability measure P σ z on the space H ∞ := (Ω × I) N of infinite histories in a canonical way (see, e.g., Bertsekas and Shreve (1996)). Let γ σ (z) := E σ z ∞ n=1 β n−1 r(z n , a n ) be the expected payoff vector under σ in the game starting from state z = z 1 . A profile σ ∈ Σ 0 is a stationary equilibrium of Γ(β) if for all z ∈ Ω, ℓ ∈ P, and τ ∈ Σ ℓ 0 . For z ∈ Ω and a ∈ I let X σ (z, a) := r(z, a) + β Ω γ σ dq(z, a). (2.1) Note that γ σ (z) = X σ (z, σ(z)). We recall the following classical dynamical programming criterion for a stationary equilibrium, which is called the one-shot deviation principle.
Proposition 2.2. A profile σ ∈ Σ 0 of stationary strategies is a stationary equilibrium of Γ(β) if and only if, for all z ∈ Ω, σ(z) is a Nash equilibrium of the game X σ (z, ·).

The Example
This section presents the example (or class of examples, insofar as there is a given function that is a parameter) of an A.C. stochastic game that does not possess stationary Nash equilibria for any positive discount factor.

Notations
Recall that ·, · denotes the inner product of vectors. In addition the following notational conventions will be used: • Throughout · denotes the L ∞ norm. That is, for a vector or bounded real-valued function f , ||f || = sup |f |, where the supremum is taken over the set of indices or the domain of f .
• If p is a mixed action over an action space I and i ∈ I, then p[i] denotes the probability that p chooses i.

The Base Game
Our construction has three phases: a) selecting four perturbations of a "base" game; b) specification of a rescaled version of the stage game; c) the stochastic game itself. The base game G has four players, A, B, C, and D. The pure strategies of player A are U and D, the pure strategies of B are L, M , and R, and players C and D are dummy players, because their sets of pure strategies are singletons. The payoffs of players A and B are shown below.
In view of (b) and the bounds on payoffs for C and D, the upper semicontinuity of the Nash equilibrium correspondence implies that there is an η 0 > 0 such that 7 8 ≤ ||G C,D (x)|| ≤ 1 whenever x is an equilibrium of a game G ′ such that ||G ′ − G|| ≤ η 0 . For each (j, k) ∈ {−1, 1} 2 we fix such a perturbation G j,k of G such that the unique Nash equilibrium x of G j,k satisfies G C,D (x) = (j, k). (The payoffs of A and B in G j,k play no role in our analysis after Lemma 3.1 has been established.) For each z = (z E , z F ) ∈ [−1, 1] 2 let G z be the convex combination of the (G j,k ) given by

The Stage Game
Next we describe a second strategic form game; in our stochastic game the stage game in each state will be a rescaling of this game. The set of players is P = {A, B, C, C ′ , D, D ′ , E, F }. As above, player A has the pure strategies U and D, and player B has the pure strategies L, M , and R, but in this game players C and D have pure strategies 0 and 1. Players C ′ and D ′ also have pure strategies 0 and 1, and players E and F have pure strategies −1 and 1. Pure and mixed strategy profiles will be denoted by The payoffs in the stage game depend on a parameter ̺ ∈ (− 1 2 , 1 2 ). For such a ̺ let v ̺ E = (1, ̺) and v ̺ F = (−̺, 1). Let The payoffs in the game g(̺, ·) are: In the stochastic game given below, the transitions are controlled by C, C ′ , D, and D ′ , so in each period the other players will only be concerned with maximizing their stage payoffs. Players A and B are in effect playing a perturbation of the game G, as described above.
The stage game payoff to C ′ is the negation of the stage game payoff to C, so C and C ′ will have opposite views concerning the desirability of the stochastic game continuing (as opposed to transitioning to an absorbing state with zero payoffs). Leaving aside the components of the stage game payoffs for C and C ′ that depend only on the behavior of A and B, the conflict between C and C ′ at time t is a zero sum game that consists of matching pennies perturbed by these concerns about future payoffs. These perturbations will be small enough that there is always a unique equilibrium which is mixed.
The conflict between D and D ′ is similar to the conflict between C and C ′ . However, the impact of A and B's behavior on the payoffs of D and D ′ is different from its impact on the payoffs of C and C ′ . Consequently the perturbation of the matching pennies game, and the resulting stage game equilibrium, will almost surely be different.
The best responses of players E and F depend on the signs of the expectations of the inner products v ̺ E , ψ(a) and v ̺ F , ψ(a) respectively. For ̺ ∈ (− 1 2 , 1 2 ) and j, k = ±1 let In the stochastic game defined below ̺ will be a function of the state t ∈ [0, 1], and we will see that in any stationary equilibrium, for almost all t, behavior at 6 state t is characterized by a mixed strategy profile x such that ψ(x) lies in D ̺(t) , so that E and F play pure strategies, and consequently A and B are playing one of the perturbations G j,k of G. In this sense the behavior of A and B is well controlled. v
• The set of players and their action spaces are the same as in g(̺, ·).
The gameΓ has the following features. First, 1 is an absorbing state, with payoff 0 for all players. The transitions from state t are mixtures of two types: U t , which distributes uniformly in [t, 1], or quitting to 1. As such, the game progresses towards the right. Note thatΓ is A.C. because for all t ∈ [0, 1] and all a ∈ I, q(t, a) is absolutely continuous w.r.t. 1 2 (U 0 + δ 1 ). Also, the stage payoffs ofΓ are continuous functions on Ω × I when ̺ is continuous.
To see the key intuition underlying the construction, suppose that σ is a stationary equilibrium ofΓ(β). The stage game payoff is by far the largest part of γ C,D σ (t), and the contribution of the perturbation of G is by far the largest part of the stage game payoff. Therefore γ C,D σ (t) = (0, 0) when t < 1.
is a bounded measurable function, W C,D is absolutely continuous and consequently, for almost every t, differentiable at t with derivative equal to −β times γ C,D σ (t). Since γ C,D σ (t) = (0, 0) when t < 1, W C,D is not identically equal to the origin in R 2 . Because ̺ is erratic, for almost all t such that W C,D (t) = (0, 0), the best responses of E and F are pure, leading the perturbation of the base game to be one of the G j,k whose equilibrium pushes the vector of future payoffs of C and D away from the origin in R 2 as we go forward in time (i.e., toward the right), which is to say that the derivative of s → W C,D (s) is positive at t, for almost all t. Since W C,D is absolutely continuous and W C,D (1) = (0, 0), this is impossible, which is the desired contradiction.
An essential feature of the construction is that G does not have any equilibria that give expected utility zero to both C and D, but nonetheless the origin is in the convex hull of the set of pairs of expected payoffs for C and D induced by the equilibria of G. For this reason one cannot replace the base game with a single agent decision problem: for a decision problem the set of optimal mixed strategies, and its image in the set of pairs of expected payoffs for C and D, are both convex. In addition, one cannot replace C and D with a single agent C: if every neighborhood of G contained a game whose unique equilibrium gave agent C an expected payoff of 1, and also a game whose unique equilibrium gave an expected payoffs of −1, then (as a consequence of a theorem of Browder (1960) and Mas-Colell (1974)) every neighborhood of G would contain games with equilibria giving C an expected payoff of 0.
To pass from Proposition 3.2 to the existence of a game without a stationary equilibrium it remains to show that there is a measurable function ̺ that disagrees a.e. with any a.e. differentiable function. It turns out that it is not enough to require that ̺ be nowhere differentiable, but there is a stronger condition that works. Let λ denote Lesbesgue measure.
Clearly, if f is differentiable at x with f ′ (x) = L, then f is approximately differentiable at x with approximate derivative L. The following is included in Theorem 3.3 of (Saks, 1937, Sec VII.3) 5 : Lemma 3.4. If f, g : [0, 1] → R are Lesbesgue measurable, f is approximately differentiable a.e., g is approximately differentiable almost nowhere, and E = { x : f (x) = g(x) }, then λ(E) = 0. Berman (1970) shows that, with probability one, the path of a Brownian motion is nowhere approximately differentiable; by definition the path of a Brownian motion is continuous. The existence of almost nowhere approximately differentiable continuous functions is also shown more directly in Jarník (1934); see also Preiss and Zajïcek (2000) and the references within. Consequently: Theorem 3.1. There exists stochastic games of the formΓ, for continuous ̺, such that for each β ∈ (0, 1),Γ(β) does not possess a stationary equilibrium.

Proof of Proposition 3.2
Let ̺ : [0, 1] → (− 1 2 , 1 2 ) satisfy the hypothesis of Proposition 3.2. Fix a discount factor β ∈ (0, 1). By way of contradiction, we suppose that σ is a stationary equilibrium ofΓ(β). We first introduce a new function, along with its most basic properties, after which our analysis has two phases.
Proof. For any t, γ C (t) and γ C ′ (t) are the expectations of random variables, each of which is the negation of the other, and similarly for γ D (t) and γ D ′ (t).
Proof. The probability of the game continuing (i.e., of the game not going to the absorbing state 1) is never greater than 1 16 , and ||r C,C ′ ,D,D ′ || ≤ 17 16 , so
For the sake of more compact notation we write x in place of σ(t) and ̺ and ω in place of ̺(t) and ω(t) in the remainder of this subsection, which extracts the relevant consequences of x being an equilibrium of g ω (̺, ·). Recall the definition of ψ(x) given in (3.1), and denote Equilibrium analysis for g ω (̺, ·) has the following consequences: Proof. Here (a) follows from g A,B ω (̺, ·) = g A,B (̺, ·). Observe that g C,C ′ ω (̺, x) is the sum of which is unaffected by x C,C ′ , and 1 16 times the payoffs resulting from applying x C,C ′ to the bimatrix game below.
It remains only to prove Proposition 4.7. Two lemmas prepare the main argument.
(d) By symmetry, the proof of (c) also establishes (d).
Proof of Proposition 4.7. Let t ∈ [0, 1) be such that all the properties of Lemma 4.9 hold. To simplify notation we drop the argument t. The chain rule gives (The final inequality is from Lemma 4.9(a).) Therefore we may suppose that one of these holds, say the first without loss of generality, and the other does not, so |W D | < 1 2 |W C |. Since |V D | ≤ 19 16 (1 − t) (Lemma 4.8),

Description of the Error
The error in Levy (2013) has the following description. Part (iv) of the Proposition 4.1 on page 1991 is not correct. On page 1990 there is the following game 13 with ε positive and small: (When ε = 0 this is an examples from Appendix B of Kohlberg and Mertens (1986).) As the paper points out, this has the pure equilibria (L, L) and (M, M ) and the mixed equilibrium The problem, which concerns (iv) of Proposition 4.1, arises from the fact that these are not the only equilibria. Indeed, in no equilibrium can both players use the strategy R with positive probability due to dominance. However, there are equilibria in which one player uses R. Specifically, the entire set of equilibria is M } (here con denotes the convex hull) and P 2 is the image of this under transposition of the players.
In Proposition 4.1 E x⊗y [ϑ] is a bilinear R 2 -valued function of (x, y), and (iv) of Proposition 4.1 holds if and only if the first component of E x⊗y [ϑ] is close to 1 for all (x, y) ∈ N E but for ε 2 L+(1− ε 2 )R, L and ε 2 L+(1− ε 2 )R, 2ε 2+ε L+ 2−ε 2+ε M this is quite far from being the case.
As we mentioned at the outset, Section 4.6 of Levy (2013) specifies conditions on a game (which would have the role played by the Kohlberg-Mertens game in the overall construction) that would allow the construction to succeed, but Corollary 5.1 below implies that they cannot hold.
Theorem 5.1. Let X be a compact convex subset of a locally convex topological space, let U ⊂ X be open with U compact, let F : U → X be an upper semicontinuous convex valued correspondence with no fixed points in U \ U , let P be a compact absolute neighborhood retract, and let ρ : U → P be a continuous function. If the fixed point index of F is not zero, then there is a neighborhood V of F in the (suitably topologized) space of upper semicontinuous convex valued correspondences from U to X such that for any continuous function g : P → V there is a p ∈ P and a fixed point x of g(p) such that ρ(x) = p.
To obtain the following result as a consequence of this, let X be the set of mixed strategy profiles of G, let F be its best reply correspondence, and for e ∈ P , let g(e) be the best response correspondence of h(e).
Corollary 5.1. If G is a finite strategic form game, NE is its set of Nash equilibria, P is a compact subset of NE that is an absolute neighborhood retract 7 , U is a neighborhood of NE in the space of mixed strategy profiles, and ρ : U → P is a retraction, then there is a neighborhood W of G in the space of games (for the given strategic form) such that for any continuous h : P → W there is some e ∈ P such that ρ −1 (e) contains a Nash equilibrium of h(e).