Approachability with Discounting and the Folk Theorem Guilherme Carmona

Hamid Sabourian

University of Surrey

University of Cambridge

August 6, 2015

Abstract We establish a version of Blackwell’s (1956) approachability result with discounted and use it to obtain a perfect monitoring Folk Theorem with mixed strategies and finite automata.

Journal of Economic Literature Classification Numbers: C72; C73; C79 Keywords: Approachability; Repeated Games; Folk Theorem.

1

Introduction

Blackwell’s (1956) approachability result has found several applications in game theory, namely on repeated two-person, zero-sum games (see, e.g., Zamir (1992)). In particular, it involves the (arithmetic, undiscounted) average of a sequence of public signals that players observe and it is often the case that Blackwell’s (1956) approachability result can be used, directly or indirectly, to derive properties of the discounted average of sequences of public signals. An example of the above is the setting of Gossner (1995) which considers finitely repeated games with no discounting and uses Blackwell’s (1956) approachability result to obtain a Folk Theorem. Because the time horizon is finite, it is clear that the same conclusion holds with discounting. However, sometimes Blackwell’s (1956) approachability result cannot be used without difficulties in settings where players discount payoffs. A recent example includes Barlo, Carmona, and Sabourian (2015) who, in particular, use Gossner’s (1995) approach to obtain a Folk Theorem with bounded memory strategies in infinitely repeated games with discounting. Motivated by the above, we establish a version of Blackwell’s (1956) approachability result with discounted. This version can indeed be used in the setting of Barlo, Carmona, and Sabourian (2015). This is illustrated by establishing a perfect monitoring Folk Theorem with mixed strategies and finite automata, which builds on ideas of Barlo, Carmona, and Sabourian (2015) but, by dispensing with the bounded memory requirement, allows for weaker assumptions and an considerably easier proof. Unfortunately no other applications yet.

2

Approachability with discounting

We consider a setting similar to that of Blackwell (1956). There are two players, 1 and 2, who interact in every period t ∈ N = {1, 2, . . .}. In every such period, player 1 chooses an action from a finite set A1 = {1, . . . , r}, with r ∈ N, and player

2

2 from A2 = {1, . . . , s} with s ∈ N. Players are allowed to randomize, i.e. choose elements of ∆(A1 ) and ∆(A2 ) respectively.1 As in Blackwell (1956), let P = ∆(A1 ) and Q = ∆(A2 ). Each player observe neither the mixed choice made by the other player nor (necessarily) the realization. Instead, both players observe a public signal from a finite subset X of RN , with N ∈ N.2 For each i ∈ A1 and j ∈ A2 , let m(i, j) ∈ ∆(X) be the probability distribution on X when player 1 chooses i and player 2 chooses j. We let mx (i, j) denote the probability that the signal x is observed when players choose (i, j), for each x ∈ X. As in Blackwell (1956), M denotes the r × s matrix with generic element m(i, j), with 1 ≤ i ≤ r and 1 ≤ j ≤ s. For any t ≥ 1, a t-stage public history is a sequence h = (x1 , . . . , xt ) ∈ X t (the t-fold Cartesian product of X). The set of all t-stage public histories is denoted by Ht = X t . We represent the initial (empty) public history by ∅ and let H0 = {∅}. The S set of all public histories is defined by H = t∈N0 Ht . A (behavior, public) strategy for player i ∈ {1, 2} is a function fi : H → ∆(Ai ) mapping public histories into mixed actions. The set of player i’s strategies is denoted by Fi , and F = F1 × F2 . We let f denote a generic element of F1 and g a generic element of F2 . Given (f, g) ∈ F , h ∈ H, i ∈ A1 and j ∈ A2 , fi (h) denotes the probability that action i is played by player 1 and gj (h) denotes the probability that action j is played by player 2. Given a strategy (f, g) ∈ F , for each t ∈ N, x ∈ X and (x1 , . . . , xt ) ∈ X t , let βx (x1 , . . . , xt ; f, g) =

r X s X

fi (x1 , . . . , xt )gj (x1 , . . . , xt )mx (i, j)

i=1 j=1

be the probability of x after public history (x1 , . . . , xt ) has occurred. Furthermore, let P P βx (∅; f, g) = ri=1 sj=1 fi (∅)gj (∅)mx (i, j). When it is clear from the context what the strategy (f, g) is, we simplify the notation and write βx (x1 , . . . , xt ) instead of βx (x1 , . . . , xt ; f, g). 1

Throughout this paper ∆(Y ) will stand for the set of probability distributions over Y , when Y

is a finite set. 2 We depart from Blackwell (1956) in the assumption that X is finite since, in that paper, X is assumed to be a compact convex subset of RN .

3

A strategy (f, g) ∈ F induces, for each t ∈ N, a probability measure P(f,g),t on X t and a probability measure π(f,g),t on X as follows. For each x ∈ X = X 1 , P(f,g),1 (x) = π(f,g),1 (x) = βx (∅). Assuming that P(f,g),1 , π(f,g),1 , . . . , P(f,g),t−1 , π(f,g),t−1 have been defined, then P(f,g),t (x1 , . . . , xt ) = P(f,g),t−1 (x1 , . . . , xt−1 )βxt (x1 , . . . , xt−1 ) for all (x1 , . . . , xt ) ∈ X t and X

π(f,g),t (x) =

(x1 ,...,xt−1

P(f,g),t (x1 , . . . , xt−1 , x)

)∈X t−1

for each x ∈ X. Let d be a metric on RN and, for each x ∈ RN and S ⊆ RN , let d(x, S) = inf y∈S d(x, y). Our notions of approachability, excludability and securability with discounting are defined as follows. A subset S of RN is approachable with discounting in M if there exists f ∈ F1 such that for every ε > 0, c ∈ (0, 1) and δ 7→ t(δ) : (0, 1) → N with limδ→1 δ t(δ) = c, there exists δ ∗ ∈ (0, 1) such that, for every δ ≥ δ ∗ and g ∈ F2 ,     t(δ)   1 − δ X k−1   < ε. P(f,g),t(δ)  (x1 , . . . , xt(δ) ) ∈ X t(δ) : d  ≥ ε δ x , S k   1 − δ t(δ) k=1

A subset S of RN is excludable with discounting in M if there exist g ∈ F2 and η > 0 such that for every ε > 0, c ∈ (0, 1) and δ 7→ t(δ) : (0, 1) → N with limδ→1 δ t(δ) = c, there exists δ ∗ ∈ (0, 1) such that, for every δ ≥ δ ∗ and f ∈ F1 ,     t(δ)   X 1−δ k−1 t(δ)    P(f,g),t(δ) δ xk , S ≥ η  > 1 − ε. (x1 , . . . , xt(δ) ) ∈ X :d   1 − δ t(δ) k=1

Furthermore, we say that S is securable with discounting in M if for every ε > 0, c ∈ (0, 1) and δ 7→ t(δ) : (0, 1) → N with limδ→1 δ t(δ) = c, there exists δ ∗ ∈ (0, 1) such that, for every δ ≥ δ ∗ and g ∈ F2 ,     t(δ)   1 − δ X k−1   < ε. P(f,g),t(δ)  (x1 , . . . , xt(δ) ) ∈ X t(δ) : d  δ x , S ≥ ε k   1 − δ t(δ) k=1

4

For each i ∈ A1 and j ∈ A2 , let m(i, ¯ j) =

P

x∈X

xmx (i, j) be the expected value

of x with respect to m(i, j). For each p ∈ P , let ( r ) r X X R(p) = co pi m(i, ¯ 1), . . . , pi m(i, ¯ s) , i=1

i=1

( s X

s X

and, for each q ∈ Q, T (q) = co

qj m(1, ¯ j), . . . ,

j=1

) qj m(r, ¯ j) .

j=1

Theorem 1 provides a sufficient condition for a closed and convex subset S of RN to be approachable and, also, a sufficient condition for it to be excludable. Moreover, it gives a characterization of those sets that are securable. Theorem 1 Let S ⊆ RN be closed and convex. Then: 1. If R(p) ⊆ S for some p ∈ P , then S is approachable with discounting with f ≡ p. 2. S is securable with discounting if and only if S ∩ T (q) 6= ∅ for each q ∈ Q. 3. If S ∩ T (q) = ∅ for some q ∈ Q, then S is excludable with discounting with g ≡ q. The above result is analogous to Theorem 3 and Corollary 1 in Blackwell (1956). As in the latter, R(p), and therefore any superset of it, is approachable for each p ∈ P using the constant strategy f ≡ p. Similarly, as in the former, any closed and convex S ⊆ RN \ T (q) for some q ∈ Q is excludable. Furthermore, the condition characterizing approachability in Theorem 1 of Blackwell (1956) now characterizes securability.3 The first step in the proof of Theorem 1 consists in showing that the discounted sum of the realization of signals is closed to its expected discounted sum with a 3

It then follows that approachability and securability coincide in the no-discounting setting of

Blackwell (1956). However, we don’t know whether an analogous result holds in the discounting case.

5

probability close to one. In other words, the distribution of the discounted sum of signals is concentrated around its expected value. This is shown in Lemma 1 below using a concentration result from probability theory. Lemma 1 For any (f, g) ∈ F , ε > 0, δ ∈ (0, 1) and t ∈ N, ! t t 1−δ X 2 X X 1 − δ − 22ε k−1 k−1 B b(δ,t) P(f,g),t βx (x1 , . . . , xk−1 )x ≥ ε ≤ |X|e , δ xk − δ 1 − δt 1 − δ t k=1 x∈X k=1 where βx (x1 , . . . , xk−1 ) = βx (x1 , . . . , xk−1 ; f, g), B = |X| max ||x|| and, x∈X

1 − δ 1 + δt . b(δ, t) = 1 + δ 1 − δt Proof. Let (f, g) ∈ F , ε > 0, δ ∈ (0, 1) and t ∈ N be given. For each x ∈ X, define Fx (x1 , . . . , xt ) =

t 1 − δ X k−1 δ (1x (xk ) − βx (x1 , . . . , xk−1 )) 1 − δ t k=1

for each (x1 , . . . , xt ) ∈ X t . We first argue that it suffices to show that  2 ε − 2ε P(f,g),t |Fx (x1 , . . . , xt )| ≥ ≤ e B2 b(δ,t) for each x ∈ X. B

(1)

Indeed, we have that t t 1−δ X X X 1−δ k−1 k−1 δ xk − δ βx (x1 , . . . , xk−1 )x = t t 1 − δ 1 − δ k=1 x∈X k=1 t t 1−δ X X X X 1 − δ k−1 k−1 δ x1x (xk ) − δ βx (x1 , . . . , xk−1 )x = t t 1 − δ 1 − δ k=1 x∈X x∈X k=1 ! t t X 1 − δ X k−1 1 − δ X k−1 x δ 1 (x ) − δ β (x , . . . , x ) = x k x 1 k−1 t t 1 − δ 1 − δ k=1 k=1 x∈X X X xFx (x1 , . . . , xt ) ≤ ||x|| |Fx (x1 , . . . , xt )| . x∈X

x∈X

Hence, (

) t t 1−δ X X X 1 − δ (x1 , . . . , xt ) ∈ X t : δ k−1 xk − δ k−1 βx (x1 , . . . , xk−1 )x ≥ ε t 1 − δt 1 − δ x∈X k=1 k=1 o [n ε ⊆ (x1 , . . . , xt ) ∈ X t : |Fx (x1 , . . . , xt )| ≥ B x∈X 6

and, therefore, if (1) holds, then ! t t 1−δ X 2 X X 1 − δ − 22ε k−1 k−1 B b(δ,t) . P(f,g),t β (x , . . . , x )x ≥ ε ≤ |X|e δ x − δ x 1 k−1 k 1 − δt 1 − δ t k=1 x∈X k=1 By the above, in the remaining of this proof, we establish (1). Fix x ∈ X. For convenience, for each 1 ≤ k ≤ t, we write Pk (resp. πk ) instead of P(f,g),k (resp. π(f,g),k ). First, note that  t X 1−δ E(Fx ) = δ k−1 πk (x) − 1 − δ t k=1 =

1−δ 1 − δt

t X

 X (x1 ,...,xk−1

Pk (x1 , . . . , xk−1 )βx (x1 , . . . , xk−1 )

)∈X k−1

(πk (x) − πk (x)) = 0.

k=1

Next, fix k ∈ {1, . . . , t} and xˆ1 , . . . , xˆk−1 ∈ X k−1 . Let Bk = {(x1 , . . . , xk−1 ) ∈ X k−1 : xi = xˆi for all i = 1, . . . , k − 1} and, for each x0 ∈ X, gk (x0 ) = E(Fx (x1 , . . . , xt )|Bk , xk = x0 ) − E(Fx (x1 , . . . , xt )|Bk ). Furthermore, let ran(ˆ x1 , . . . , xˆk−1 ) = sup{|gk (x0 ) − gk (¯ x)| : x0 , x¯ ∈ X}. We have that, for each x0 ∈ X and l ∈ {k + 1, . . . , t}, E(1x (xl ) − βx (x1 , . . . , xl−1 )|Bk , xk = x0 ) = 0

(2)

since E(1x (xl )|Bk , xk = x0 ) = l−1 Y X βxn (ˆ x1 , . . . , xˆk−1 , x0 , . . . , xn−1 )βx (ˆ x1 , . . . , xˆk−1 , x0 , xk+1 , . . . , xl−1 ) = (xk+1 ,...,xl−1 ) n=k+1

E(βx (x1 , . . . , xl−1 )|Bk , xk = x0 ). Hence, it follows from (2) that 1 − δ k−1 0 |gk (x ) − gk (¯ x)| = δ (1 (x ) − 1 (¯ x )) x x 1 − δt 0

and, hence, ran(ˆ x1 , . . . , xˆk−1 ) = 7

1 − δ k−1 δ . 1 − δt

For each (ˆ x1 , . . . , xˆt ) ∈ X t , let R2 (ˆ x1 , . . . , xˆt ) = rˆ2 =

x1 , . . . , xˆk−1 ))2 k=1 (ran(ˆ

and

R2 (ˆ x1 , . . . , xˆt ).

sup (ˆ x1 ,...,ˆ xt

Pt

)∈X t

Thus, we obtain that 

2

R (ˆ x1 , . . . , xˆt ) =

1−δ 1 − δt

2 X t

δ 2(k−1)

k=1

and, hence, 2

rˆ =



1−δ 1 − δt

2 X t

δ 2(k−1) =

k=1

(1 − δ)2 1 − δ 2t 1 − δ 1 + δt = = b(δ, t). (1 − δ t )2 1 − δ 2 1 + δ 1 − δt

It follows by Theorem 3.7 in McDiarmid (1998) that P(f,g),t



2 ε − 2ε ≤ e B2 b(δ,t) , |Fx (x1 , . . . , xt )| ≥ B

as desired. This completes the proof. It follows by Lemma 1 that, for the purpose of establishing Theorem 1, we can foP Pt 1−δ k−1 cus on the expected discounted sum of signals 1−δ t x∈X βx (x1 , . . . , xk−1 )x k=1 δ P t 1−δ k−1 instead of the discounted sum of the realized signals 1−δ xk . That this is t k=1 δ convenient is shown in the following three lemmas. For instance, when S ⊇ R(p) for some p ∈ P and f ≡ p as in part 1 of Theorem 1, we have that t 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 ; f, g)x ∈ S 1 − δ t k=1 x∈X

for each (x1 , . . . , xt ) ∈ X t , δ ∈ (0, 1) and g ∈ F2 and therefore ! ! t t t 1 − δ X k−1 1 − δ X k−1 1 − δ X k−1 X d δ xk , S ≤ d δ xk , δ βx (x1 , . . . , xk−1 )x . 1 − δ t k=1 1 − δ t k=1 1 − δ t k=1 x∈X Lemma 2 If f ≡ p, then t 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ R(p) 1 − δ t k=1 x∈X

for all t ∈ N, (x1 , . . . , xt ) ∈ X t , δ ∈ (0, 1) and g ∈ F2 .

8

Proof. Let t ∈ N, (x1 , . . . , xt ) ∈ X t , δ ∈ (0, 1) and g ∈ F2 be given. The definition of f implies that, for each 1 ≤ k ≤ t, X

βx (x1 , . . . , xk−1 )x =

s X

gj (x1 , . . . , xk−1 )

j=1

x∈X

Hence, as R(p) is convex and

Pt

1−δ 1−δ t

r X

pi m(i, ¯ j) ∈ R(p).

i=1

k=1

δ k−1 = 1,

t 1 − δ X k−1 X βx (x1 , . . . , xk−1 )x ∈ R(p). δ 1 − δ t k=1 x∈X

This concludes the proof. Lemma 3 Let S ⊆ RN be convex and such that S ∩ T (q) 6= ∅ for each q ∈ Q. Then, for each δ ∈ (0, 1) and g ∈ F2 , there exists f ∈ F1 such that t 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ S 1 − δ t k=1 x∈X

for all t ∈ N and (x1 , . . . , xt ) ∈ X t . Proof. Let δ ∈ (0, 1) and g ∈ F2 be given. For each t ∈ N and (x1 , . . . , xt ) ∈ X t , there exists z ∈ S ∩ T (g(x1 , . . . , xt )). Hence, for some p ∈ P , z=

r X

pi

i=1

s X

gj (x1 , . . . , xt )m(i, ¯ j).

j=1

Define f (x1 , . . . , xt ) = p. Fix (x1 , . . . , xt ) ∈ X t . The definition of f implies that, for each 1 ≤ k ≤ t, X

βx (x1 , . . . , xk−1 )x =

r X

fi (x1 , . . . , xk−1 )

i=1

x∈X

Hence, as S is convex and

1−δ 1−δ t

s X

gj (x1 , . . . , xk−1 )m(i, ¯ j) ∈ S.

j=1

Pt

k=1

δ k−1 = 1,

t 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ S. 1 − δ t k=1 x∈X

This concludes the proof.

9

Lemma 4 If g ≡ q, then t 1 − δ X k−1 X βx (x1 , . . . , xk−1 )x ∈ T (q) δ 1 − δ t k=1 x∈X

for all t ∈ N, (x1 , . . . , xt ) ∈ X t , δ ∈ (0, 1) and f ∈ F1 . Proof. Let t ∈ N, (x1 , . . . , xt ) ∈ X t , δ ∈ (0, 1) and f ∈ F1 be given. The definition of g implies that, for each 1 ≤ k ≤ t, X

βx (x1 , . . . , xk−1 )x =

r X

fi (x1 , . . . , xk−1 )

i=1

x∈X

Hence, as T (q) is convex and

1−δ 1−δ t

s X

qj m(i, ¯ j) ∈ T (q).

j=1

Pt

k=1

δ k−1 = 1,

t 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ T (q). 1 − δ t k=1 x∈X

This concludes the proof. We turn next to the proof of Theorem 1. Proof of Theorem 1. Let ε > 0, c ∈ (0, 1) and δ 7→ t(δ) : (0, 1) → N such that limδ→1 δ t(δ) = c. Let η > 0 be such that b(δ, t(δ)) < η implies that |X|e

−

2ε2 B 2 b(δ,t(δ))

< ε.

Since limδ→1 b(δ, t(δ)) = 0, then there exists δ ∗ ∈ (0, 1) such that δ ≥ δ ∗ implies b(δ, t(δ)) < η and, hence, −

|X|e

2ε2 B 2 b(δ,t(δ))

< ε.

We now establish part 1. Let p ∈ P be such that R(p) ⊆ S and f ≡ p. Fix δ ≥ δ ∗ and g ∈ F2 . Lemma 2 implies that t(δ) 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ R(p) ⊆ S 1 − δ t(δ) k=1 x∈X

for all (x1 , . . . , xt(δ) ) ∈ X t(δ) and, hence, Lemma 1 implies that     t(δ)   1 − δ X k−1   < ε, P(f,g),t(δ)  d  δ x , S ≥ ε k   1 − δ t(δ) k=1 10

as desired. Sufficiency in part 2 follows in a similar way. Fix δ ≥ δ ∗ and g ∈ F2 . By Lemma 3, there exists f ∈ F1 such that t(δ) 1 − δ X k−1 X βx (x1 , . . . , xk−1 )x ∈ S δ 1 − δ t(δ) k=1 x∈X

for all (x1 , . . . , xt(δ) ) ∈ X t(δ) . Hence, Lemma 1 implies that     t(δ)   X 1−δ k−1    P(f,g),t(δ) d δ xk , S ≥ ε  < ε,   1 − δ t(δ) k=1

as desired. We next prove part 3. Let q ∈ Q be such that S ∩ T (q) = ∅ and let g ≡ p. Fix δ ≥ δ ∗ and f ∈ F1 . Lemma 4 implies that t(δ) 1 − δ X k−1 X δ βx (x1 , . . . , xk−1 )x ∈ T (q) 1 − δ t(δ) k=1 x∈X

for all (x1 , . . . , xt(δ) ) ∈ X t(δ) . Since S ∩T (q) = ∅, then d(z, S) > 0 for all z ∈ T (q) and, since T (q) is compact, then minz∈T (q) d(z, S) > 0. Let η = min{ε, minz∈T (q) d(z, S)/2} > 0. By Lemma 1,   t(δ) t(δ) X X X 1−δ 1−δ P(f,g),t(δ)  δ k−1 xk − δ k−1 βx (x1 , . . . , xk−1 )x < η  ≥ t(δ) t(δ) 1−δ 1 − δ x∈X k=1 k=1 −

1 − |X|e

2η 2 B 2 b(δ,t(δ))

−

≥ 1 − |X|e

2ε2 B 2 b(δ,t(δ))

> 1 − ε.

Since   t(δ) t(δ)   1 − δ X 1 − δ X k−1 X k−1 δ xk − δ βx (x1 , . . . , xk−1 )x < η   1 − δ t(δ) 1 − δ t(δ) k=1 x∈X k=1     t(δ)   X 1−δ k−1  ⊆ d δ x , S ≥ η , k   1 − δ t(δ) k=1 then

    t(δ)   1 − δ X k−1   > 1 − ε, P(f,g),t(δ)  d  δ x , S ≥ η k   1 − δ t(δ) k=1

11

as desired. Finally, we prove the necessity part of part 2. Indeed, if S ∩ T (q) = ∅ for some q ∈ Q, then S is excludable and, hence, S is not securable. Since the condition characterizing securability is the same as in Blackwell (1956), we obtain the following corollary from Corollary 2 in Blackwell (1956). Corollary 1 A convex set S ⊆ RN is securable if and only if, for every u ∈ RN , v(u) ≥ min u · s, s∈S

where v(u) is the value of the game G = (A1 , A2 , π1 , −π1 ) with π1 (i, j) = u · m(i, ¯ j) for all (i, j) ∈ A1 × A2 .

3

Folk Theorem with Perfect Monitoring and Finite Automata

A normal form game G is defined by G = (Ai , ui )i∈N , where N = {1, . . . , n} is a Q finite set of players, Ai is the set of player i’s actions and ui : j∈N Aj → R is player i’s payoff function. We assume that Ai is finite for all i ∈ N . Q Q Let A = i∈N Ai and A−i = j6=i Ai . We shall denote the maximum payoff in absolute value some player can obtain by B = maxi∈N maxa∈A |ui (a)|. The set of Q mixed action of player i ∈ N is denoted by ∆i . As above, we let ∆ = i∈N ∆i and Q ∆−i = j6=i ∆i . For each i ∈ N , the mixed extension of player i’s payoff function is also denoted by ui . For any i ∈ N denote, respectively, the minmax payoff and a minmax profile for player i by v˜i = minσ−i ∈∆−i maxai ∈Ai ui (ai , σ−i ) and µi ∈ ∆, where µi−i ∈ arg minσ−i ∈∆−i maxai ∈Ai ui (ai , a−i ) and µii ∈ arg maxai ∈Ai ui (ai , µi−i ). Let U˜ = {u ∈ co (u(A)) : ui ≥ v˜i for all i ∈ N } denote the set of mixed individually rational payoffs and U˜ 0 = {u ∈ co(u(A) : ui > v˜i for all i ∈ N }. The game G is full-dimensional if the interior of U˜ in Rn is nonempty.

12

Theorem 2 Let G be a n-player game and suppose that G is full-dimensional. Then, for all ε > 0, there exist δ ∗ ∈ (0, 1) such that, for all u ∈ U˜ and δ ≥ δ ∗ , there exists a finite automata SPE f ∈ F of G∞ (δ) such that kU (f, δ) − uk < ε. Proof. For all x ∈ Rn , let ||x|| = maxi=1,...,n |xi |. Since U˜ is compact, it suffices to ˜ there exists δ ∗ ∈ (0, 1) such that for all δ ≥ δ ∗ , show that for all ε > 0 and all u ∈ U, there exists a finite automata SPE f of G∞ (δ) with kU (f, δ) − uk < ε. Furthermore, since U˜ equals the closure of U˜ 0 , we only need to show that the above holds for any U˜ 0 . Therefore, we show that for all ε > 0 and u ∈ U˜ 0 , there exists δ ∗ ∈ (0, 1) such that for all δ ≥ δ ∗ , there exists a finite automata SPE f of G∞ (δ) with kU (f, δ) − uk < ε. For convenience, we normalize payoffs so that νi = 0 for all i ∈ N . Fix any ε > 0 and any u ∈ U˜ 0 . Since G is full-dimensional, we may assume that u ∈ int(U˜ 0 ). Let u0 ∈ int(U˜ 0 ) such that u0 < u, and ρ > 0 be such that (i) u0i + ρ < ui for all i ∈ N and (ii) ||ˆ u − u0 ||∞ ≤ ρ implies uˆ ∈ U˜ 0 . We make the following construction for each δ ∈ (0, 1), t ∈ N and η > 0, and then show below that these parameters can be chosen to have certain desirable properties. ˆ = (ˆ For each i, d ∈ N with i 6= d, a ∈ A and h a1 , . . . , a ˆt ) ∈ Ht , let ˆ = n(a, h)

t X

δ k−1 1a (ˆ ak ),

k=1

ˆ = n(a−i , h)

X

ˆ n((bi , a−i ), h),

bi ∈Ai

X ˆ = 1−δ ˆ − n(a−i , h)µ ˆ d (ai )|, |n(a, h) Φi (d, h) i t 1 − δ a∈A and

  1 if Φ (d, h) ˆ < η, i ˆ η) = αi (d, h,  0 otherwise.

Lemma 5 For every 0 < ε1 < 1, there exists η > 0 such that, for every d ∈ N , ˆ = (ˆ ˆ η) = 1 for all i 6= d, δ ∈ (0, 1), t ∈ N and h a1 , . . . , a ˆt ) ∈ Ht such that αi (d, h, t 1 − δ X k−1 δ ud (ˆ ak ) < ε1 . 1 − δ t k=1

13

Proof. Let 0 < ε1 < 1. Moreover, let η > 0 be such that (n − 1)B|A|2 η < ε1 . ˆ = (ˆ ˆ η) = Consider d ∈ N , δ ∈ (0, 1), t ∈ N and h a1 , . . . , a ˆt ) ∈ Ht such that αi (d, h, 1 for all i 6= d. First, we reorder the players such that the player to be punished is ˆ called player n, i.e. d = n. Second, we write, for each a ∈ A, n(a) instead of n(a, h). Then, for every a ∈ A and every i 6= n: |n(a) − n(a−i )µni (ai )| < η

1 − δt 1−δ

Fix a = (a1 , a2 , . . . , an ) ∈ A. In particular, we have that X

|n(a) −

n(b1 , a2 , . . . , an )µn1 (a1 )| < η

b1 ∈A1

1 − δt 1−δ

Since, for every b1 ∈ A1 , X

|n(b1 , a−1 ) −

n(b1 , b2 , a3 , . . . , an )µn2 (a2 )| < η

b2 ∈A2

1 − δt , 1−δ

we obtain: X

|n(a) −

n(b1 , b2 , a3 , . . . , an )µn1 (a1 )µn2 (a2 )| < 2|A|η

(b1 ,b2 )∈A1 ×A2

1 − δt . 1−δ

Repeating the same procedure n − 1 times implies that n−1 X Y 1 − δt n n(a) − n(b , b , . . . , b , a ) µ (a ) < (n − 1)|A|η . 1 2 n−1 n j j 1−δ j=1 (b1 ,...,bn−1 )∈A−n Hence, 1−δ |n(a) − 1 − δt

X

n(b1 , b2 , . . . , bn−1 , an )

n−1 Y

µnj (aj )| < (n − 1)|A|η.

(3)

j=1

(b1 ,...,bn−1 )∈A−n

As a ∈ A is arbitrary, it follows that (3) holds for all a ∈ A. Define, for each an ∈ An , rn (an ) =

1−δ 1 − δt

We then have that rn ∈ ∆(An ) since

X

n(b−n , an ).

b−n ∈A−n

P

an

rn (an ) = 1. It follows from the definition

of rn and (3) that, for all a ∈ A, n−1 Y 1−δ n(a) < r (a ) µnj (aj ) + (n − 1)|A|η. n n 1 − δt j=1

14

Hence, 1−δ X n(a)un (a) < un (rn , µn−n ) + B(n − 1)|A|2 η < ε1 , 1 − δ t a∈A and, therefore, t 1 − δ X k−1 1−δ X k n(a)un (a) < ε1 . δ u (ˆ a ) = n 1 − δ t k=1 1 − δ t a∈A

This concludes the proof. Fix i, d ∈ N with i 6= d and let µ ˜di be player i’s strategy consisting of playing µdi each period independently of the history. Given a strategy σ−i for the remaining players and t ∈ N, let P(˜µdi ,σ−i ),t be the probability measure on Ht induced by (˜ µdi , σ−i ). Given c ∈ (0, 1), for each δ ∈ (0, 1), let t(c, δ) = 1 if δ ≤ c and, if δ > c, let t(c, δ) be the highest integer t ∈ N such that δ t ≥ c. Hence, |δ t(c,δ) − c| < (1 − δ)/δ whenever δ > c and, therefore, limδ→1 δ t(c,δ) = c. Lemma 6 For all c ∈ (0, 1), η > 0 and ε2 > 0, there exists δ¯ ∈ (0, 1) such that, for ¯ all δ ≥ δ,   ˆ ∈ Ht(c,δ) : Φi (d, h) ˆ ≥ η} < ε2 P(˜µdi ,σ−i ),t(c,δ) {h for all i, d ∈ N with i 6= d and σ−i ∈ F−i . Proof. This result will be a consequence of Theorem 1. Let c ∈ (0, 1), η > 0 and ε2 > 0 be given. Note first that it is enough to show that, for each i, d ∈ N with i 6= d, there exists δ¯i,d ∈ (0, 1) such that, for all δ ≥ δ¯i,d ,   ˆ ∈ Ht(c,δ) : Φi (d, h) ˆ ≥ η} < ε2 P(˜µdi ,σ−i ),t(c,δ) {h for all σ−i ∈ F−i . Indeed, the conclusion of the lemma will follow by letting δ¯ = maxi,d:i6=d δ¯i,d . Fix i, d ∈ N such that i 6= d. We embed A in R|A| by, first, letting θ : A → {1, . . . , |A|} be 1-1 and onto and, second, letting x(a) ∈ R|A| be such that   1 if l = θ(a), xl (a) =  0 otherwise. 15

Let X = {x(a) : a ∈ A} and m(a) = 1x(a) for all a ∈ A. Consider a 2-player game where player I’s action set is Ai and player II’s action set is A−i , and hence P = ∆(Ai ) and Q = ∆(A−i ). Let p ∈ P be defined by pai = µdi (ai ) for all ai ∈ Ai and let Di : co(X) → R be defined by X X Di (z) = z − p z a ai (bi ,a−i ) bi ∈Ai

a∈A

for each z ∈ co(X). Since Di is continuous and co(X) is compact, there exists 0 < ε < ε2 such that ||z − z 0 || < ε and z, z 0 ∈ co(X) imply |Di (z) − Di (z 0 )| < η. We have that R(p) = {z ∈ co(X) : Di (z) = 0}. Indeed, if z ∈ R(p), then P P z = a−i jqa−i ai p∗ai x(a) for some q ∈ Q. Thus, for each a ∈ A, za = pai qa−i and, P P P therefore, Di (z) = a∈A pai qa−i − bi ∈Ai pai pbi qa−i = a∈A pai qa−i − pai qa−i = 0. Conversely, let z ∈ co(X) be such that Di (z) = 0. Since z ∈ co(X), then z = P P a λa = 1 and, since Di (z) = 0, then a λa x(a) for some {λa }a∈A ⊂ R+ with P P za = bi ∈Ai p∗ai z(bi ,a−i ) . Define qa−i = bi ∈Ai λ(bi ,a−i ) for all a−i ∈ A−i and note that q ∈ Q. Since za = λa for all a ∈ A given the definition of x(a), then, it follows from P za = p∗ai bi ∈Ai z(bi ,a−i ) that λa = pai qa−i for all a ∈ A. Thus, z ∈ R(p). Hence, by Theorem 1, R(p) is approachable with discounting with f ≡ p. Hence, there exists δ¯i,d ∈ (0, 1) such that, for every δ ≥ δ¯i,d and g,     t(c,δ)   1 − δ X k−1   < ε. P(f,g),t(c,δ)  d  δ x , R(p) ≥ ε k   1 − δ t(c,δ) k=1 Thus,     t(c,δ)   1 − δ X k−1    δ xk ≥ η  < ε < ε 2 . P(f,g),t(c,δ) Di   1 − δ t(c,δ) k=1

Let σ−i ∈ F−i be given. Define g ∈ G as follows: for all t ∈ N and (x1 , . . . , xt ) ∈ X t , let, for each 1 ≤ k ≤ t, ak ∈ A be such that xk = x(ak ) and set g(x1 , . . . , xt ) = σ−i (a1 , . . . , at ). Hence, we have that P(˜µdi ,σ−i ),t(c,δ) (ˆ a1 , . . . , a ˆt(c,δ) ) = P(f,g),t(c,δ) (x(ˆ a1 ), . . . , x(ˆ at(c,δ) )). ˆ = (ˆ Furthermore, note that, for each h a1 , . . . , a ˆt(c,δ) ) ∈ Ht(c,δ) ,   t(c,δ) X ˆ = Di  1 − δ Φi (d, h) δ k−1 x(ˆ ak ) . t(c,δ) 1−δ k=1 16

Hence, it follows that   ˆ ∈ Ht(c,δ) : Φi (d, h) ˆ ≥ η} < ε2 , P(˜µdi ,σ−i ),t(c,δ) {h as desired. Let ε1 > 0 such that ε1 < min u0d .

(4)

d∈N

Let η > 0 be as in Lemma 5, corresponding to ε1 just defined. Let ε2 > 0 be such that (1 − 2ε2 )n ε1 + (1 − (1 − 2ε2 )n ) B < min u0d . d∈N

(5)

Define ε¯ = (1 − 2ε2 )n ε1 + (1 − (1 − 2ε2 )n ) B. Let c ∈ (0, 1) be such that cρε2 > (1 − c)2B

(6)

Let δ¯ be as in Lemma 6, corresponding to ε2 , η and c just defined. Define ξ > 0 to be such that 2ξ < ε,

(7)

(1 − c)(min u0d − ε¯) > (1 + c)2ξ,

(8)

c(ρε2 − 4ξ) > (1 − c)2B,

(9)

d

Such ξ > 0 exists due to ε > 0, (5) and (6), respectively. For all i = 1, . . . , n and β ∈ Rn , let ui (β) be defined by uii (β) = u0i and uij (β) = u0j + βj ρ. Furthermore, define Wi = {ui (β) : βj ∈ {0, 1} for all j ∈ N \ {i}}. ˆ = ∪ni=1 Wi . By our choice of ρ (specifically, by (ii) above), then Wi ⊆ U˜ 0 . Define W ˆ is finite, order W ˆ = {u1 , . . . , uω¯ }, where ω ˆ |. For notational conveSince W ¯ = |W ˆ ∪ {u0 }. nience, let u0 = u and W = W 17

P For all k ∈ N, let Vk be the set of u0 ∈ co(u(A)) such that u0 = a∈A pa u(a)/k P for some {pa }a∈A satisfying pa ∈ N and a∈A pa = k. Using an analogous argument to Sorin (1992, Proposition 1.3), it follows that Vk converges to co(u(A)). Therefore, let K ∈ N such that co(u(A)) ⊆ ∪x∈VK Bξ (x).

(10)

For all ω ∈ {0, . . . , ω ¯ }, let xω ∈ VK be such that ||xω − uω || < ξ and {pωa }a∈A be such that

1 K

PK

a∈A

(11)

pωa uj (a) = xωj , for all j ∈ N . For all ω ∈ {0, . . . , ω ¯ },

define π ˆ (ω) as the repetition of the cycle
(a1 ; pωa1 ), . . . , (ar ; pωar ) . In the construction below, π ˆ (ω) will be the equilibrium path when ω = 0 (also sometimes denoted by π (0) ) and a “reward path” when ω > 0. ¯ 1) be such that for all δ ≥ δ ∗ , letting t(c, δ) be as in Lemma 6, Let δ ∗ ∈ [δ,

K K 1−δ X X 1 k−1 k k δ x − x sup < ξ, K K x∈[−B,B]K 1 − δ k=1 k=1

(12)

δ t(c,δ) (ρε2 − 4ξ) > (1 − δ t(c,δ) )2B, and

(13)

u0d − 2ξ > (1 − δ)B + δ(1 − δ t(c,δ) )¯ ε + δ 1+t(c,δ) (u0d + 2ξ) for all d ∈ N. (14) Note that such δ ∗ ∈ (0, 1) exists because of (8) and (9), and because the limit of the left hand side of (12) as δ → 1 is 0. Fix any δ ≥ δ ∗ and set T = t(c, δ). We will now demonstrate the result by constructing a finite automata SPE f with ||U (f ) − u|| < ε.

3.1

Punishment play

Next we define the mixed actions to be played during the punishment phases. Let C : W → Rn be defined by setting C(uω ) = V (ˆ π (ω) ) for all 0 ≤ ω ≤ ω ¯ . We next define a function w : N × HT → C(W ) that determines the reward payoff after 18

ˆ = (ˆ a punishment phase. Let d ∈ N and h a1 , . . . , a ˆT ) ∈ HT . Set, for all j 6= d, ˆ = αj (d, h, ˆ η). βj (d, h) Also, set ˆ = C(u0 (β(d, h))). ˆ w(d, h) T −1 −1 Let σ ∗ : N × ∪t=0 Ht → ∆ and V ∗ : N × ∪Tt=0 Ht → Rn be such that the following

ˆ ∈ Ht and i ∈ N , then: property holds: For all d ∈ N , 0 ≤ t ≤ T − 1, h ˆ solves ˆ = T − 1, then σ ∗ (d, h) (a) If `(h) i ∗ ˆ +δ max [(1 − δ)ui (σi , σ−i (d, h))

σi ∈∆i

X

∗ ˆ a wi (d, h ˆ · a)] (σi , σ−i (d, h))

a∈A

and ˆ +δ ˆ = (1 − δ)ui (σ ∗ (d, h)) Vi∗ (d, h)

X

ˆ i (d, h ˆ · a). σa∗ (d, h)w

a∈A

ˆ < T − 1, then σ ∗ (d, h) ˆ solves (b) If `(h) i ∗ ˆ +δ max [(1 − δ)ui (σi , σ−i (d, h))

σi ∈∆i

X ∗ ˆ a V ∗ (d, h ˆ · a)] (σi , σ−i (d, h)) i a∈A

and ˆ = (1 − δ)ui (σ ∗ (d, h)) ˆ +δ Vi∗ (d, h)

X

ˆ ∗ (d, h ˆ · a). σa∗ (d, h)V i

a∈A ∗

The existence of σ and V

∗

can be established using, for each fixed d ∈ N ,

backwards induction and Nash’s existence theorem. For notational convenience, for all d ∈ N , let σ d = σ ∗ (d, ·). Lemma 7 For all i, d ∈ N , with i 6= d,   ˆ ˆ Pσd ,T {h ∈ HT : Φi (d, h) ≥ η} < 2ε2 . Proof. Suppose not. Then, for some i, d ∈ N with i 6= d,   ˆ ∈ HT : Φi (d, h) ˆ ≥ η} ≥ 2ε2 . Pσd ,T {h Consider strategy µ ˜di for player i and let Vi (d, H0 ) be player i’s expected discounted payoff from (˜ µdi , σ d ), i.e. Vi (d, H0 ) = (1 − δ)

X

ˆ P(˜µdi ,σ−i d ),T (h)

ˆ a1 ,...,ˆ h=(ˆ aT )∈HT

T X k=1

19

! ˆ . δ t−1 ui (ˆ ak ) + δ T wi (d, h)

Given the definition of σ ∗ , we have that Vi∗ (d, H0 ) ≥ Vi (d, H0 ).

(15)

Furthermore, by Lemma 6, (11) and (12), Vi (d, H0 ) ≥ −B(1 − δ T ) + δ T (u0i − 2ξ) + δ T ρ(1 − ε2 ). By (11) and (12), Vi∗ (d, H0 ) ≤ B(1 − δ T ) + δ T (u0i + 2ξ) + δ T ρ(1 − 2ε2 ). Hence, by (13), Vi∗ (d, H0 ) − Vi (d, H0 ) ≤ 2B(1 − δ T ) + 4ξδ T − ρε2 δ T < 0. But this contradicts (15).

3.2

The strategy profile

We next define the strategy profile f . The set of states is S = (W × {1, . . . , K}) ∪
T −1 N × ∪t=0 Ht , with initial state s0 = (u0 , 1). The transition function is τ : S×A → S defined as follows: First, let Di (a) = {(a0i , a−i ) ∈ A : a0i 6= ai } and D(a) = ∪i∈N Di (a). Then, for each (s, a) ∈ S × A,    (uω , (k + 1)modK)      (d, H ) 0 τ (s, a) =   (d, h · a)      (u0 (β(d, h)), 1)

if s = (uω , k) and a 6∈ D(ˆ π (ω),k ), if s ∈ W × K, and a ∈ Dd (ˆ π (ω),k ) for some d ∈ N, if s = (d, h), and h 6∈ HT −1 , if s = (d, h), and h ∈ HT −1 .

Q Finally, the behavior function is g : S → i∈N ∆(Ai ) defined by   π ˆ (ω),k if s = (uω , k), g(s) =  σ d (h) if s = (d, h). Let f be the strategy defined by the automaton (S, s0 , τ, g). We have that U (f, δ) = V (π (0) ) and, by (11), (12) and (7), ||U (f, δ)−u|| < 2ξ < ε. 20

To complete the proof of the theorem, we next establish the following for all s ∈ S: Vd (s) ≥ (1 − δ)ud (ad , g−d (s)) + δVd (sτ (s, (ad , g−d (s)))) for all d ∈ N and ad ∈ Ad . (16) −1 By construction, (16) holds for each s ∈ N × ∪Tt=0 Ht . Consider then the case

s ∈ W × {1, . . . , K}. In this case, the left-hand side of (16) is, by (11) and (12), greater or equal to u0d − 2ξ. By Lemmas 5 and 7, the right-hand side of (16) is less than or equal to (1 − δ)B + δ(1 − δ T )¯ ε + δ 1+T (u0d + 2ξ). Thus, by (14), (16) holds.

References Barlo, M., G. Carmona, and H. Sabourian (2015): “Bounded Memory Folk Theorem,” Sabancı University, University of Surrey and University of Cambridge. Blackwell, D. (1956): “An Analog of the Minmax Theorem for Vector Payoffs,” Pacific Journal of Mathematics, 6, 1–8. Gossner, O. (1995): “The Folk Theorem for Finitely Repeated Games with Mixed Strategies,” International Journal of Game Theory, 24, 95–107. McDiarmid, C. (1998): “Concentration,” in Probabilistic Methods for Algorithmic Discrete Mathematics, ed. by M. Habib, C. McDiarmid, J. Ramirez, and B. Reed. Springer-Verlag, Berlin. Sorin, S. (1992): “Repeated Games with Complete Information,” in Handbook of Game Theory, Volume 1, ed. by R. Aumann, and S. Hart. Elsevier Science Publishers. Zamir, S. (1992): “Repeated Games of Incomplete Information: Zero-Sum,” in Handbook of Game Theory, Volume 1, ed. by R. Aumann, and S. Hart. Elsevier Science Publishers.

21