Novel Lower Bounds on the Entropy Rate of Binary Hidden Markov - - PowerPoint PPT Presentation

Novel Lower Bounds on the Entropy Rate of Binary Hidden Markov Processes

Or Ordentlich MIT ISIT, Barcelona, July 11, 2016

1 / 13

Binary Markov Processes

1 q01 q10 1 − q10 1 − q01 P = 1 − q01 q01 q10 1 − q10

• ,

πP = π = [π0 π1] X1 ∼ Bernoulli(π1), Pr(Xn = j|Xn−1 = i, Xn−2, . . . , X1) = Pij

2 / 13

Binary Markov Processes

1 q01 q10 1 − q10 1 − q01 P = 1 − q01 q01 q10 1 − q10

• ,

πP = π = [π0 π1] X1 ∼ Bernoulli(π1), Pr(Xn = j|Xn−1 = i, Xn−2, . . . , X1) = Pij

Entropy Rate

For a stationary process {Xn} the entropy rate is defined as ¯ H(X) lim

n→∞

H(X1, . . . , Xn) n = lim

n→∞ H(Xn|Xn − 1, . . . , X1)

2 / 13

Binary Markov Processes

1 q01 q10 1 − q10 1 − q01 P = 1 − q01 q01 q10 1 − q10

• ,

πP = π = [π0 π1] X1 ∼ Bernoulli(π1), Pr(Xn = j|Xn−1 = i, Xn−2, . . . , X1) = Pij

Entropy Rate

For the Markov process above ¯ H(X) = H(Xn|Xn−1) = π0h(q01) + π1h(q10) h(α) −α log2(α) − (1 − α) log2(1 − α)

2 / 13

Binary Hidden Markov Processes

{Xn} : 1 q01 q10 1 − q10 1 − q01

3 / 13

Binary Hidden Markov Processes

{Xn} : 1 q01 q10 1 − q10 1 − q01 {Yn}: PY |X Xn Yn

3 / 13

Binary Hidden Markov Processes

{Xn} : 1 q01 q10 1 − q10 1 − q01 {Yn}: Zn ∼ Bernoulli(α) Xn Yn = Xn ⊕ Zn

3 / 13

Binary Hidden Markov Processes

{Xn} : 1 q01 q10 1 − q10 1 − q01 {Yn}: Zn ∼ Bernoulli(α) Xn Yn = Xn ⊕ Zn

Entropy Rate Unknown

¯ H(Y ) = f (α, q10, q01) =???

3 / 13

Binary Hidden Markov Processes

{Xn} : 1 q01 q10 1 − q10 1 − q01 {Yn}: Zn ∼ Bernoulli(α) Xn Yn = Xn ⊕ Zn

Entropy Rate Unknown

¯ H(Y ) = f (α, q10, q01) =??? Our contribution: new lower bounds on ¯ H(Y )

3 / 13

Binary Symmetric Hidden Markov Processes

{Xn} : 1 q q 1 − q 1 − q {Yn}: Zn ∼ Bernoulli(α) Xn Yn = Xn ⊕ Zn

Entropy Rate Unknown

¯ H(Y ) = f (α, q) =???

4 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0)

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62]

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• a ∗ b = a(1 − b) + b(1 − a),

h−1 : [0, 1] → [0, 1/2]

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(X1, . . . , Xn) n

• Continuity of MGL function ϕ(u) = h
• α ∗ h−1(u)
• 5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h (α ∗ q) ¯ H(X) = h(q)

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h (α ∗ q) The same as Cover-Thomas bound of order n = 1

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h (α ∗ q) Standard MGL gives a weak estimate

5 / 13

Binary Symmetric HMP - Simple Bounds

“Cover-Thomas bounds”: H(Yn|Yn−1 . . . , Y1, X0) ≤ ¯ H(Y ) ≤ H(Yn|Yn−1 . . . , Y0) Accuracy improves exponentially with n [Birch’62] Simple lower bound by Mrs. Gerber’s Lemma: H(Y1, . . . , Yn) ≥ nh

• α ∗ h−1

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(X1, . . . , Xn) n

• ⇒ ¯

H(Y ) ≥ h (α ∗ q) Standard MGL gives a weak estimate We will use an improved version of MGL

5 / 13

Samorodnitsky’s MGL

X, Y ∈ {0, 1}n are the input and output of a BSC(α) λ (1 − 2α)2 The projection of X onto a subset of coordinates S ⊆ [n] is XS {Xi : i ∈ S} Let V be a random subset of [n] generated by independently sampling each element i with probability λ

Theorem [Samorodnitsky’15]

H(Y) ≥ nh

• α ∗ h−1

H(XV |V ) λn

• 6 / 13

Samorodnitsky’s MGL

X, Y ∈ {0, 1}n are the input and output of a BSC(α) λ (1 − 2α)2 The projection of X onto a subset of coordinates S ⊆ [n] is XS {Xi : i ∈ S} Let V be a random subset of [n] generated by independently sampling each element i with probability λ

Theorem [Samorodnitsky’15]

H(Y) ≥ nh

• α ∗ h−1

H(XV |V ) λn

• By Han’s inequality H(XV |V )

λn

is nonincreasing∗ in λ

6 / 13

Samorodnitsky’s MGL

X, Y ∈ {0, 1}n are the input and output of a BSC(α) λ (1 − 2α)2 The projection of X onto a subset of coordinates S ⊆ [n] is XS {Xi : i ∈ S} Let V be a random subset of [n] generated by independently sampling each element i with probability λ

Theorem [Samorodnitsky’15]

H(Y) ≥ nh

• α ∗ h−1

H(XV |V ) λn

• ⇒ The new bound is stronger than MGL

6 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• ϕ(x) h
• α ∗ h−1(x)
• 7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• Need to upper bound

I(Xi; Y i−1

1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• Need to upper bound

I(Xi; Y i−1

1

) = I(Xi; Y i−2

1

) + I(Xi; Yi−1|Y i−2

1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• Need to upper bound

I(Xi; Y i−1

1

) = I(Xi; Y i−2

1

) + I(Xi; Yi−1|Y i−2

1

) (SDPI) ≤ I(Xi; Y i−2

1

) + λI(Xi; Xi−1|Y i−2

1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• Need to upper bound

I(Xi; Y i−1

1

) = I(Xi; Y i−2

1

) + I(Xi; Yi−1|Y i−2

1

) (SDPI) ≤ I(Xi; Y i−2

1

) + λI(Xi; Xi−1|Y i−2

1

) = (1 − λ)I(Xi; Y i−2

1

) + λ

• I(Xi; Y i−2

1

) + I(Xi; Xi−1|Y i−2

1

)

• 7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• Need to upper bound

I(Xi; Y i−1

1

) = I(Xi; Y i−2

1

) + I(Xi; Yi−1|Y i−2

1

) (SDPI) ≤ I(Xi; Y i−2

1

) + λI(Xi; Xi−1|Y i−2

1

) = (1 − λ)I(Xi; Y i−2

1

) + λ

• I(Xi; Y i−2

1

) + I(Xi; Xi−1|Y i−2

1

)

• (Chain Rule) = (1 − λ)I(Xi; Y i−2

1

) + λI(Xi; Xi−1, Y i−2

1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• We have I(Xi; Y i−1

1

) ≤ (1 − λ)I(Xi; Y i−2

1

) + λI(Xi; Xi−1, Y i−2

1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• We have I(Xi; Y i−1

1

) ≤ (1 − λ)I(Xi; Y i−2

1

) + λI(Xi; Xi−1, Y i−2

1

) Using this to form and solve a suitable linear program [Samorodnitsky’15], or using induction [Polyanskiy-Wu’16], gives

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• We have I(Xi; Y i−1

1

) ≤ (1 − λ)I(Xi; Y i−2

1

) + λI(Xi; Xi−1, Y i−2

1

) Using this to form and solve a suitable linear program [Samorodnitsky’15], or using induction [Polyanskiy-Wu’16], gives I(Xi; Y i−1

1

) ≤ I(Xi; YBEC,(1−λ)

i−1 1

)

7 / 13

Samorodnitsky’s MGL - Proof Outline

H(Y) =

n

• i=1

H(Yi|Y i−1

1

) ≥

n

• i=1

ϕ(H(Xi|Y i−1

1

)) =

n

• i=1

ϕ

• H(Xi) − I(Xi; Y i−1

1

)

• We have I(Xi; Y i−1

1

) ≤ (1 − λ)I(Xi; Y i−2

1

) + λI(Xi; Xi−1, Y i−2

1

) Using this to form and solve a suitable linear program [Samorodnitsky’15], or using induction [Polyanskiy-Wu’16], gives I(Xi; Y i−1

1

) ≤ I(Xi; YBEC,(1−λ)

i−1 1

) From here, standard arguments give the theorem

7 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Theorem [Samorodnitsky’15]

¯ H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(XV |V ) λn

• 8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Theorem [Samorodnitsky’15]

¯ H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(XV |V ) λn

• Need to find limn→∞

H(XV |V ) λn

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ).

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ). Define: q∗k q ∗ q ∗ · · · ∗ q

• k times

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ). Define: q∗k q ∗ q ∗ · · · ∗ q

• k times

= 1 − (1 − 2q)k 2

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Proposition

lim

n→∞

H(XV |V ) λn = Eh

• q∗G

where G ∼ Geometric(λ). Define: q∗k q ∗ q ∗ · · · ∗ q

• k times

= 1 − (1 − 2q)k 2

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

Eh

• q∗G

, where G ∼ Geometric((1 − 2α)2).

8 / 13

Back to HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q q 1 − q 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

Eh

• q∗G

, where G ∼ Geometric((1 − 2α)2). For small α (high-SNR) bound approaches MGL For large α (low-SNR) much better than MGL

8 / 13

New Bound - Behavior with α

q = 0.11

α

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

¯ H(Y )

0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1

MGL Lower Bound from Theorem 2 Approximated Value 9 / 13

New Bound - Behavior with α

Theorem (low-SNR)

Let q be fixed and α = 1

2 − ǫ. Then

¯ H(Y ) ≥ 1 − 16ǫ4

∞

• k=1

log(e) 2k(2k − 1) (1 − 2q)2k 1 − (1 − 2q)2k + o(ǫ4)

9 / 13

New Bound - Behavior with α

Theorem (low-SNR)

Let q be fixed and α = 1

2 − ǫ. Then

¯ H(Y ) ≥ 1 − 16ǫ4

∞

• k=1

log(e) 2k(2k − 1) (1 − 2q)2k 1 − (1 − 2q)2k + o(ǫ4) Our result shows that lim sup

ǫ→0

1 − ¯ H(Y ) ǫ4 ≤ 16

∞

• k=1

log(e) 2k(2k − 1) (1 − 2q)2k 1 − (1 − 2q)2k

9 / 13

New Bound - Behavior with α

Theorem (low-SNR)

Let q be fixed and α = 1

2 − ǫ. Then

¯ H(Y ) ≥ 1 − 16ǫ4

∞

• k=1

log(e) 2k(2k − 1) (1 − 2q)2k 1 − (1 − 2q)2k + o(ǫ4) Our result shows that lim sup

ǫ→0

1 − ¯ H(Y ) ǫ4 ≤ 16

∞

• k=1

log(e) 2k(2k − 1) (1 − 2q)2k 1 − (1 − 2q)2k Best previously known bound [E. Ordentlich-Weissman’11]: lim sup

ǫ→0

1 − ¯ H(Y ) ǫ4 ≤ 2 log(e)(1 − 2q)2(1 − 4q + 16q2 − 32q3 + 32q4) q2

9 / 13

New Bound - Behavior with q

α = 0.11

q

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

¯ H(Y )

0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1

MGL Lower Bound from Theorem 2 Approximated Value 10 / 13

New Bound - Behavior with q

Theorem (fast transitions)

Let α be fixed and q = 1

2 − ǫ. Then

1 − ¯ H(Y ) ≤ 2 log(e)(1 − 2α)4ǫ2 + O(ǫ4)

10 / 13

New Bound - Behavior with q

Theorem (fast transitions)

Let α be fixed and q = 1

2 − ǫ. Then

1 − ¯ H(Y ) ≤ 2 log(e)(1 − 2α)4ǫ2 + O(ǫ4) Recovers the expression found in [E. Ordentlich-Weissman’11]

10 / 13

New Bound - Behavior with q

Theorem (fast transitions)

Let α be fixed and q = 1

2 − ǫ. Then

1 − ¯ H(Y ) ≤ 2 log(e)(1 − 2α)4ǫ2 + O(ǫ4) Recovers the expression found in [E. Ordentlich-Weissman’11] Bound is tight [E. Ordentlich-Weissman’11]

10 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Theorem [Samorodnitsky’15]

¯ H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(XV |V ) λn

• 11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Theorem [Samorodnitsky’15]

¯ H(Y ) ≥ h

• α ∗ h−1
• lim

n→∞

H(XV |V ) λn

• Need to find limn→∞

H(XV |V ) λn

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ).

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ). Define: P = 1 − q01 q01 q10 1 − q10

• ,

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Proposition

lim

n→∞

H(XV |V ) λn = EH(XG+1|X1) where G ∼ Geometric(λ). Define: P = 1 − q01 q01 q10 1 − q10

• ,

q#k

ij

• Pk

ij = Pr (Xn = j|Xn−k = i)

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Proposition

lim

n→∞

H(XV |V ) λn = π0Eh

• q#G

01

• + π1Eh
• q#G

10

• where G ∼ Geometric(λ).

Define: P = 1 − q01 q01 q10 1 − q10

• ,

q#k

ij

• Pk

ij = Pr (Xn = j|Xn−k = i)

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

• ,

where G ∼ Geometric((1 − 2α)2).

11 / 13

Nonsymmetric HMPs

{Xn} : Yn = Xn ⊕ Zn 1 q01 q10 1 − q10 1 − q01

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

• ,

where G ∼ Geometric((1 − 2α)2). For small α (high-SNR) bound approaches MGL For large α (low-SNR) much better than MGL

11 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

, where G ∼ Geometric((1 − 2α)2).

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

, where G ∼ Geometric((1 − 2α)2). P = 1 − q q 1

• ,

π0 = 1 1 + q , π1 = q 1 + q

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

, where G ∼ Geometric((1 − 2α)2). Pk = 1−(−q)k+1

1+q q+(−q)k+1 1+q 1−(−q)k 1+q q+(−q)k 1+q

• ,

π0 = 1 1 + q , π1 = q 1 + q

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1

π0Eh

• q#G

01

• + π1Eh
• q#G

10

, where G ∼ Geometric((1 − 2α)2). β 1 1 + qEh 1 − (−q)G+1 1 + q

• +

q 1 + q Eh 1 − (−q)G 1 + q

• 12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1(β)
• ,

where G ∼ Geometric((1 − 2α)2). β 1 1 + qEh 1 − (−q)G+1 1 + q

• +

q 1 + q Eh 1 − (−q)G 1 + q

• 12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1(β)
• ,

where G ∼ Geometric((1 − 2α)2). β = EH(XG+1|X1)

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1(β)
• ,

where G ∼ Geometric((1 − 2α)2).

Proposition

β ≥ h(π1) − (1 − 2α)2q (1 + q)(1 − 4α(1 − α)q)

• 2h (π1) − h

1 − q 1 + q

• 12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem

¯ H(Y ) ≥ h

• α ∗ h−1(β)
• ,

where G ∼ Geometric((1 − 2α)2).

Proposition

β ≥ h(π1) − cǫ2, for α = 1 2 − ǫ, c > 0

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ h 1 2 − ǫ

• ∗ h−1

h(π1) − cǫ2

Proposition

β ≥ h(π1) − cǫ2, for α = 1 2 − ǫ, c > 0

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ h 1 2 − ǫ

• ∗
• π1 −

c h′(π1)ǫ2

• 12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

Proposition: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Upper Bound

¯ H(Y ) ≤ H(Yn)

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

Proposition: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Upper Bound

¯ H(Y ) ≤ h(α ∗ π1)

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

Proposition: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Upper Bound

¯ H(Y ) ≤ h 1 2 − ǫ

• ∗ π1
• 12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Lower Bound

¯ H(Y ) ≥ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

Proposition: Low-SNR (α = 1

2 − ǫ, 0 ≤ q < 1) Upper Bound

¯ H(Y ) ≤ 1 − 2 log(e)(1 − 2π1)2ǫ2 + O(ǫ4)

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR

For α = 1

2 − ǫ and 0 ≤ q < 1, we have

lim

ǫ→0

1 − ¯ H(Y ) ǫ2 = 2 log(e) 1 − q 1 + q 2

12 / 13

Special Case - (1, ∞)-RLL Constraint

{Xn} : Yn = Xn ⊕ Zn 1 q 1 1 − q

Theorem: Low-SNR

For α = 1

2 − ǫ and 0 ≤ q < 1, we have

lim

ǫ→0

1 − ¯ H(Y ) ǫ2 = 2 log(e) 1 − q 1 + q 2 Recovers result from Han-Marcus’07 and Pfister’11

12 / 13

Summary and Conclusions

We derived a new lower bound for the entropy rate of binary hidden Markov processes

13 / 13

Summary and Conclusions

We derived a new lower bound for the entropy rate of binary hidden Markov processes The bound relies on a strengthened version of MGL, due to Samorodnitsky

13 / 13

Summary and Conclusions

We derived a new lower bound for the entropy rate of binary hidden Markov processes The bound relies on a strengthened version of MGL, due to Samorodnitsky We improved the best known bound for symmetric processes in low-SNR, and recovered the best known results in some other regimes

13 / 13

Summary and Conclusions

We derived a new lower bound for the entropy rate of binary hidden Markov processes The bound relies on a strengthened version of MGL, due to Samorodnitsky We improved the best known bound for symmetric processes in low-SNR, and recovered the best known results in some other regimes Our technique can be generalized to hidden Markov processes over larger alphabets

13 / 13