Subgoals in Hierarchical Reinforcement Learning Tianren Tang Tian - - PowerPoint PPT Presentation

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

Tianren Tang Shangqi Guo Tian Tan Xiaolin Hu Feng Chen

Background

• Goal-Conditional HRL
• High policy suffers from non-stationary problem
• From MARL's perspective, agent's policy is influenced by other agents
• Another Perspective
• Usually the action space for high policy is too large, therefore its action which

is sub-goal for low policy usually unreachable

• Intuitively, action space reduction or action elimination
• Drawbacks:
• no similiar literature shows how to do space reduction
• Reduction or elimination may cause sub-optimal

Intuition

• Restrict space into k-step adajecent region

Theoretical Analysis

• Shortest Transition Time
• For optimal policy 𝜌∗
• where 𝜒−1: 𝐻 → 𝑇 is a mapping from goal to state s

Theoretical Analysis

• k-step adjacent region of s is defined:
• Theorem 1:
• there is always a surrogate goal 𝑕’ ∈ 𝐻𝐵 that 𝜌∗(𝑏∗|𝑡, 𝑕’) = 𝜌∗(𝑏∗|𝑡, 𝑕)
• Theorem 2:
• 𝑕’ ∈ 𝐻𝐵, 𝑅∗(𝑡, 𝑕’) = 𝑅∗(𝑡, 𝑕)

Theoretical Optimizations

• Original optimization objective

where 𝜐∗ = (𝑡0. . . 𝑡𝑈𝐿), 𝜍∗ = (𝑕0. . . 𝑕(𝑈−1)𝐿)

• Relax above equations:

HRL with Adjacency Constraint

• Adjacent Matrix approximation
• Contrasitive Loss

Final Optimization Objective

• With a learned adjacency network

Algorithm

Experiment Environment

• Discrete & Continuous
• Result

Abalation Study

• Difference:
• HRAC-O: HRAC with perfect adajency matrix from environment
• NegReward: Relabel reward to negative and bound critic function

Visualization

Summary

• Although Intuition is easy, this paper is overall good.