SLIDE 1
In-Home Daily-Life Captioning Using Radio Signals
Lijie Fan* Tianhong Li* Yuan Yuan Dina Katabi MIT CSAIL
* denotes equal contribution
In-Home Daily-Life Captioning Using Radio Signals Lijie Fan* - - PowerPoint PPT Presentation
In-Home Daily-Life Captioning Using Radio Signals Lijie Fan* - - PowerPoint PPT Presentation
In-Home Daily-Life Captioning Using Radio Signals Lijie Fan* Tianhong Li* Yuan Yuan Dina Katabi MIT CSAIL * denotes equal contribution How can I make sure grandma is fine? How can I make sure grandma is fine? Daily Life Captioning
SLIDE 2
SLIDE 3
How can I make sure grandma is fine? Daily Life Captioning
08:30am: Grandma wakes up and leaves bedroom 10:30am: Grandma takes medicine and eats breakfast 02:00pm: Grandma is watching TV
SLIDE 4
Camera is not acceptable
Camera
SLIDE 5
How to do Daily Life Captioning?
SLIDE 6
What about Radio-Frequency(RF) Signals?
RF Device
SLIDE 7
RF signals are privacy-preserving β¦
RGB Video RF Signals
SLIDE 8
RGB Video RF Signals
but are capable of capturing peopleβs movements and activities
SLIDE 9
Challenge I. Object Information
SLIDE 10
Challenge I. Object Information
SLIDE 11
Challenge I. Object Information
SLIDE 12
Solution I. Skeleton + Floormap
RF Signal Skeleton Generation Network Skeleton
SLIDE 13
Floormap Illustration
Bed Stove Sink TV RF Device Fridge Wardrobe Shelf Window Dish Washer Sofa
Solution I. Skeleton + Floormap
X Y
Table
SLIDE 14
Challenge II. No Existing RF Captioning Dataset! Can We Leverage Existing RGB Captioning Dataset?
SLIDE 15
Solution II. Multi-modal Feature Alignment
RF+Floormap Feature Extraction Feature Extraction Network
RF Signal Floormap
+
π―π
SLIDE 16
Solution II. Multi-modal Feature Alignment
RF+Floormap Feature Extraction
Paired Video ππ Video Encoder
Feature Extraction Network
RF Signal Floormap
+
Video Feature Extraction π―π π°π
π
π°π
π
Spa Spati tial πΈππππππ
SLIDE 17
Paired Data Alignment Loss
βππππ
π2
Solution II. Multi-modal Feature Alignment
RF+Floormap Feature Extraction
Paired Video ππ Video Encoder
Feature Extraction Network
RF Signal Floormap
+
Video Feature Extraction π―π π°π
π
π°π
π
Spa Spati tial πΈππππππ
SLIDE 18
Paired Data Alignment Loss
βππππ
π2
Solution II. Multi-modal Feature Alignment
RF+Floormap Feature Extraction
Paired Video ππ Video Encoder
Feature Extraction Network
RF Signal Floormap
+
Video Feature Extraction
Unpaired Video ππ Video Encoder
π―π π°π
π
π°π
π
π°π
π
π°π
π
Spa Spati tial πΈππππππ Spa Spati tial πΈππππππ
SLIDE 19
Paired Data Alignment Loss
βππππ
π2
Solution II. Multi-modal Feature Alignment
RF+Floormap Feature Extraction
Paired Video ππ Video Encoder
Feature Extraction Network
RF Signal Floormap
+
Video Feature Extraction
Unpaired Video ππ Video Encoder
π―π
Unpaired Data Alignment Loss
βπ£πππππ
πΈπ πΈπ
π°π
π
π°π
π
π°π
π
π°π
π
Spa Spati tial πΈππππππ Spa Spati tial πΈππππππ
SLIDE 20
RF-Diary System Structure
SLIDE 21
RF-Diary can caption peopleβs daily life in home β¦
RF Signals Floormap RGB Video RF-Caption
A person enters the
- kitchen. He takes off
his clothes, sits at table and starts playing laptop.
SLIDE 22
Even when the light is off β¦
RF Signals Floormap RGB Video RF-Caption
A person walks to the
- kitchen. He then pours
water into a cup and drinks from it. Not Applicable
SLIDE 23
Quantitative Results
SLIDE 24
Summary
- RF-Diary enables captioning peopleβs daily life in their home.
- RF-Diary uses radio signals as input to address the privacy issues of
camera.
- RF-Diary achieves comparable results of camera-based captioning
and keeps working under poor lighting or occluded scenarios.
SLIDE 25