Position Detection For a Camera Pen Using LLAH and Dot Patterns - PowerPoint PPT Presentation

Position Detection For a Camera Pen Using LLAH and Dot Patterns Matthias Sperber German Research Center for Artificial Intelligence Osaka Prefecture University

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Goal: Develop Digital Pen n Paper documents n Digital documents n Higher readability n Flexible organizing n Handwriting n Convenient editing applicable n Searchable

Goal: Develop Digital Pen n Example Use Case OCR handwriting recognition notes notes from from Full-text class class search for “class”

Goal: Develop Digital Pen n Capture Pen 
 n Inexpensive Position n Portable n Reconstruct 
 n Require no special Handwriting paper n Distinguish documents

Pen Tablet + Write using stylus or ordinary pen & paper + Fairly inexpensive (no running 
 costs) - Not portable - Documents 
 not 
 distinguished

Ultrasonic Pen + Works on any writing surface + Inexpensive - Device must be calibrated - Documents not distinguished

Anoto Pen + Can distinguish documents + Highly portable - Special paper must be bought - Expensive (high running costs) - Black dots rather apparently visible

Proposed Camera Pen n Yellow Dot Pattern n Print on ordinary paper n Ordinary pen equipped with cheap, low- resolution camera

Proposed Camera Pen Print yellow dots on paper Print document foreground (or leave empty) Write using camera pen Reconstruct handwriting

Comparison of Technologies Pen Ultra- Anoto Proposed tablet sonic pen ü üü û ü Low Cost Normal ( ü ) ü û ü paper û ( û ) ü ü Portable Distinguish û û ü ü documents

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Locally Likely Arrangement Hashing n Retrieve document images n Feature points from document foreground Captured camera image n Retrieval by matching individual points n Determine position

Locally Likely Arrangement Hashing Storage Retrieval Feature Feature points points from partial for every document document image Calculation of Indices # documents & dots affect accuracy! LLAH Database

Calculation of Indices n Local arrangements A n Geometric invariant : 
 Area Ratio B C P ( A , C , D ) D P ( A , B , C )

LLAH With Dot Patterns è Print yellow dots in background ✖ No feature points!

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

The Dot Pattern n Yellow n Almost invisible to human eye n Can still be extracted by computer n Randomized n Start: Regular Grid n Gauss distribution n Bounding box n Avoid “holes”

The Dot Pattern n Dot spacing: 2.7mm n Diameter: 0.2mm n (Anoto’s dot spacing: 0.3mm)

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Overview: Retrieval Steps Feature points Camera image LLAH Calculate pen position (Doc ID, Coordinates)

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Dot Extraction Feature points Camera image LLAH Calculate pen position (Doc ID, Coordinates)

Dot Extraction n Distance-Image è Adaptive Thresholding n “Distance Image”: For each pixel, determine how close its color is to the color yellow

Dot Extraction n Distance-Image è Adaptive Thresholding

Dot Extraction n Distance-Image è Adaptive Thresholding

Dot Extraction n Distance-Image è Adaptive Thresholding

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

LLAH Feature points Camera image LLAH Calculate pen position (Doc ID, Coordinates)

Calculate Pen Position Feature points Camera image LLAH Calculate pen position (Doc ID, Coordinates)

Position Calculation n Estimate geometric transformation n Reconstruct handwriting by drawing lines between consecutively determined positions

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Examples & Results No foreground Little foreground Much foreground Bounding box DB size Retrieval accuracy (% of correctly determined positions) 100 100.0% 94.0% 74.4% 1000 96.2% 80.4% 59.3%

Examples & Results

Examples & Results n Smaller document DB (100 docs)

Examples & Results n Larger document DB (1000 docs)

Examples & Results n Performance Measurements n Intel Core CPU @ 2.13GHz, 3GB RAM Area of captured image 2.2 × 1.6cm 2 3.1 × 2.3cm 2 DB size 1 document 18.5ms 23.8ms 100 documents 22.0ms 30.6ms 1,000 documents 49.4ms 53.5ms

Outline n Introduction n Locally Likely Arrangement Hashing n The Dot Pattern n Overview: Retrieval Steps n Dot Extraction from the Camera Image n Position Calculation n Examples & Results n Outlook & Conclusion

Outlook (using large Document DB) n Problem: yellow dots 
 hidden by too much 
 document foreground

Outlook n Problem: yellow dots 
 hidden by too much 
 document foreground n Solution: Use feature points from both background (yellow dots) and foreground (characters)

Outlook n Combination of techniques: n Proposed method to establish absolute position and current document n Tracking method to measure relative movement and reconstruct handwriting

Conclusion n Low-cost camera pen n Used cheap USB camera n Yellow dots printable on ordinary hardware n Support of 1,000+ documents n Reasonably fast retrieval speed n Future work: n Make more stable when too much document foreground: Incorporate features from foreground!

Thank you for your attention