Shaping the Future of Medical Ultrasound Imaging with AI and GPU - - PowerPoint PPT Presentation
Shaping the Future of Medical Ultrasound Imaging with AI and GPU Computing GTC 2019 Conference Session S8712 Raphael Prevost Senior Research Scientist @ ImFusion In this session, you will see how AI enables to transform a video clip
Senior Research Scientist @ ImFusion
March 20th, 2019
… transform a video clip into a 3D volume… …and reconstruct my carotid in real-time ! …improve orthopedic and brain surgery… Develop an auto-focus system…
2/59
in Munich, Germany
in medical imaging and computer vision
clinical products and used by large companies, start-ups and research labs
March 20th, 2019 3/59
Proj
ect Consu nsulti lting ng
From feasibility studies to implementation
Rese search arch & Deve velo lopm pment nt
Solving challenging problems with state-of-the-art algorithms
Software ftware Deve velo lopm pment ent Kit
ImFusion SDK serves as an ideal platform for R&D
Implementation plementation & Inte ntegra grati tion
Running our software within your medical product
March 20th, 2019 4/59
March 20th, 2019 5/59
March 20th, 2019 6
March 20th, 2019
Credit: Yale University
7/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 8/59
March 20th, 2019 9/59
March 20th, 2019
Shadows
9/59
March 20th, 2019
Shadows Mirroring
9/59
March 20th, 2019
10/59
March 20th, 2019
10/59
March 20th, 2019
✓ Portable
11/59
March 20th, 2019
✓ Portable ✓ Cheap
1M $ 5 5 - 50K K $
11/59
March 20th, 2019
✓ Portable ✓ Cheap ✓ Safe
1M $ 5 5 - 50K K $
11/59
March 20th, 2019
✓ Portable ✓ Cheap ✓ Safe ✓ High spatial resolution
1M $ 5 5 - 50K K $
0.05 mm
11/59
March 20th, 2019
✓ Portable ✓ Cheap ✓ Safe ✓ High spatial resolution ✓ Real-time acquisition → suitable for OR
1M $ 5 5 - 50K K $
0.05 mm
11/59
March 20th, 2019
AI and GPU computing to unlock this potential Our vision
✓ Portable ✓ Cheap ✓ Safe ✓ High spatial resolution ✓ Real-time acquisition → suitable for OR
1M $ 5 5 - 50K K $
0.05 mm
11/59
March 20th, 2019
Image Filtering Image Segmentation
12/59
March 20th, 2019
Image Filtering Image Segmentation For many clinical applications, we need 3D information (measurements, navigation during surgery, etc.)
12/59
March 20th, 2019
Matrix Array “3D probe” Tracking (optical or electro-magnetic) Motorized Transducer “wobbler”
Philips xMatrix
13/59
Ultraso trasoun und Swee eep 2D ultrasound frames, each associated with a 4x4 matrix (position + orientation)
March 20th, 2019 14/59
March 20th, 2019
in partnership with
15
March 20th, 2019
Before Surgery
16/59
March 20th, 2019
During Surgery Before Surgery
16/59
March 20th, 2019
During Surgery Before Surgery
16/59
March 20th, 2019
During Surgery Before Surgery
16/59
March 20th, 2019
During Surgery Before Surgery
Missing transformation
16/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones and detect landmarks Open the region of surgery Digitize landmarks on the patient’s bone Register pre-op/intra-op landmarks Navigate using the pre-op data
17/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones and detect landmarks Open the region of surgery Digitize landmarks on the patient’s bone Register pre-op/intra-op landmarks Navigate using the pre-op data
17/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones and detect landmarks Open the region of surgery Digitize landmarks on the patient’s bone Register pre-op/intra-op landmarks Navigate using the pre-op data
17/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones and detect landmarks Open the region of surgery Digitize landmarks on the patient’s bone Register pre-op/intra-op landmarks Navigate using the pre-op data
17/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones and detect landmarks Open the region of surgery Digitize landmarks on the patient’s bone Register pre-op/intra-op landmarks Navigate using the pre-op data
17/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones Register pre-op/intra-op bone surface Navigate using the pre-op data
Acquire a tracked 3D Ultrasound sweep Extract the bone surface
18/59
March 20th, 2019
Acquire CT/MR image before the operation Segment the bones Register pre-op/intra-op bone surface Navigate using the pre-op data
Acquire a tracked 3D Ultrasound sweep Extract the bone surface
18/59
March 20th, 2019
High shape and appearance variability
19/59
March 20th, 2019
High shape and appearance variability
19/59
March 20th, 2019
U-Net Architecture (most popular for medical image segmentation)
from https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/
20/59
March 20th, 2019
US Image
The segmentation is then refined at the pixel-level
Random Forest Neural Network
21/59
March 20th, 2019
US Image
The segmentation is then refined at the pixel-level
Random Forest Neural Network
21/59
March 20th, 2019
US Image
The segmentation is then refined at the pixel-level
Random Forest Neural Network
21/59
March 20th, 2019
US Image
The segmentation is then refined at the pixel-level If tracking data is available for each frame, a 3D segmentation can be generated
Random Forest Neural Network
21/59
March 20th, 2019
Opt ptimiza mization tion prob
em Minimize the distance between each point and the closest point on the surface
22/59
March 20th, 2019
Opt ptimiza mization tion prob
em Minimize the distance between each point and the closest point on the surface
22/59
March 20th, 2019
Salehi & Prevost et al. Precis ise e Ultra rasoun und d Bone Registrati stration n with Learning ning-Based sed Segmenta mentati tion n and Speed ed of Sound d Calibr brati tion MICCAI 2017
23/59
March 20th, 2019 24/59
March 20th, 2019
Train a neural network on different bones separately by encoding them as multiple channels
24/59
March 20th, 2019
Train a neural network on different bones separately by encoding them as multiple channels
24/59
March 20th, 2019 25/59
The whole system needs to be precisely calibrated
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized
March 20th, 2019
Bone surface can be more or less fuzzy
25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally
March 20th, 2019
Temporal delay between images and position information How to synchronize hronize them em ?
Position Images
Ultrasound system Tracking camera
25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally ✓ Speed of sound must be compensated
March 20th, 2019 25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally ✓ Speed of sound must be compensated
March 20th, 2019
US systems assume a constant speed of sound However, sound travels at different speeds in fat and muscle US System Assumption 1540 m/s Fat 1470 m/s Muscle 1620 m/s
25/59
The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally ✓ Speed of sound must be compensated
March 20th, 2019 25/59
Such processes are usually tedious and complex The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally ✓ Speed of sound must be compensated
March 20th, 2019 25/59
Such processes are usually tedious and complex ...but we can leverage our real-time algorithms to solve them! The whole system needs to be precisely calibrated ✓ Acquisition parameters must be optimized ✓ System must be calibrated geometrically and temporally ✓ Speed of sound must be compensated
March 20th, 2019 25/59
March 20th, 2019
Cameras can automatically find the region of interest in an image and optimize the acquisition parameters Exposure Time Focus
26/59
March 20th, 2019
Cameras can automatically find the region of interest in an image and optimize the acquisition parameters Exposure Time Focus
26/59
March 20th, 2019
Cameras can automatically find the region of interest in an image and optimize the acquisition parameters Exposure Time Focus
26/59
March 20th, 2019
Focus Frequency Brightness
27/59
March 20th, 2019
Focus Frequency Brightness
27/59
March 20th, 2019
Focus Frequency Brightness
27/59
March 20th, 2019 28/59
March 20th, 2019
depth of the bone
28/59
March 20th, 2019
depth of the bone
(high frequencies do not travel deep enough)
28/59
March 20th, 2019
depth of the bone
(high frequencies do not travel deep enough)
can be adjusted by computing intensity statistics
28/59
March 20th, 2019
in partnership with
29
March 20th, 2019
Speed of sound correction Temporal calibration
Salehi & Prevost et al. Precis ise e Ultra rasoun und d Bone Regis istrat tratio ion n with Learning ning-Based sed Segment entati tion n and Speed ed of Sound d Calibr brati tion MICCAI 2017
30/59
March 20th, 2019 31
Where is the tumor? How big is it?
March 20th, 2019 32/59
Where is the tumor? How big is it?
n shif ift: t: When the skull is opened, gravity causes the brain to collapse
March 20th, 2019
Lu, Jun-Feng, et al. NeuroImage: Clinical 2 (2013): 132-142
32/59
Where is the tumor? How big is it?
n shif ift: t: When the skull is opened, gravity causes the brain to collapse
Deformable registration to the MR image → Planning can be used
March 20th, 2019
Lu, Jun-Feng, et al. NeuroImage: Clinical 2 (2013): 132-142
32/59
Where is the tumor? How big is it?
n shif ift: t: When the skull is opened, gravity causes the brain to collapse
Deformable registration to the MR image → Planning can be used
March 20th, 2019
Lu, Jun-Feng, et al. NeuroImage: Clinical 2 (2013): 132-142
32/59
March 20th, 2019 33/59
March 20th, 2019 33/59
Correction of Brainshift with Intra-Operative Ultrasound https://curious2018.grand-challenge.org
March 20th, 2019 34/59
March 20th, 2019
… but still computationally intensive→ GPU implementation Top 3 methods were not based on machine learning
Wein et al. Global Regis istr tratio ation n of Ultra rasound und to MRI Using g the LC2 Metri ric for Enabli ling ng Neuro rosurg urgic ical al Guidanc nce MICCAI 2013
35/59
March 20th, 2019
in partnership with www.pi .piurimag rimaging ing.com .com
36
March 20th, 2019
and Surveillance
scanner after injection of contrast agents → Expensive, long, toxic → Not suited for screening or monitoring
MR
37/59
March 20th, 2019
and Surveillance
scanner after injection of contrast agents → Expensive, long, toxic → Not suited for screening or monitoring
source: piurimaging.com MR US
37/59
March 20th, 2019
Imag age-Ba Base sed Reconstru structio ction
No hardware
Our Goal
Matrix ix Array ay “3d probe” Track ckin ing g (opt ptic ical/ al/EM) M)
Limited field of view Decreased image quality Expensive Not portable
Motori rized d Trans nsdu ducer cer “wobbler”
Limited field of view Temporal artifacts
Philips xMatrix
Existi sting ng Hardware Solutions lutions
38/59
March 20th, 2019
Imag age-Ba Base sed Reconstru structio ction
No hardware
Our Goal
Matrix ix Array ay “3d probe” Track ckin ing g (opt ptic ical/ al/EM) M)
Limited field of view Decreased image quality Expensive Not portable
Motori rized d Trans nsdu ducer cer “wobbler”
Limited field of view Temporal artifacts
Philips xMatrix
Existi sting ng Hardware Solutions lutions
38/59
March 20th, 2019
Imag age-Ba Base sed Reconstru structio ction
No hardware
Our Goal
Matrix ix Array ay “3d probe” Track ckin ing g (opt ptic ical/ al/EM) M)
Limited field of view Decreased image quality Expensive Not portable
Motori rized d Trans nsdu ducer cer “wobbler”
Limited field of view Temporal artifacts
Philips xMatrix
Existi sting ng Hardware Solutions lutions
38/59
March 20th, 2019
Imag age-Ba Base sed Reconstru structio ction
No hardware
Our Goal
Matrix ix Array ay “3d probe” Track ckin ing g (opt ptic ical/ al/EM) M)
Limited field of view Decreased image quality Expensive Not portable
Motori rized d Trans nsdu ducer cer “wobbler”
Limited field of view Temporal artifacts
Philips xMatrix
Existi sting ng Hardware Solutions lutions
38/59
March 20th, 2019
Imag age-Ba Base sed Reconstru structio ction
No hardware
Our Goal
Matrix ix Array ay “3d probe” Track ckin ing g (opt ptic ical/ al/EM) M)
Limited field of view Decreased image quality Expensive Not portable
Motori rized d Trans nsdu ducer cer “wobbler”
Limited field of view Temporal artifacts
Philips xMatrix
Existi sting ng Hardware Solutions lutions
38/59
March 20th, 2019
I1 I2 T1→2
Frame-to-frame motion estimation
I1 I2 T1→2
Rigid Transformation Algorithm
39/59
March 20th, 2019
I1 I2 T1→2 I3 I4 T2→3 T3→4
Frame-to-frame motion estimation
I1 I2 T1→2
Rigid Transformation Algorithm
39/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching)
40/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching)
40/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching)
40/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching) Out-of
plane motion ion is much more difficult because the image content changes
40/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching) Out-of
plane motion ion is much more difficult because the image content changes
Out-of-plane Displacement tz Patch similarity
The more the content changes, the higher the out-of-plane displacement
40/59
March 20th, 2019
In In-pl plane ane motio ion n is easy to detect (optical flow, block matching) Out-of
plane motion ion is much more difficult because the image content changes
Out-of-plane Displacement tz Patch similarity
The more the content changes, the higher the out-of-plane displacement Stand ndard ard approach ch = Speckle eckle dec ecorrelat ation ion
into patches
= 3D vector field
40/59
March 20th, 2019
Out-of-plane Displacement Correlation
Issues es of spec eckl kle e dec ecorrelati elation
(depends on the tissues, on the acquisitions parameters, etc.)
(2D registration, decorrelation, transformation fitting)
41/59
March 20th, 2019
Out-of-plane Displacement Correlation
Issues es of spec eckl kle e dec ecorrelati elation
Ou Our End-to to-end end Approach ach
(depends on the tissues, on the acquisitions parameters, etc.)
(2D registration, decorrelation, transformation fitting)
pair of images → transformation parameters One model to solve the whole problem
Convolutional Neural Network
41/59
March 20th, 2019
No need for manual labeling We just need to acquire a lot of tracked sweeps (but calibration must be super accurate) 800 sweeps (400k frames)
42/59
3 translation + 3 rotation parameters (probe motion between the 2 images)
March 20th, 2019
Regression L2 loss Consecutive frames encoded as a multi-channel image
43/59
March 20th, 2019
Pre-compute the optical flow (in-plane motion) and use it as additional channel
4-channel input 2 ultrasound images + 2D vector field 2-channel input 2 ultrasound images
44/59
March 20th, 2019 45/59
March 20th, 2019 46/59
March 20th, 2019
Reconstruction of sweeps with strong motions and rotations
47/59
March 20th, 2019
Reconstruction of a sweep following the great saphenous vein (more than 60cm) Reconstruction of sweeps with strong motions and rotations
47/59
March 20th, 2019
Accuracy racy study y conducted cted on 800 US US sweep eeps s on vario ious us anatomie mies s Outperforms state-of-the-art methods Median drift of 5% over long sweeps
48/59
March 20th, 2019
All papers and references available on www.imfusion.com
49/59
March 20th, 2019
in partnership with
50
March 20th, 2019 51/59
March 20th, 2019 51/59
March 20th, 2019 51/59
March 20th, 2019
Anatomical structures do not match because of compression
51/59
March 20th, 2019
Anatomical structures do not match because of compression
51/59
March 20th, 2019
52/59
March 20th, 2019
52/59
March 20th, 2019
52/59
March 20th, 2019
52/59
March 20th, 2019
52/59
March 20th, 2019 53/59
March 20th, 2019
Schulte zu Berge et al. Ultrasound sound Dec ecom
pression for Large e Field ld-of
ew Rec econ
struc ucti tions ns VCBM 2018
54/59
March 20th, 2019 55
March 20th, 2019 56/59
March 20th, 2019
Auto-tuning of the parameters Real-time anatomy recognition Trackingless 3D Reconstruction
56/59
March 20th, 2019
Auto-tuning of the parameters Real-time anatomy recognition Trackingless 3D Reconstruction Orthopedic surgery Neuro surgery Vascular imaging
56/59
March 20th, 2019
Auto-tuning of the parameters Real-time anatomy recognition Trackingless 3D Reconstruction Orthopedic surgery Neuro surgery Vascular imaging
With AI + GPU computing + advanced algorithms, US becomes more accessible and create new applications … maybe even replace other modalities in the long run
56/59
March 20th, 2019 57/59
March 20th, 2019
Download the ImFusion Suite demo www.imf .imfusion usion.com .com
Image Visualization, Segmentation, Registration, Mesh/Point Cloud Processing, … and more!
57/59
NVIDIA Clara initiative for transparent access to accelerated computing (closer to the sensor/raw data for certain applications & high-end systems
March 20th, 2019
source: https://developer.nvidia.com/clara
58/59
March 20th, 2019 59/59
Raphael Prevost E-Mail: prevost@imfusion.com Web: www.imfusion.com
Auto-tuning of the parameters Real-time anatomy detection Tracking-less 3D ultrasound Multi-modal registration Decompression & Stitching