Eliot Siegel, MD, FSIIM, FACR Professor of Radiology University of - - PowerPoint PPT Presentation

Eliot Siegel, MD, FSIIM, FACR Professor of Radiology University of Maryland School of Medicine Chief Imaging Services, VA Maryland Healthcare System

Dw yer AI Session Outline

• Tanveer F. Syeda-Mahmood, PhD

– Chief Scientist, Medical Sieve Radiology Grand Challenge – IBM Almaden Research Center

• Panel Discussion

– Rasu B. Shrestha, MD, MBA, Chief Innovation Officer, University of Pittsburgh Medical Center, Executive Vice President, UPMC Enterprises; – Khan M. Siddiqui, MD, Co-Founder, Chief Technical Officer, higi, Co- Director, Center for Biomedical & Imaging Informatics, Visiting Associate Professor Radiology, Johns Hopkins University

Disclaimer

– ACR – AGFA – Amalga – Anatomic Travelogue – Anthro – Applied Radiology – ATL – Barco – Bracco – Brightfield – Carestream – Cydar – Dejarnette – Dell – Diagnostic Imaging – Digital Art Forms – Dynamic Imaging – Eizo – Fovia – Fuji – Galileo – Herman Miller – IBM – Intel – Kodak – NIBIB – NIST – NLM – NCI – Life images – McCoy – McKesson – Medrad – Medscape – Merge – Microsoft – Montage – GE – Philips – RADLogics – Radsite – Redrick/Evolve – RSNA – Siemens – SIIM – Sonare – Steelcase – TeraRecon – Topoderm – Toshiba – Virtual Radiology – Vital Images – Xybix – YYESIT – Zebra

Samuel J. Dw yer, III, PhD, FSIIM (1932–2008)

• On May 4, 2002 Sam became the ninth SCAR

member to be inducted into the SCAR College of Fellows.

• Dr. Dwyer received his PhD in Electrical

Engineering at the University of Texas-Austin specializing in systems and signal processing

• Dr. Dwyer at the time of his retirement was a

Professor of Radiology at the University of Virginia Health Sciences System

Steve Horii, M.D.

• I knew him to be always ready with a smile or

infectious laugh and with a perpetual gleam in his eye that spoke of his friendly manner

• There are some who would claim the title of

“PACS Man”, but it is Sam Dwyer who led the revolution in PACS

• Sam Dwyer was a major pioneer who brought

many of the important advances in technology to us and helped move concepts from the realm of engineering to that of healthcare

• I will miss Sam very much, but the strong

memory of him is never further than the PACS workstation I use every day

• R. Gilbert Jost, MD

Past President RSNA and RISC

• If one were to identify a “father of PACS”,

unquestionably it would be Sam Dwyer…

• He is truly a pioneer who has changed the

specialty of radiology for the better in innumerable ways

Continued Relationship w ith Sam

• Sam was very reassuring when we became the

world’s first filmless hospital that our problems would not be insurmountable and that the time was right to go filmless

• As this “kid” right out of residency aspiring to the first

filmless hospital, I think I amused Sam

• In subsequent years we got together frequently at

RSNA and he often scribbled messages and drawings on napkins and handed them to me

• Kept in contact by phone and e-mails and always

enjoyed talking with him

• Makes me wonder what Sam would have thought

about “Artificial Intelligence” and its potential in diagnostic imaging

Where Are We in 2016 in AI in Diagnostic Imaging?

• Sam would be surprised that CAD hasn’t

made more progress in diagnostic imaging

• 10’s of thousands of machine learning

algorithms but almost no connection between the research and clinical application of these

• Relatively small incremental improvements

in fairly narrowly defined image analysis algorithms, e.g. mammography CAD, lung nodule detection, vascular stenosis analysis

Where Are We in 2016?

• Is there a generalized learning

algorithm/program for imaging that could create a jumpstart to a major advance in diagnostic imaging?

• Amazing advances in ML and AI in many

domains high publicity

• Lots of companies lately are claiming to

have made that jump

• How much is reality and how much is

hype?

Dedication: to João Louro

11

12

The Economist and Others Are Talking about the 4 th Industrial Revolution Based

• n Cyber-Physical Systems

The substitution of machinery for machine labour” may “render the population redundant The discovery of this mighty pow er” has come “before w e knew how to employ it rightly” Debate in early 1800s about the industrial revolution in England

Easy to Replace?!

• Andrew Ng, renowned Stanford Professor

and expert on machine learning was quoted in The Economist this week as saying “a highly trained and specialized radiologist may now be in greater danger of being replaced by a machine than his own executive assistant: She does so many different things that I don’t see a machine being able to automate everything she does any time soon.”

Ezekiel Emanuel, PhD, MD, MSc

• Gave keynote presentation at ACR 2016

– Faculty member at the Wharton School and School of Medicine at University of Pennsylvania – Founding chair of the Clinical Center of the National Institutes of Health – Former special advisor on health policy for the Office of Management and Budget.

Ezekiel Emanuel, PhD, MD, MSc Keynote ACR 2016 Five Megatrends

• Decline in the use of hospitals
• More outpatient care
• More care in patients’ homes
• Fewer medical tests
• Machine learning

– “While all of these factors will shape the future landscape, machine learning will be the most pressing for radiology – Emanuel called the machine learning “the real

threat to radiology.”

• “The biggest barrier will not be technical but human willingness to

accept machine based diagnoses.”

Major “Inspiration”/”Motivation” for This Year’s 2016 Sam Dw yer Lecture

• Visiting Professor at Hospital of

University of Pennsylvania

• CEO of well funded and well known

start-up company in medical imaging space related that he wanted to (paraphrased) “get rid of the wasted protoplasm sitting in front of the workstation that was the radiologist and replace it with a much better and reliable and consistent alternative in the next few months”

Stephen Haw king on AI

• “Success in creating AI would be the biggest event in human

history,” wrote Stephen Hawking in an op-ed, which appeared in The Independent in 2014.

• “Unfortunately, it might also be the last, unless we learn how to

avoid the risks. In the near term, world militaries are considering autonomous-weapon systems that can choose and eliminate targets.” Professor Hawking added in a 2014 interview with BBC, “humans, limited by slow biological evolution, couldn’t compete and would be superseded by A.I.”

• Hawking told the BBC: “The primitive forms of artificial intelligence

we already have, have proved very useful. But I think the development of full artificial intelligence could spell the end of the human race.”

19

Elon Musk

• Elon Musk has spoken out against artificial intelligence

(AI), declaring it the most serious threat to the survival of the human race to students from Massachusetts Institute

• f Technology (MIT)
• “I think we should be very careful about artificial
• intelligence. If I had to guess at what our biggest

existential threat is, it’s probably that. So we need to be very careful,” said Musk

• “I’m increasingly inclined to think that there should be

some regulatory oversight, maybe at the national and international level, just to make sure that we don’t do something very foolish.”

Bill Gates

• Microsoft co-founder Bill Gates has also expressed

concerns about Artificial Intelligence

• During a Q&A session on Reddit in January 2015,
• Mr. Gates said, “I am in the camp that is concerned

about super intelligence. First the machines will do a lot of jobs for us and not be super intelligent. That should be positive if we manage it well

• A few decades after that though the intelligence is

strong enough to be a concern

• I agree with Elon Musk and some others on this and

don’t understand why some people are not concerned.”

More Dangerous Than Nuclear Weapons?

• Mr. Hawking recently joined Elon Musk, Steve

Wozniak, and hundreds of others in issuing a letter unveiled at the International Joint Conference in Buenos Aires, Argentina

• The letter warns that artificial intelligence can

potentially be more dangerous than nuclear weapons.

22

Who Is Investing All Those Dollars in Artificial Intelligence?

Investment in Artificial Intelligence

• Ironically, given Elon Musk and Sam Altman’s

concern that artificial intelligence will take over the world, the two entrepreneurs are putting more than a billion dollars into a not-for-profit company that will maximize the power of AI— and then share it with anyone who wants it

• In an interview with Steven Levy of

Backchannel about Open AI, Altman said they expect this decades-long project to surpass human intelligence

• But they believe that any risks will be mitigated

because the technology will be “usable by everyone instead of usable by, say, just Google.”

Many Many AIs and Dr. Evil

• They were asked whether their plan to freely share

this technology would actually empower bad actors, if they would end up giving state-of-the-art AI to the Dr. Evils of the world. But they played down this risk

• They feel that the power of the many will outweigh

the power of the few. “Just like humans protect against Dr. Evil by the fact that most humans are good, and the collective force of humanity can contain the bad elements,” said Altman, “we think its far more likely that many, many AIs, will work to stop the occasional bad actors.”

25

AI/Machine Learning Basic Terms

Deep Learning Falls Within Machine Learning Within AI

Artificial Intelligence

• Basically an umbrella term for a variety of

applications and techniques

• Artificial intelligence refers to "a broad set of

methods, algorithms and technologies that make software 'smart' in a way that may seem human- like to an outside observer”

» Lynne Parker, director of the division of Information and Intelligent Systems for the National Science Foundation

• John McCarthy, who coined the term “Artificial

Intelligence” in 1956, complained that “as soon as it works, no one calls it AI anymore.”

Artificial Intelligence

• Machine learning, computer vision,

natural language processing, robotics and related topics are all part of A.I.

• Also referred to as “machine intelligence”
• r “computational intelligence”
• Can distinguish different types of AI
• When will AI Arrive?

– It’s here already!!!

Is There A General Equation for Winning at AI?

• The action-value function is the maximum sum of

rewards rt discounted by γ at time step t, achievable by a behavior policy Π=P(a|s), after making an observation (s) and taking an action (a)

• Can be optimized using a Deep convolutional

neural network

• Key to winning at Atari Video games
• Key to “happiness”?
• Key to LIFE?

Where is AI? Everyw here. My Monday Morning Diary

• Wake up by iphone, One Dance/Drake
• Check indoor temperature on Nest
• Amazon Echo checks out weather and

traffic on the way to work and turns on the lights

• Google Now says flying to Portland

tomorrow with itinerary

• Set temperature in the Car on iphone
• Get read for work while Siri plays latest

unread messages and e-mails

• Car displays today’s schedule from

Google calendar and goes on autopilot and does 95% of driving to work autonomously

• Arrive at work at VA Hospital
• Big stack of papers on desk to be

signed-rummage through drawers to find pen and move papers from one side of the desk to the other

• Take 10 minutes to sign into EMR to

check consults after waiting about

• Take another 8 minutes to sign into

PACS

• Take 10 minutes to play messages on

phone machine

• Grab stack of paper requisitions to

protocol

Artificial Intelligence (Narrow )

• Also referred to as Weak AI
• AI that specializes in one area
• There’s AI that can beat the world

chess champion in chess, but that’s the only thing it does

– Speech recognition – Translation – Self-driving cars – Siri, Alexa, Cortana, Google Now

Artificial General Intelligence

• Sometimes referred to as Strong AI, or

Human-Level AI

• Computer that is as smart as a human

across the board—a machine that can perform any intellectual task that a human being can

• Creating AGI is a much harder task than

creating ANI, and we are nowhere near close to it

Artificial General Intelligence (AGI)

• Professor Linda Gottfredson describes

intelligence as “a very general mental capability that, among other things, involves the ability to:

– Reason – Plan

– Solve problems – Think abstractly – Comprehend complex ideas – Learn quickly – Learn from experience”

When Will AGI Arrive?

• A study, conducted recently by author

James Barrat at Ben Goertzel’s annual AGI Conference asked when participants thought AGI would be achieved—by 2030, by 2050, by 2100, after 2100, or never. The results:

• By 2030: 42% of respondents
• By 2050: 25%
• By 2100: 20%
• After 2100: 10%
• Never: 2%

Artificial Superintelligence

• Oxford philosopher and leading AI

thinker and author Nick Bostrom defines super-intelligence as “an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.”

Machine Learning

• Also blanket term that covers multiple

technologies

• Doesn’t necessarily have to actually

“learn” as we think of it and doesn’t necessarily provide feedback over time just refers to a class of statistical techniques to characterize, discover, classify data

• Vast majority of these have been

around for many years/decades

Machine Learning

• As a part of A.I., machine learning refers to

a wide variety of algorithms and methodologies that can also enable software to improve its performance over time as it obtains more data

• "Fundamentally, all of machine learning is

about recognizing trends from data or recognizing the categories that the data fit in so that when the software is presented with new data, it can make proper predictions," (Parker)

Neural Networks

• Neural networks are a type of

machine learning, and deep learning refers to one particular kind

• Neural networks -- also known as

"artificial" neural networks -- are one type of machine learning that's loosely based on how neurons work in the brain, though "the actual similarity is very minor”

Neural Networks

• There are many kinds of neural networks, but in

general they consist of systems of nodes with weighted interconnections among them

• Nodes, also known as "neurons," are arranged

in multiple layers, including an input layer where the data is fed into the system; an output layer where the answer is given; and one or more hidden layers, which is where the learning takes place

• Typically, neural networks learn by updating the

weights of their interconnections

Examples Neural Netw ork Types

Types of Neural Networks: Autoencoder

• Autoencoder is a simple 3-layer neural network where
• utput units are directly connected back to input units.
• Relatively simple and intuitive

Restricted Boltzman Machine

• Intuition behind RBMs is that there are some

visible random variables (e.g. film reviews from different users) and some hidden variables (like film genres or other internal features), and the task

• f training is to find out how these two sets of

variables are actually connected to each other

Convolutional Neural Networks

• Like Autoencoders and RBMs- translate many

low-level features (e.g. user reviews or image pixels) to a compressed high-level representation (e.g. film genres or edges) - but now weights are learned only from neurons that are spatially close to each other.

• CNNs are very specifically optimal for image
• recognition. Most of the top-level algorithms in

image recognition are somehow based on CNNs today

Purpose of These Neural Networks is Dimensionality Reduction

• Autoencoders and RBMs both take a vector in nn-

dimensional space they translate it into an mm- dimensional one, trying to keep as much important information as possible and, at the same time, remove noise

• If training of autoencoder/RBM was successful, each

element of resulting vector (i.e. each hidden unit) represents something important about the object - shape

• f an eyebrow in an image, genre of a film, field of study

in scientific article, etc.

• You take lots of noisy data as an input and produce

much less data in a much more efficient representation

Neural Networks Used for Pretraining then Other Classifier Used

• None of models mentioned here work as

classification algorithms per se

• Instead, they are used for pre-training -

learning transformations from low-level and hard-to-consume representation (like pixels) to a high-level one

• Once deep (or maybe not that deep)

network is pretrained, input vectors are transformed to a better representation and resulting vectors are finally passed to real classifier (such as SVM or logistic regression)

Deep Learning Neural Networks

Deep Learning

• Deep learning refers to what's sometimes

called a "deep neural network," or one that includes a large system of neurons arranged in several hidden layers

– A "shallow" neural network, by contrast, will typically have just one or two hidden layers.

• The idea behind deep learning is not new,

but it has been popularized more recently because we now have lots of data and fast processors that can achieve successful results on hard problems

Commonly Used Machine Learning Techniques

• Regression techniques
• Neural networks
• Support vector machines
• Decision trees
• Bayesian belief networks
• k-nearest neighbors
• Self-organizing maps
• Case-based reasoning
• Instance-based learning

Hidd M k d l

Machine Learning Vs. Data Mining

• Machine learning focuses on

prediction, based on known properties learned from the training data.

• Data mining focuses on the

discovery of (previously) unknown properties in the data

Machine Learning vs. Optimization

• Optimization algorithms can

minimize the loss on a training set

• Machine learning is

concerned with minimizing the loss on unseen samples

Machine Learning and Statistics and “Statistical Learning”

• Machine learning and statistics are closely related

fields and machine learning can be considered a statistical technique

• Leo Breiman distinguished two statistical modeling

paradigms: data model and algorithmic model, wherein 'algorithmic model' means more or less the machine learning algorithms like Random forest

• Some statisticians have adopted methods from

machine learning, leading to a combined field that they call statistical learning

What is Deep Learning?

• DL consists of multiple hidden layers in an

artificial neural network

• This approach tries to model the way the

human brain processes light and sound into vision and hearing

• Two very successful applications of deep

learning are computer vision and speech recognition

• Falling hardware prices and the development
• f GPUs for personal use in the last few years

have contributed to the development of the concept of Deep Learning (DL)

Deep Learning vs. Machine Learning

56

ImageNet Large Scale Image Recognition Challenge Started in 2010

• Computers have always had trouble

identifying objects in real images so it is not hard to believe that the winners

• f these competitions have always

performed poorly compared to humans.

• But all that changed in 2012 when a

team from the University of Toronto in Canada entered an algorithm called SuperVision, which wiped the floor with the opposition.

SuperVision

• SuperVision, for example,

consists of some 650,000 neurons arranged in five convolutional layers

• It has around 60 million

parameters that must be fine- tuned during the learning process to recognize objects in particular categories.

61

Speech Recognition Deep Learning Breakthrough

Human Vision: The Hardest Task for Computers?

Introduced by Alan Turing in his 1950 paper “Computing Machinery and Intelligence” Opens with the words “I propose to consider the question, ‘Can machines think?” Asks whether a computer could fool a human being in another room into thinking it was a human being Modified Dr. Watson Turing Test might ask: Can a computer fool a human being into thinking it was a doctor?

What’s Wrong w ith this Picture?

Ultimate Challenge: Medical Imaging Scientific American June 2011 Testing for Consciousness Alternative to Turning Test Highlights for Kids “What’s Wrong w ith this Picture?”

Christof Koch and Giulio Tononi

Imaging May Be Ultimate/Future Frontier For “AI” Softw are

Machine Learning Algorithms: Like Standards, So Many to Choose From!

Can We Apply Those Incredible Advances in Object Recognition to Diagnostic Radiology?

• These image challenges have used 24 big “RGB”

color images with no experience with gray scale imaging in medicine

• They can identify a chair but can’t tell if it’s

– Broken – Something is missing – Something extra is there – Comfortable – Beautiful or ugly – Dirty or clean

• Black box – Can’t explain why something is

identified as abnormal

• Adrenal challenge 5th Grader– Need to know

anatomy

“Magic” Aspect of Deep Learning

• One major challenge is that we don’t understand

what’s inside black box of deep learning when it solves a visual recognition challenge

• Don’t need deep learning for Tic Tac Toe or

Checkers or even chess because we can use combination of brute force to look at every possible move (chess out to 20 to 30 or more moves and further at the end game)

• But game like Go or playing video games, can’t do

brute force but can learn by trial and error even though black box without understanding of why, like magic

• No general purpose learning system for

diagnostic imaging like we train our residents

• Our eyes and brains have evolved to detect

patterns and our knowledge of medicine, physiology, a priori likelihood of disease and recognition of trends evolved over millions of years

Challenges for Machine Learning Algorithms

• Which to choose from?
• How do evaluate different machine learning algorithms

and determine which is most efficient for a particular problem?

• Black box?
• How to optimize parameters?

– Practical Beyesian Optimization Machine Learning

• Computational Time
• Very specific MLA’s do a good job at different tasks which

makes it difficult to select a single one as a generalized deep AI approach for image analysis or for data analysis

• Problems with High Dimensional Datasets like electronic

medical record requires different approach

Applications of “Machine Learning” in Medical Imaging

These Machine Learning Techniques Have Been Utilized in Imaging for Decades With Tens of Thousands of Published Papers

10,000s of Narrow Machine Learning Applications in Medicine Challenge is Binding these together and can one develop general learning theory?

• Fracture detection
• Brain hemorrhage
• Mammography
• MS diagnosis and quantification
• Bone age determination
• Lung nodule detection
• Liver mass determination
• Meniscal tear
• Brain segmentation and diagnosis
• Bone mineral density on CT
• Carotid stenosis evaluation
• Coronary Artery stenosis evaluation
• Cardiac function evaluation
• MRI Mammography CAD

Eliot Siegel, M.D.

• Prof. and Vice Chair University of Maryland

Chief Imaging VA Maryland Healthcare System

Black Box of CAD

• The “black box” nature of CAD is seen as a

substantial issue by many radiologists

• If my residents and fellows told me they

thought this right upper nodule was cancer and I asked why and they wouldn’t say why

• r how confident they were, I’d:

– Be less confident – Be suspicious about their analysis – Be frustrated

What Made You Circle the Lesion?

• Lesion size
• Lesion morphology (shape: smooth,

spiculated)

• Density distribution (solid, ground glass,

partially calcified)

• Location (subpleural, which lobe)
• Connectedness (is it connected to vessels or
• ther structures?)

Level of Confidence and Quality

• f Evidence
• What was your level of confidence in the magic box formula of the

above that made you circle it?

– Did it have to meet size, morphology, density and connectedness or even location characteristics – What database did you use to determine level of suspicious of cancer and how many cases were in it

• r was it based on expert opinion?
• 10
• 100
• 1000
• 10,000
• More?

Left Upper Lobe Lung Nodule

CAD is Pretty Sure It’s There

CAD is Not So Confident

Where Are We Today With Clinical Use of CADe?

• Mammography is far and away the most

utilized application

• But what do radiologists really think of

Mammography CAD?

Is CAD gaining momentum in clinical practice?

– It seems to be happening too slowly, more slowly so than most of us had anticipated – In cases (unlike mammography) where there is no reimbursement for CAD, the radiologists and practices are feeling that their margins are low enough and there is major pressure related to decreased reimbursement and the impression that reimbursement will continue to drop

• Difficult to make business case for added

expenditure for CAD to radiologists

Is CAD Gaining Momentum In Clinical Practice?

– There is much skepticism among my colleagues about the added value of CAD and many only use it for mammography because of the reimbursement model – Colleagues will not pay any significant amount for say, CAD lung nodule detection for chest radiography even with a hypothetical scenario of a 10 or even 20% increase in sensitivity – I believe that they would pay more for something that increased their efficiency and productivity than their accuracy

• Separate links to an online survey were

posted on the website of the Society of Breast Imaging and circulated to subscribers

• f Diagnostic Imaging.com, in order to

evaluate opinions regarding CAD use and its underlying legal issues

M ATERIALS AND M ETHODS

:

• 89% indicated they always use CAD

when reading screening mammograms

• 4% indicated that they rarely or never

use CAD

RESUL TS: Use and Reliance on CAD?

Use and Reliance on CAD and Reimbursement

• However the extent to which clinicians are

relying on CAD to provide an accurate diagnosis is split

–2% indicated that they always rely on CAD to

provide an accurate diagnosis

–49% indicated they sometimes rely on CAD – 49% clinicians rarely or never rely on CAD

– It is likely that the mismatch between use of and

reliance on CAD relates to the reimbursements ($12, or $1000 per approximately 83 cases or $2,400 per day for 200 screening mammograms) radiologists receive when using CAD

RESUL TS: Use and Reliance on CAD?

§ M ost radiologists have not changed a

read based on the results of CAD

§ Only 2% indicated they alters their

• pinion after CAD

§ 36% sometimes change interpretation

based on CAD

§ 61.7% rarely or ever change their

interpretation based on CAD

Use and Reliance on CAD

• 15% found that CAD was often helpful
• 49% considered it sometimes helpful 36%

considered it rarely or never helpful

What Do I Need from Next Generation CAD Clinically?

• Improve efficiency/productivity
• Increases my accuracy/reliability

without compromising efficiency

• Affordable
• Increases my confidence
• Allows me to measure things I couldn’t

measure otherwise such as liver or pulmonary “texture”

• Provide Imaging “Physical Exam”

Next Generation of CAD

• The next generation of CAD will reflect the trend

toward big data and personalized medicine and shift away from the current second reader approach and toward one in which CAD algorithms increasingly serve as visualization and image measurement/annotation and quantification tools

– Examples of probability maps rather than just binary yes and no and FDA requirements shaped the second reader – Tracking lesions over time – Highlighting certain types of findings to draw attention to the reader

CAD Tools Requirements and Challenges

• CAD applications must be able to be integrated into the

image acquisition, display and interpretation workflow

• They will not be adopted if they constrain the throughput of

the radiologist

• Need high level of accuracy in a single patient, need to

more than just demonstrate efficacy in separating two groups

• Commercialization and U.S. Food and Drug Administration

(FDA) clearance is a big hurdle and needs to be revisited

Next Generation CAD Apps Store for CAD Algorithms?

• Want to be able to utilize all of these on a

single platform, e.g. using API specified by DICOM working group 23

• Would like to see business ecosystem such

as GRID that could allow users to have a payment model for these so you could download algorithm on the fly or send images up to a web service or could get consensus from multiple CAD programs

Israeli Start- Up

• Platform to create and integrate a

variety of algorithms to test against 12 million anonymized, indexed and catalogued imaging studies

• Supports multiple coding languages and libraries,

including Machine Learning Convolutional Network libraries such as Torch, Cafe and Theano, image processing libraries

• All work is saved and projects can be collaborated
• n by several users
• In addition, provide high end, dedicated GPUs and

CPUs to run algorithms

Second Start-up Company Creates Preliminary Report for Chest CT

IBM Medical Sieve

Recommendations for CAD

• I believe the FDA has often limited the

challenges to CAD as a second reader rather than as a tool that can be toggled on and off

– Would like to see highlighted images like a spell checker that could also color code probability that a finding is real/confidence of the CAD algorithm – I’d like to see CAD to do image recognition before a study is reviewed as screening for things such as rib fractures, compression spine fractures, pneumothorax, etc. the equivalent of an imaging physical exam

Applying Human Vision Research to CAD Adrenal Challenge

• Challenge:

–Can any CAD program find the adrenals as well as you could teach an 8 year old child in ten minutes? –I have never seen anyone successfully tackle the problem of finding the adrenals –This would have substantial value

MACHINE LEARNING AND AI FOR LUNG NODULE SCREENING

Eliot Siegel, MD, FACR Prof/Vice Chair IS Univ. Maryland

Five Levels IT Decision Support for Lung Nodule Screening

• 1. Radiologist interprets nodule based on

clinical experience and makes measurement and report

• 2. Radiologist interprets study based on Lung-

RADS criteria and reports score after looking up information on paper or online

• 3. Radiologist has Lung-RADS criteria

available and EMR/PACS automatically pulls up required information to make it easy and in one place

Five Levels IT Decision Support for Lung Nodule Screening With Mr. Akami Example

• 4. PACS brings up lung rads criteria in the context
• f interpretation automatically with ACR Assist

with data automatically classified according to template

• 5. Finally, Automatic click on nodule initiates

search of database such as NLST which then finds similar patients, categorizes based on risk

• f a specific cohort based on detailed nodule

and patient information, utilizes a priori probability based on PLCO and combines those for patient specific probability of disease and then also maps out to LungRads

ACR Assist:

Future Automated Reporting Using Template Directly to Report and Registry

Standa rds Commi ttee

Critical Test Results Mgmt Actionable Reports Registries

Integrating Workflow Structured Content

Evidence-Based Algorithms XML Encoding Structured Input Structured Output

Advanced Decision Support in Action: Your Next Door Neighbor Mr. Akami

• Your next door neighbor and friend,
• Mr. Akami, a 62 year old native

Hawaiian smoker with COPD who gets admitted for an elective Bunionectomy

• 7 mm spiculated soft tissue density

left lower lobe nodule is discovered

• n “routine” pre-op exam and

confirmed on CT with no other abnormalities

• What is the likelihood that it is

malignant?

• How should this nodule be followed

up?

• Do we have tools at the workstation

while reporting to help apply the ACR LungRads criteria to help out the radiologist?

National Lung Screening Trial Dataset and Decision Support Project T aking Personalization to the Next Level Beyond LungRads Reporting T

• ol
• Can we personalize the ACR Lung Rads criteria

using data from the National Lung Screening Trial?

• Could the criteria for follow-up be refined and

personalized more than high risk smoker vs. lower risk patient based on:

• Geographic location?
• Patient age/sex?
• Characteristics of nodule e.g. shape

(spiculated or smoothly rounded), containing calcification?

• Presence of additional nodules?

National Lung Screening Trial

(NLST)

26,721 participants 32,289 nodules

Searching By Cohort Match

5% of Nodules for Males 60 to 65 that were 5-7mm were Malignant

N/ A <1 % 1- 2% 5- 15% >15% 1, 2 3 4A 4B

• Mr. Akami category 3 suggesting

1-2% prob. of malignancy with 6 month follow up

2 9 8 1 5 4 6

Next Phases of NLST Analysis

• Problem with cohort analysis is that cohorts get small very

quickly with increasing number of variables

• Linear regression analysis
• Look up formula for linear regression from literature
• More advanced multi-regression analysis would come

closer to being the best fit

• Bayesian Approach
• Machine learning algorithm even better such as
• Support Vector Machine
• Many other machine learning possibilities
• Deep Learning

Now Add Nodule Shape Matching To Further Personalize and Refine Accuracy

Statistics Features

1. Center Of Gravity 2. Histogram 3. Kurtosis 4. Maximum 5. Maximum Index 6. Mean 7. Median 8. Minimum 9. Minimum Index

• 10. Skewness
• 11. Standard Deviation
• 12. Sum
• 13. Variance
• 14. Weighted Elongation
• 15. Weighted Flatness
• 16. Weighted Principal Axes
• 17. Weighted Principal Moments

Shape Features

1. Bounding Box 2. Centroid 3. Elongation 4. Equivalent Ellipsoid Diameter 5. Equivalent Spherical Perimeter 6. Equivalent Spherical Radius 7. Feret Diameter 8. Flatness 9. Number Of Pixels

• 10. Number Of Pixels On Border
• 11. Perimeter
• 12. Perimeter On Border
• 13. Perimeter On Border Ratio
• 14. Physical Size
• 15. Principal Axes
• 16. Principal Moments
• 17. Roundness

Next Step: Pixel Analysis NLST Images

Extract Nodules and Then Apply One of A Wide Variety of Machine Learning Algorithms Especially Convolutional Neural Network

PLCO Dataset

PLCO Dataset

“Instant Research” Personalized Clinical Care

122

PLCO Participants Who Qualify for NLST

123

Creating Local/Regional Databases from Clinical Data

• Would also like to be able to collect data at the University
• f Maryland, within the Department of Veterans Affairs

Hospitals in Maryland and then nationally that could establish a similar database

• Then could provide report that gave reference database

such as NLST with likelihood of malignancy and also gave local reference to a specific population and then taking into account PLCO data

NLST and PLCO Next Steps

• Huge implications for screening, e.g. reduce cost

from over $200,000 per life saved for smokers over 50 years old to a lower cost for a higher risk cohort for screening studies

• PLCO use has major implications for Bayesian pre-

test probability data to assist in diagnosis

• Working with multiple vendors, demonstrating

ability to incorporate this into the workflow with ability to “click” on nodule and then have automated lesion characterization, lookup from EMR and then access reference database “service” to get information about likelihood of malignancy

• Would like to incorporate these data into

routine applications such as CAD software that could take a priori probability of disease to help to CADx in addition to current CADe, e.g. if patient had prior breast cancer CAD should “realize” odds

• f another breast cancer higher and adjust

accordingly

• Would like to create on the fly

statistical/machine learning models rather than just finding similar patients in databases such as NLST or PLCO

126

• IT is becoming increasingly critical to the success of today’s

practice of radiology and is especially critical as we move to implement the complex process that is associated with Lung Cancer Screening

• Clinical decision support tools are evolving from the current

state of the art to next generation and beyond systems that will allow us to take care of patients such as Mr. Akami in an increasingly safe and effective manner

• This will allow us to maximize the likelihood that our CT

screening studies will save lives and reduce morbidity associated with lung cancer

127

128

129

130

Professor University of Maryland School of Medicine Chief Imaging VAMHCS Adjunct Professor Computer Science UMBC

Board Scientific Counselors National Library of Medicine

What is ORiGAMI ?

An Artificial Intelligence Workflow for Discovering Novel Associations in Massive Medical Knowledge Graphs

ORiGAMI under the hood

ORiGAMI: Case “Patient as Art”

This patient, like the artist who made her famous, was a “cripple and an outsider,” though she was not always so. She began life as a small, blond-haired girl with a “silver giggle,” who seemed no different from other children. However, by the time she reached the age of three, she was walking

• n the outside of her feet with an odd gait. Even so, she was a bright, fiercely determined child, who

stomped around ignoring her disability as it gradually increased in severity. By the time she was 13, she stumbled and fell frequently, though with a mind so “bright, curious and hungry,” her teacher had hopes that she too would one day be a teacher. The patient, if not beautiful at age 19, was slim and handsome enough to attract a suitor, who claimed that during their brief courtship, she could do anything–“row a boat, climb a tree, harness a horse, and drive a carriage.” Her letters at that time, however, told a different story, one involving a series of “bad falls.” Her one and only suitor vanished from her life as suddenly as he had appeared. The patient’s balance soon worsened to the point that it was unsafe for her to look up without having a firm grip on something for steadiness. Although she was still able to walk, her crablike gait forced her to use the entire width of the road when ambulating. Her mother made her kneepads to wear under her skirt as protection against her many falls. Her hands, as yet unaffected, were capable of the intricate work of a talented

• seamstress. By the time she reached 26, the patient could walk only three or four steps without

assistance, and her hands had become so misshaped and unsteady she had to her wrists, elbows, and knees to do those things formerly done with her hands. Offers of help were gently but firmly

• refused. By the end of her fifth decade, she had lost the ability to stand and resorted to crawling to

get where she wanted to go. Her mind continued to be as sharp as ever. No neurological disorders are known to have affected other members of the patient’s family. Her father was a Swedish sailor with a disabling arthritis, who died at age 72 of unknown cause. Her mother developed kidney disease in her 40s and died edematous at age 68 of either renal failure or congestive heart failure. There were three brothers, one who died in his 80s of metastatic bone cancer. The medical histories of the other two are unknown. The patient was evaluated medically just once, when she was 26, at the Boston City Hospital. After a week of observation and tests failed to produce a diagnosis, she was told “to just go on living as [she] had always done. When the patient was 56, she developed a severe illness thought to have been pneumonia. One evening, while recuperating, she sat with one leg stretched out beneath a stove and fell asleep. When she awoke, the heat from the fire had seared the flesh from her withered leg. The third-degree burn healed slowly in response to repeated application of cod liver oil. At age 74, the patient finally consented to the use of a wheelchair and died shortly thereafter.

ORiGAMI: Natural Language Processing

This patient, like the artist who made her famous, was a “cripple and an outsider,” though she was not always so. She began life as a small, blond-haired girl with a “silver giggle,” who seemed no different from other children. However, by the time she reached the age of three, she was walking

• n the outside of her feet with an odd gait. Even so, she was a bright, fiercely determined child, who

stomped around ignoring her disability as it gradually increased in severity. By the time she was 13, she stumbled and fell frequently, though with a mind so “bright, curious and hungry,” her teacher had hopes that she too would one day be a teacher. The patient, if not beautiful at age 19, was slim and handsome enough to attract a suitor, who claimed that during their brief courtship, she could do anything–“row a boat, climb a tree, harness a horse, and drive a carriage.” Her letters at that time, however, told a different story, one involving a series of “bad falls.” Her one and only suitor vanished from her life as suddenly as he had appeared. The patient’s balance soon worsened to the point that it was unsafe for her to look up without having a firm grip on something for steadiness. Although she was still able to walk, her crablike gait forced her to use the entire width of the road when ambulating. Her mother made her kneepads to wear under her skirt as protection against her many falls. Her hands, as yet unaffected, were capable of the intricate work of a talented

• seamstress. By the time she reached 26, the patient could walk only three or four steps without

assistance, and her hands had become so misshaped and unsteady she had to use her wrists, elbows, and knees to do those things formerly done with her hands. Offers of help were gently but firmly refused. By the end of her fifth decade, she had lost the ability to stand and resorted to crawling to get where she wanted to go. Her mind continued to be as sharp as ever. No neurological disorders are known to have affected other members of the patient’s family. Her father was a Swedish sailor with a disabling arthritis, who died at age 72 of unknown cause. Her mother developed kidney disease in her 40s and died edematous at age 68 of either renal failure or congestive heart failure. There were three brothers, one who died in his 80s of metastatic bone

• cancer. The medical histories of the other two are unknown. The patient was evaluated medically

just once, when she was 26, at the Boston City Hospital. After a week of observation and tests failed to produce a diagnosis, she was told “to just go on living as [she] had always done. When the patient was 56, she developed a severe illness thought to have been pneumonia. One evening, while recuperating, she sat with one leg stretched out beneath a stove and fell asleep. When she awoke, the heat from the fire had seared the flesh from her withered leg. The third-degree burn healed slowly in response to repeated application of cod liver oil. At age 74, the patient finally consented to the use of a wheelchair and died shortly thereafter.

Case: “Patient as Art” – Christina’s World

Birth 3 50 26 74

Swedish/Scandinavian Blonde Hair Lost ability to stand Pneumonia

Age Death

Odd Gait Fell Frequently

13

No mental health issues Crab-like Gait

19

Misshaped hands

56

Third-degree burn Family history: Arthritis , Kidney Disease, Bone Cancer

Relevance Mapping and Disambiguation

Case Re-annotation

Patient ‘Is a’ Female. Patient has normal childhood Patient ‘Has’ Blonde Hair Patient ‘became’ Crippled Patient ‘is’ Scandinavian . . Disease ‘Affects’ Patient Disease ‘Is a’ Degenerative Disorder Disease ‘Is a’ Neuromuscular Disease . . . . . Disease ‘Affects’ Child Disease ‘Affects’ Women Disease ‘Affects’ Gait Disease ‘Causes’ Standing Pain Disease ‘Co-exists with’ Distal Muscle Weakness

Google Search

Step 1: Automatic Case-Context Generation

Step 2: Reasoning w ith N-ary associations

Neuromuscular disease - Distal muscle weakness Gait - Distal muscle weakness Gait abnormality - Falls frequently

Step 3: Hypothesis from Random Walks

Symptom Disease

Case vs. Control Random Walk

Hypothesis / Results – Charcot Marie Tooth Not Polio as in Art/History Books

Hypothesis Probability Hereditary Motor and Sensory Neuropathies

Category

Charcot Marie Tooth Disease

1

Welander Distal Myopathy (common in Sweden)

2

Fasciitis Plantar Talocalcaneal coalition Cerebellar atrophy Friedreich Ataxia Hypolipoproteinemia Multi infarct state Neuroleptic Induced Parkinson Quadriplegic spastic cerebral palsy Subcortical vascular encephalopathy

Base probability for random disease : 1e-6

Mining for potential causes and credibility

Blonde_hair CAUSES (Rev) Tyrosinase_related _protein_1 COEXISTS WITH PER2_protei n__mammali an COEXIST S WITH (Rev) MTMR2 ASSOCIATED WITH Charcot_Marie_Toot h_Disease

Evaluating relevance to weak observations

SCANDINAVIAN PART OF (Rev) 9p21 PART_OF 11q22 CAUSES Charcot_Marie_Tooth_Disease

Opening up previous year cases…

Character Diagnosis (Expert) ORiGAMI Hypothesis

2014 – Oliver Cromwell Malaria, Typhoid, Salmonella infection Malaria, Sepsis, Urinary Tract Infection 2006 – Booker T. Washington Nephrosclerosis, Hypertensive cardiomyopathy Acute congestive heart failure, atrial flutter 2004 – Schliemann Temporal lobe abscess/ exostoses of the external auditory canal, Post-operative meningitis Actinomycotic brain abscess, Enterovirus Infection 2002 – Herod Uremia complicated by Fournier gangrene, Generalized atherosclerosis/hypertension Fournier Gangrene, Deep Vein Thrombosis, Pulmonary Edema 2001 - Claudius Congenital dystonia/Amanita mushroom poisoning, Atherosclerosis Sclerosing lipogranuloma, Acute toxic hepatitis 2000 – Mozart Acute rheumatic fever, post- streptococcus equi- glomerulonephritis Fever disorder, Legionnaires disease 2003 – Florence Nightingale Bipolar disorder, PTSD, (Heart failure/old age) Vascular dementia, atrial flutter, high pressure neurological syndrome Top results listed….

Lessons Learned

• Positive Bias in Literature

– Not enough negations (elimination) in literature

• Fusion of Data (Statistics) and Meta-Data (Text)

– Quality of publication, size of control group, etc.

• Disambiguation and resolution with hierarchies in

terminology

– Can be handled with advanced computing architectures

• Mapping of Spoken-English to Medical-Speak

– Solvable in the near future with “deep learning” techniques that translate languages today.

Major Advances in Non-Medical Artificial Intelligence

Nature February 2015 Groundbreaking Article Closest So Far to AI for Radiology?

150

The Atlantic March 28, 2016 “Go” Was Called the “Holy Grail” of AI Not Achievable for Another Decade

How Google’s AlphaGo Beat Lee Sedol

• Most major South Korean television networks are

carrying the game. In China, 60 million people are tuning in

• A few hundred members of the press are in adjacent

rooms, watching the game alongside expert commentators

• Potential board positions:
• 208,168,199,381,979,984,699,478,633,344,862,770,2

86,522,453,884,530,548,425,639,456,820,927,419,61 2,738,015,378,525,648,451,698,519,643,907,259,916, 015,628,128,546,089,888,314,427,129,715,319,317,5 57,736,620,397,247,064,840,935 more than atoms in the Universe

Go

• Before the match, Lee claimed that the challenge wasn’t

whether he would beat AlphaGo, but whether it would be 5-0

• r 4-1
• Other Korean players stated that it was the easiest million

dollars a top level player could make

• Lee goes on to lose Game 1, resigning after 186 moves
• In game 2 AlphaGo plays a move 37 after which Lee walks
• ut of the room, he resigns after 211 moves
• After losing game 3, Lee apologizes to the entire world, “I

apologize for being able to satisfy a lot of people’s expectations”

• Lee went on to win game 4 with a “hand of God” move at

turn 78 and then lose game 5 using the same strategy

Does Artificial Intelligence Just Emerge With Enough Speed and Memory?

• When Mike was installed in Luna, he was pure thinkum, a flexible
• logic--"High-Optional, Logical, Multi-Evaluating Supervisor, Mark IV,
• Mod. L"--a HOLMES FOUR. He computed ballistics for pilotless freighters
• and controlled their catapult. This kept him busy less than one percent
• f time and Luna Authority never believed in idle hands. They kept
• hooking hardware into him--decision-action boxes to let him boss other
• computers, bank on bank of additional memories, more banks of
• associational neural nets, another tubful of twelve-digit random numbers,
• a greatly augmented temporary memory. Human brain has around ten-to-the tenth neurons. By third year

Mike had better than one and a half times that number of neuristors.

• And woke up.
• He winked lights at me. "Hello, Man."
• "What do you know?"
• He hesitated. I know--machines don't hesitate. But remember, Mike was designed to operate on incomplete
• data. Lately he had reprogrammed himself to put emphasis on words; his hesitations were dramatic. Maybe

he spent pauses stirring random numbers to see how they matched his

• Memories.
• "'In the beginning,'" Mike intoned, "God created the heaven and the earth. And the earth was without form,

and void; and darkness was upon

• the face of the deep. And--'"
• "Hold it!" I said. "Cancel. Run everything back to zero.”

Can We Really Trust Deep Learning Algorithms to Drive or Practice Medicine? How Do We Debug Them?

But Could AI Ever Be Creative? If equal to human w hich human? Prehistoric Man Me? Tanveer, Khan, and Rasu?

Wired Magazine

Conclusion Reasons Why Radiologists Won’t be Replaced Any Time Soon

• There are tens of thousands of algorithms that

have been developed for image analysis and decision support over the past 30 years and for the most part none are in clinical practice

• In order to replace a radiologist, someone would

have to find the best of these and consolidate them into a package that could work independently (unsupervised) for image review but these are written in different “languages” with different assumptions about the images

• Each of these algorithms is generally super

narrow, so in order to replace a radiologist, one would have to have a general portfolio that did everything all specialists do currently

• The work on computer vision recognizing water bottles

in an image database is fundamentally different from diagnostic images including the fact that the images are 24 bit color (8 bit) and that there is no algorithm or methodology that is comparable for these image challenges for diagnostic radiology

• Assuming you had all of these

available and somehow integrated you would then have to start getting FDA approval which could take another 30 years for each and every one given resources and rate of approval of software

• Let’s assume that you actually

discovered/created a program so “smart” that it could read textbooks and journal articles and review prior images on PACS and reports in the EMR and that it actually was better than any subspecialty radiologist at all tasks (far fetched from today’s reality)

• If so, then how would you test it to make

sure it had knowledge in all of those areas satisfactorily

Will Computers Replace Radiologists Soon?

• How long would it take to complete those tests? Years?

Decades? We barely know how to test humans in these areas.

• How long would it take to get the datasets ready to train

these

• How long would it take to get FDA approval for these
• What would be the medico-legal issues associated with

implementation of these?

• How long would it take people to purchase these and

what would they cost given the cost of development?

• Much more likely incremental changes that will be

informed by machine learning

Incredibly Exciting Potential for Machine Learning in Medicine and Diagnostic Imaging

• Intelligent screening criteria for mammography, lung cancer, and other

cancers including genomic/liquid biopsy data and other lab info

• Patients at risk for contrast
• Automatic protocoling of studies
• Smart PACS hanging protocols and synchronization protocols
• Smart transfer of findings from workstation to speech recognition
• Assessment of patients at high risk to have positive findings (or low

risk)

• Communication and tracking of findings
• Multiparametric analysis across multiple modalities
• Improved departmental efficiency with decreased waiting times
• Dose optimization
• Quality improvement in scanning

• So I’m here to tell all of the “worthless

protoplasm” that are radiologists and the rest of you human beings that you can continue to ingest food, reproduce, and create waste products without fear of being replaced by computers, at least in radiology, any time soon!

170

And Finally, It’s Not Easy To Tw eet Like a Human: Microsoft Bot

171

172

173

Conclusions: So João

• Finish your residency
• Lots of work to do before we have a general

ML, DL, AI program that can learn radiology conceptually, rapidly, like radiology resident

• Loads of low hanging fruit for machine

learning techniques that we should be pursuing much more aggressively now

Humans Have Been Around 1 minute 17 seconds - Computers closer to a

• Millisecond. Who Know s What Will

Happen in the Next Microsecond?!

Summer Reading?

176

Coming Events

• SIIM AI Conference and potential

role of SIIM with Dr. Brad Erickson

• Debate at RSNA about whether

radiologists will be replaced in 20 years also Dr. Bradley Erickson

Eliot Siegel, MD, FSIIM, FACR Professor of Radiology University of Maryland School of Medicine Chief Imaging Services, VA Maryland Healthcare System