VetTag: improving automated veterinary diagnosis coding via - - PowerPoint PPT Presentation

VetTag: improving automated veterinary diagnosis coding via large-scale language modeling

Published in npj (Nature) Digital Medicine

Yuhui Zhang, Allen Nie, James Zou

Introduction

Large-scale veterinary clinical records can become a powerful resource for patient care and research. However, clinicians lack the time and resource to annotate patient records with standard medical diagnostic codes.

• Veterinary medicine domain lacks coding infrastructure and standardized nomenclatures

across medical institutions Why is automatically annotating clinical notes important for medicine and care?

• DeepTag automatically infers 41 high-level disease codes but suffers from high performance drop

when applied across hospitals. (Nie, 2018)

• Identifying clinical cohorts of veterinary patients on a large scale for clinical research (Baraban,

2014).

• Animals have important translational impact on the study of human disease (Kol, 2015).
• Spontaneous models of disease in companion animals are used in drug development pipelines

(Hernandez, 2018).

Introduction

Jem is a 10 year old male castrated hound mix that was presented for continuation of chemotherapy for previously diagnosed B-cell multicentric lymphoma. Jem was started on CHOP chemotherapy last week and has been doing very well since receiving doxorubicin. The owners have noted his lymph nodes have gotten much smaller. He has some loose stool, yet improved with metronidazole. Current medications include prednisolone. Assessment: Jem is in a strong partial remission based on today’s physical exam. He is also doing very well since starting chemotherapy. A CBC today was unremarkable and adequate for

• chemotherapy. She was dispensed oral cyclophosphamide and furosemide that the owners

were instructed to give at home. Expert annotated diseases: Disorder of hematopoietic cell proliferation, Neoplasm and/or hamartoma

CSU: Colorado State University PP: private practice PSVG: private specialty veterinary group

Dataset

SNOMED-CT Codes: We use 4577 SNOMED-CT codes

Dataset

Model Overview

Pretrain Auxiliary Pretrain Auxiliary Pretrain Auxiliary

Encoder Classifier Generator

Binary Cross Entropy Loss Predicted PSVG Text

̂ x1, ̂ x2, . . . , ̂ xT

Predicted CSU Text Predicted Diseases PSVG Text

x1, x2, . . . , xT

CSU Text

x1, x2, . . . , xT

Negative Log Likelihood Loss

̂ x1, ̂ x2, . . . , ̂ xT

Negative Log Likelihood Loss

Unsupervised Language Modeling Supervised Learning

Disease √ × × √ √ √ Chronic Congenital Infectious × Bacterial Toxic Botryomycosis

Subtree Prediction

stack n times

u1 u2 u3 u4

Multi- Head Attenti

• n

Feed Forwa rd

}

h1 h2 h3 h4

Transformer

Label Name

Language Modeling

Given a sequence X = ($%, $', … , $)), we model the sequence probability as + , = Π./%

)

+ $. $%, $', … , $.0%). We use hidden state ℎ.0% to estimate + $. = $ $%, $', … , $.0%) and optimize on cross entropy loss. In our experiments, we use language modeling objective in two parts of our system: 1) pretrain encoder’s parameters; 2) serve as an auxiliary task during training of the classifier.

Language Modeling

• +P uses language modeling objective during unsupervised pretraining on

PSVG

• +A uses language modeling objective in addition to classification during

supervised learning on CSU

• +W uses Word2Vec trained embedding on PSVG

Hierarchical Training

Diagnosis codes at a higher depth in the SNOMED hierarchy tend to be more specific, and thus fewer positive cases can be found for it.

Disease √ × × √ √ √ Chronic Congenital Infectious × Bacterial Malformation Botryomycosis × Anemia

Botryomycosis Anemia Bacterial Malformation Chronic Infectious Congenital Disease

√ √ √ √ × × × × .9 .8 .7 .8 .1 .2 .0 .6 1 1 1 1 1 1 1 1 1 Labels Probabilities Predictions Masked Predictions Subtree of SNOMED-CT Codes 1 2 3 4 1 2 3 4

{

{

{

{

Depth

{

{

{

{

Depth

Hierarchical Training

We use hierarchical training to address this imbalance problem.

Interpretation

We implement a saliency-based interpretation method, which is computed as the gradient of the predicted probability with respect to the input word (gradient × input). We compute the frequency of words that have score ≥ 0.2 (threshold chosen to select on average 3% words per note), use MetaMap dictionary as a filter to extract medical relevant terms.

deacon , a 2 year old male neutered boxer , presented to csu dermatology service for a recheck of allergic dermatitis . it was reported that deacon 's pruritus is relatively seasonal , worse in the spring and summer , but does not completely go away in the winter . deacon 's main problem areas were his paws and his muzzle . melissa recently had been adding nettles to deacon 's food and saw a noticeable decrease in pruritus for a few days but not so much anymore . Hypersensitivity condition

Interpretation

Top 10 words in the MetaMap medical dictionary that the model most strongly associates with the phenotype

Discussion

• (Overall) We build a system to predict 4577 SNOMED codes, and propose cross-

hospital generalization problem.

• (Unstructured Text) We propose a language model framework to leverage a massive

amount of unlabeled clinical notes, demonstrating that this type of unsupervised learning is crucial in improving the performance and robustness of the diagnosis coding model

• (Structured Hierarchy) We analyze the impact of depth (specificity) of a diagnosis to

the performance of the model, and demonstrate that hierarchical training is significantly better than the standard multi-label prediction scheme

• (Interpretation) We provide a saliency method to explain our model by visualizing the

words in the clinical note that most significantly influences model’s prediction

Thank you