VetTag: improving automated veterinary diagnosis coding via large-scale language modeling Published in npj (Nature) Digital Medicine Yuhui Zhang, Allen Nie, James Zou
Introduction Large-scale veterinary clinical records can become a powerful resource for patient care and research. However, clinicians lack the time and resource to annotate patient records with standard medical diagnostic codes. ● Veterinary medicine domain lacks coding infrastructure and standardized nomenclatures across medical institutions Why is automatically annotating clinical notes important for medicine and care? ● DeepTag automatically infers 41 high-level disease codes but suffers from high performance drop when applied across hospitals. (Nie, 2018) ● Identifying clinical cohorts of veterinary patients on a large scale for clinical research (Baraban, 2014). ● Animals have important translational impact on the study of human disease (Kol, 2015). ● Spontaneous models of disease in companion animals are used in drug development pipelines (Hernandez, 2018).
Introduction Jem is a 10 year old male castrated hound mix that was presented for continuation of chemotherapy for previously diagnosed B-cell multicentric lymphoma . Jem was started on CHOP chemotherapy last week and has been doing very well since receiving doxorubicin . The owners have noted his lymph nodes have gotten much smaller. He has some loose stool, yet improved with metronidazole. Current medications include prednisolone . Assessment: Jem is in a strong partial remission based on today’s physical exam. He is also doing very well since starting chemotherapy . A CBC today was unremarkable and adequate for chemotherapy . She was dispensed oral cyclophosphamide and furosemide that the owners were instructed to give at home. Expert annotated diseases : Disorder of hematopoietic cell proliferation, Neoplasm and/or hamartoma
Dataset CSU: Colorado State University PP: private practice PSVG: private specialty veterinary group
Dataset SNOMED-CT Codes: We use 4577 SNOMED-CT codes
̂ ̂ Model Overview Predicted PSVG Text x 1 , ̂ x 2 , . . . , ̂ x T Negative Log Likelihood Pretrain Pretrain Loss PSVG Text x 1 , x 2 , . . . , x T Generator Pretrain Negative Auxiliary Auxiliary Transformer Log Auxiliary Likelihood u 1 h 1 Loss Multi- u 2 Feed h 2 Unsupervised Head Forwa x 1 , ̂ x 2 , . . . , ̂ x T Attenti Language u 3 h 3 rd on Predicted CSU Text Modeling u 4 h 4 } stack n times Label Name Disease √ Encoder Prediction Congenital Infectious Chronic × × √ x 1 , x 2 , . . . , x T Binary Cross CSU Text Subtree Classifier Entropy Bacterial Toxic × √ Loss Botryomycosis √ Supervised Predicted Diseases Learning
Language Modeling Given a sequence X = ($ % , $ ' , … , $ ) ), we model the sequence probability as ) + , = Π ./% + $ . $ % , $ ' , … , $ .0% ) . We use hidden state ℎ .0% to estimate + $ . = $ $ % , $ ' , … , $ .0% ) and optimize on cross entropy loss. In our experiments, we use language modeling objective in two parts of our system: 1) pretrain encoder’s parameters; 2) serve as an auxiliary task during training of the classifier.
Language Modeling ● +P uses language modeling objective during unsupervised pretraining on PSVG ● +A uses language modeling objective in addition to classification during supervised learning on CSU ● +W uses Word2Vec trained embedding on PSVG
Recommend
More recommend
