前后文记忆矩阵引导的胸部放射影像报告生成模型

doi:10.13229/j.cnki.jdxbgxb.20230341

摘要/Abstract

摘要：

针对现有的放射影像报告生成方法主要关注前文信息在报告生成过程中的指导作用，忽略了后文信息在报告描述生成中的作用，存在语义信息不完整、关键信息缺失等问题，提出了一种前后文记忆矩阵引导的胸部放射影像报告生成模型（RadRG）。为了在生成过程中获得更丰富的语义信息，提出记忆矩阵生成模块（MA），使用记忆矩阵记录报告的模式信息和语义信息，并使用门控单元对其进行调控，以防止训练过程中出现梯度消失或信息爆炸。将本文模型在IU X-Ray和MIMIC-CXR公开数据集上进行了测试，实验结果表明：与现有主流模型对比，本文模型在BIEU指标达到了2.6%的提升。

关键词: 医学报告生成, 前后文记忆矩阵, 门控单元, 跨模态

Abstract:

Aiming at the existing radiographic report generation methods that mainly focus on the guiding role of the preceding information in the report generation process and ignore the role of the following information in the report description generation， which has the problems of incomplete semantic information and missing key information， anterior-posterior memory matrix model for chest radiology image report generation（RadRG） is proposed. In order to obtain richer semantic information in the generation process， anterior-posterior memory matrix（MA） is proposed， which uses the memory matrix to record the schema information and semantic information of the report， and uses a gating unit to regulate it in order to prevent the gradient from disappearing or the information explosion in the training process. The model in this paper is tested on the IU X-Ray and MIMIC-CXR public datasets， and the experimental results show that the model in this paper achieves a 2.6% improvement in the BIEU metrics compared with the existing mainstream models.

Key words: medical report generation, context memory matrix, gating unit, cross model

中图分类号:

TP391

刘利军,张云峰,黄青松. 前后文记忆矩阵引导的胸部放射影像报告生成模型[J]. 吉林大学学报(工学版), 2024, 54(8): 2355-2363.

Li-jun LIU,Yun-feng ZHANG,Qing-song HUANG. Anterior-posterior memory matrix model for chest radiology image report generation[J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(8): 2355-2363.

图/表 9

图1

图2

图3

图4

图5

表1

实验参数设置"

参数名	参数值	参数说明
lr_ve	$5 × 10 - 5$	视觉特征初始学习率
lr_ed	$1 × 10 - 4$	其他参数学习率
weight_decay	$5 × 10 - 5$	权重的衰减率
batch	10	一次输入网络的数量
m_f	512	记忆矩阵的维度
mma_heads	8	多头注意力的头数
gamma	0.1	学习率的衰减率（IU X-Ray）
gamma	0.8	学习率的衰减率（MIMIC-CXR）
epoch	100	迭代次数（IU X-Ray）
epoch	50	迭代次数（MIMIC-CXR）

表1

表2

表3

表4

参考文献 27

1	刘桂霞, 田郁欣, 王涛, 等. 基于双输入3D卷积神经网络的胰腺分割算法[J]. 吉林大学学报: 工学版, 2023, 53(12): 3565-3572.
	Liu Gui-xia, Tian Yu-xin, Wang Tao,et al. Pancreas segmentation algorithm based on dual input 3D convolutional neural network[J]. Journal of Jilin University (Engineering and Technology Edition), 2023, 53(12): 3565-3572.
2	Kaur N, Mittal A, Singh G. Methods for automatic generation of radiological reports of chest radiographs: a comprehensive survey[J]. Multimedia Tools and Applications, 2022, 81(10): 13409-13439.
3	Li M, Liu R, Wang F, et al. Auxiliary signal-guided knowledge encoder-decoder for medical report generation[J]. World Wide Web, 2023, 26(1): 253-270.
4	Li C Y, Liang X, Hu Z, et al. Knowledge-driven encode, retrieve, paraphrase for medical image report generation[DB/OL].[2023-03-22]..
5	Wang Z, Zhou L, Wang L, et al. A self-boosting framework for automated radiographic report generation[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Nashville, USA,2021: 2433-2442.
6	Messina P, Pino P, Parra D, et al. A survey on deep learning and explainability for automatic report generation from medical images[J]. ACM Computing Surveys (CSUR), 2022, 54(10s): 1-40.
7	Kuo C W, Kira Z. Beyond a pre-trained object detector: cross-modal textual and visual context for image captioning[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,New Orleans,USA,2022: 17969-17979.
8	Luo Y, Ji J, Sun X, et al. Dual-level collaborative transformer for image captioning[DB/OL].[2023-03-22]..
9	Lu J, Xiong C, Parikh D, et al. Knowing when to look: adaptive attention via a visual sentinel for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Honolulu, USA, 2017: 375-383.
10	Diao H, Zhang Y, Ma L, et al. Similarity reasoning and filtration for image-text matching[DB/OL].[2023-03-23]..
11	Wang L, Li Y, Lazebnik S. Learning deep structure-preserving image-text embeddings[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA,2016: 5005-5013.
12	Wang H, Zhang Y, Ji Z, et al. Consensus-aware visual-semantic embedding for image-text matching[C]∥Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 2020: 18-34.
13	Kim W, Son B, Kim I. Vilt: Vision-and-language transformer without convolution or region supervision[DB/OL].[2023-03-23].∥.
14	Srinivasan P, Thapar D, Bhavsar A, et al. Hierarchical X-ray report generation via pathology tags and multi head attention[C]∥Proceedings of the Asian Conference on Computer Vision. Berlin:Springer, 2020: 600-616.
15	Chen Z, Song Y, Chang T H, et al. Generating radiology reports via memory-driven transformer[DB/OL].[2023-03-23]..
16	Liu F, You C, Wu X, et al. Auto-encoding knowledge graph for unsupervised medical report generation[J]. Advances in Neural Information Processing Systems, 2021, 34: 16266-16279.
17	Vu Y N T, Wang R, Balachandar N, et al. Medaug: contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation[DB/OL].[2023-03-24]..
18	Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C]∥Proceedings of the European conference on computer vision (ECCV). Berlin: Springer,2018: 3-19.
19	肖明尧, 李雄飞, 朱芮. 基于NSST域像素相关分析的医学图像融合[J]. 吉林大学学报:工学版, 2023, 53(9): 2640-2648.
	Xiao Ming-yao, Li Xiong-fei, Zhu Rui. Medical image fusion based on pixel correlation analysis in NSST domain[J]. Journal of Jilin University (Engineering and Technology Edition), 2023, 53(9): 2640-2648.
20	Demner-Fushman D, Kohli M D, Rosenman M B, et al. Preparing a collection of radiology examinations for distribution and retrieval[J]. Journal of the American Medical Informatics Association, 2016, 23(2): 304-310.
21	Liu F, Wu X, Ge S, et al. Exploring and distilling posterior and prior knowledge for radiology report generation[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Nashville, USA, 2021: 13753-13762.
22	Chen Z, Shen Y, Song Y, et al. Cross-modal memory networks for radiology report generation[DB/OL].[2023-03-24]..
23	Yang S, Wu X, Ge S, et al. Radiology report generation with a learned knowledge base and multi-modal alignment[J]. Medical Image Analysis, 2023, 86: No.102798.
24	Jing B, Xie P, Xing E. On the automatic generation of medical imaging reports[DB/OL].[2023-03-24]..
25	Li Y, Liang X, Hu Z, et al. Hybrid retrieval-generation reinforced agent for medical image report generation[DB/OL].[2023-03-24]..
26	Jing B, Wang Z, Xing E. Show, describe and conclude : on exploiting the structure information of chest X-ray reports[DB/OL].[2023-03-24]..
27	Anderson P, He X, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Salt Lake City, USA,2018: 6077-6086.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

数据集	模型	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE
IU X-Ray	ADAATT^［9］	0.220	0.127	0.089	0.068	—	0.308
	CoATT^［24］	0.455	0.288	0.205	0.154	—	0.369
	HRGR^［25］	0.438	0.298	0.208	0.151	—	0.322
	CMAS-RL^［26］	0.464	0.301	0.210	0.154	—	0.362
	CMN^［22］	0.475	0.309	0.222	0.170	0.181	0.375
	PPKED^［21］	0.480	0.309	0.224	0.168	0.183	0.376
	V-TL^［23］	0.481	0.311	0.224	0.169	—	0.380
MIMIC-CXR	Ours	0.496	0.325	0.237	0.179	0.192	0.391
	ADAATT	0.269	0.145	0.084	0.058	0.098	0.226
	TOPDOWN^［27］	0.277	0.155	0.090	0.052	0.108	0.237
	CMN	0.278	0.165	0.104	0.065	0.118	0.247
	PPKED	0.282	0.168	0.107	0.064	0.116	0.247
	V-TL	0.283	0.169	0.107	0.068	—	0.249
	Ours	0.294	0.176	0.116	0.082	0.133	0.256

数据集	模型	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE
IU X-Ray	BASE	0.470	0.304	0.219	0.165	0.187	0.371
	BASE+Attention	0.476	0.300	0.215	0.169	0.186	0.371
	BASE+MA	0.477	0.304	0.224	0.171	0.191	0.383
	BASE+MA+MCLN	0.481	0.311	0.223	0.173	0.193	0.387
	Ours	0.491	0.319	0.231	0.176	0.192	0.391
MIMIC-CXR	BASE	0.274	0.161	0.104	0.060	0.117	0.246
	BASE+Attention	0.273	0.161	0.104	0.072	0.115	0.252
	BASE+MA	0.283	0.163	0.108	0.073	0.110	0.251
	BASE+MA+MLCN	0.285	0.165	0.109	0.072	0.111	0.250
	Ours	0.286	0.168	0.108	0.073	0.123	0.251

数据集	真实值	生成文本
IU X-Ray	The heart is enlarged. There is pulmonary vascular congestion with diffusely increased interstitial and mild patchy airspace opacities. The distribution XXXX pulmonary edema. There is no pneumothorax or large pleural effusion. There are no acute bony findings	there is stable cardiomegaly with xxxx pulmonary vascular congestion and probable mild interstitial edema. there are bilateral pleural effusions with bibasilar airspace disease right greater than left. there is no pneumothorax. there are no acute bony findings
IU X-Ray	The lungs are clear. there is no pleural effusion or pneumothorax. the heart and mediastinum are normal. the skeletal structures are normal.	cardiac and mediastinal contours are within normal limits. there is no pleural effusion or pneumothorax. the lungs are clear. bony structures are intact
MIMIC-CXR	moderate cardiomegaly is re-demonstrated. the aorta is tortuous. pulmonary vasculature is not engorged. patchy opacities are seen in the left lung base potentially atelectasis but infection or aspiration cannot be excluded. streaky atelectasis is also demonstrated in the left lung base. no pleural effusion or pneumothorax is present. no acute osseous abnormality is visualized	lung volumes are low. heart size is mildly enlarged. mediastinal and hilar contours are unremarkable. pulmonary vasculature is not engorged. streaky opacities in the lung bases likely reflect atelectasis. no focal consolidation pleural effusion or pneumothorax is present. no acute osseous abnormality is detected.