基于敏感度分析的后训练式对话摘要混合精度量化

doi:10.13229/j.cnki.jdxbgxb.20240809

摘要/Abstract

摘要：

本文提出了一种后训练对话摘要模型上层级混合精度整数量化方法，采用较为精确的基于增广Hessian矩阵方法评估层敏感度，考虑层间相关性的同时可以有效保留层内信息。由于对话摘要模型激活中存在大量离群值，直接对其量化会导致模型精度大幅下降。对此，本文又提出了一种先平滑，后混合精度量化的方法。首先使用平滑因子将激活离群值平移到权重，再根据权重和激活根据敏感度评估得分进行混合精度量化。该方法可以将模型由16 bit或32bit浮点数量化为4~16 bit混合精度，将模型存储需求降低，同时加快推理速度。在基准数据集SAMSum上，与经典基线系统比较取得了标志性性能提高，几乎和不量化模型性能持平。

关键词: 敏感度, 混合精度, 离群值, 对话式摘要

Abstract:

This paper proposes a mixed precision integer quantization method for the layer of a trained dialogue summarization model， which uses a more accurate method based on augmented Hessian matrix to evaluate layer sensitivity， considering inter-layer correlation while effectively preserving intra-layer information. Due to the large number of outliers in the conversation summarization model activation， quantifying them directly will result in significant decrease in model accuracy. In this paper， we propose a method of smoothing first and then mixed precision quantization. Activation outliers are first translated to weights using smoothing factors， and then weights and activations are quantified with mixed precision based on sensitivity assessment scores. This method can quantize the model from 16 bit or 32 bit floating-point to 4-16 bit mixed precision， which reduces the storage requirement of the model and accelerates the inference speed. On the benchmark dataset SAMSum， a significant performance improvement is achieved compared with the classical baseline system， which is almost the same as the performance of the non-quantized model.

Key words: sensitivity, mixed precision, outliers, conversation summarization

中图分类号:

TP312

刘宇鹏,张禹豪,孟鑫. 基于敏感度分析的后训练式对话摘要混合精度量化[J]. 吉林大学学报(工学版), 2026, 56(3): 793-801.

Yu-peng LIU,Yu-hao ZHANG,Xin MENG. Mixed-precision quantization of post-trained conversation summarization based on sensitivity analysis[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(3): 793-801.

图/表 11

图1

图2

图3

图4

表1

表2

图5

图6

图7

表3

表4

参考文献 28

[1]	Dettmers T, Lewis M, Belkada Y, et al. GPT3. int8 (): 8-bit matrix multiplication for transformers at scale[J]. Advances in Neural Information Processing Systems,2022,35: 30318-30332.
[2]	Zafrir O, Boudoukh G, Izsak P, et al. Q8BERT: Quantized 8bit bert[C]∥Proceedings of the Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition, Vancouver, Canada, 2019: 36-39.
[3]	Bai H L, Hou L, Shang L F, et al. Towards efficient post-training quantization of pre-trained language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 1405-1418.
[4]	Li Y H, Gong R H, Tan X, et al. BRECQ: pushing the limit of post-training quantization by block reconstruction[C]∥International Conference on Learning Representations, Vienna, Austria, 2021, 2: 1-17.
[5]	Wang K, Liu Z, Lin Y, et al. HAQ: hardware-aware automated quantization with mixed precision[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 8612-8620.
[6]	Schaefer C J S, Joshi S, Li S,et al. Edge inference with fully differentiable quantized mixed precision neural networks[C]∥Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, USA, 2024: 8460-8469.
[7]	Park E, Yoo S. PROFIT: a novel training method for sub-4-bit mobilenet models[C]∥Proceedings of the Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, 2020: 430-446.
[8]	Yao Z W, Dong Z, Zheng Z C, et al. HAWQ-V3: dyadic neural network quantization[C]∥Proceedings of the International Conference on Machine Learning, Shenzhen, China, 2021: 11875-11886.
[9]	Nahshan Y, Chmiel B, Baskin C, et al. Loss aware post-training quantization[J]. Machine Learning, 2021, 110(11-12): 3245-3262.
[10]	Nagel M, Amjad R A, Van B M, et al. Up or down? Adaptive rounding for post-training quantization[C]∥Proceedings of the International Conference on Machine Learning, Vienna, Austria, 2020: 7197-7206.
[11]	Yao Z W, Aminabadi R Y, Zhang M J, et al. ZeroQuant: efficient and affordable post-training quantization for large-scale transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 27168-27183.
[12]	Cai Y, Yao Z, Dong Z, et al. Zeroq: a novel zero shot quantization framework[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 13169-13178.
[13]	Hubara I, Nahshan Y, Hanani Y, et al. Accurate post training quantization with small calibration sets[C]∥Proceedings of the International Conference on Machine Learning, Online, 2021: 4466-4475.
[14]	Frantar E, Alistarh D. Optimal brain compression: a framework for accurate post-training quantization and pruning[J]. Advances in Neural Information Processing Systems, 2022, 35: 4475-4488.
[15]	Demidovskij A, Smirnov E.Effective post-training quantization of neural networks for inference on low power neural accelerator[C]∥International Joint Conference on Neural Networks, Glasgow, UK, 2020: 1-7.
[16]	Zandonati B, Pol A A, Pierini M, et al. Fit: a metric for model sensitivity[C]∥Proceedings of ICLR, Addis Ababa, Ethiopia, 2022: 1-20.
[17]	Zheng D, Liu Y, Li L. Leveraging inter-layer dependency for post-training quantization[J]. Advances in Neural Information Processing Systems, 2022, 35: 6666-6679.
[18]	Wei X Y, Zhang Y C, Zhang X G, et al. Outlier suppression: pushing the limit of low-bit transformer language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 17402-17414.
[19]	Bondarenko Y, Nagel M, Blankevoort T. Understanding and overcoming the challenges of efficient transformer quantization[C]∥Proceedings of the Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 2021: 7947-7969.
[20]	Zeng A, Liu X, Du Z, et al. GLM-130B: an open bilingual pre-trained model[C]∥Proceedings of ICLR, Kigali, Rwanda, 2023: 1-56.
[21]	Wu H, Judd P, Zhang X, et al. Integer quantization for deep learning inference: principles and empirical evaluation[J/OL].[2024-07-06]. arXiv Preprint arXiv:.
[22]	Fabbri A, Rahman F, Rizvi I, et al. ConvoSumm: conversation summarization benchmark and improved abstractive summarization with argument mining[C]∥Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Bangkok, Thailand, 2021: 6866-6880.
[23]	Lewis M, Liu Y, Goyal N, et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]∥Proceedings of the 58th Annual Meeting of the Association for Computational Linguistic, Florence, Italy, 2019: 7871-7880.
[24]	Liu Y P, Zhang Y H, Liu G. A conversation summary generation method for medical consultations[P]. China Patent: ZL115964475A, 2023-04-14.
[25]	Shen S, Dong Z, Ye J Y, et al. Q-BERT: hessian based ultra low precision quantization of bert[C]∥Proceedings of the AAAI Conference on Artificial Intelligence, New York, USA, 2020: 8815-8821.
[26]	Wei X Y, Gong R H, Li Y H, et al. QDrop: randomly dropping quantization for extremely low-bit post-training quantization[C]∥Proceedings of ICLR, Online, 2022: 1-19.
[27]	Jeon Y, Lee C, Cho E, et al. Mr. BIQ: post-training non-uniform quantization based on minimizing the reconstruction error[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 12319-12328.
[28]	Goo C W, Chen Y N. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts[C]∥Proceedings of the IEEE Spoken Language Technology Workshop, Athens, Greece, 2018: 735-742.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

模型名称	介绍
LBART^［22］	基于BART^［23］的预训练模型，在对话摘要任务上微调模型，共1亿4千万参数
DSMC ^［24］	基于Transformer的非预训练模型，参数量约7千万

量化方法	主要思想	量化位宽（A为激活，W为权重）
QBert^［25］	基于分组的量化方案，使用二阶泰勒展开来评估权重	A8W8
MREM-P^［3］	将模型分成多个模块，采用并行策略同时为每个模块最小化量化带来的重构误差	A8W8
BRECQ^［4］	利用神经网络的基本构建块，逐个进行重构，同时在跨层依赖和泛化误差之间取得了良好平衡	A8W8
QDrop^［26］	通过随机丢弃来实现激活量化，并首次将激活的量化推进到2 bit	A4W8
MrBiQ^［27］	利用多级二值化表示权重并允许不同数据表示激活	A8W8
AdaQuant^［13］	通过分别优化每个层或块的参数，并使用校准集来设置激活动态范围，以最小化量化误差	A4W8

评分	4 bit	8 bit	16 bit
99.99%	3	112	78
98%	18	174	1

方法	LBART 99%		LBART 99.9%
方法	8 bit	4 bit	8 bit	4 bit
Hessian	6	6	6	7
AugHessian	6	8	6	7