藏语文本标准化方法

doi:10.13229/j.cnki.jdxbgxb.20230098

摘要/Abstract

摘要：

针对现代藏语文本表征形式复杂多样且不规范，影响语音合成系统的性能问题，提出了具有易于维护及可扩展性特点的藏语文本标准化方法。首先，对藏文标记符号和来自其他语言的非藏文特殊符号在藏语文本中的不同表现形式进行了深度解析，并通过不同特征对特殊符号进行了分类；其次，根据归纳的不同类型，分别建立起了15种特殊符号转化为藏语的书写规则；最后，以13 490个句子作为实验数据，通过藏语字音转换测试识别并检测文本中特殊符号和藏文音节的有效性，采用规则匹配的方法对含有特殊符号的句子进行标准化处理。实验结果表明：标准化之前藏语音素转写的遗漏率高达4.69%，而经过标准化之后音素转写的遗漏率降低到0.01%，其藏语文本标准化准确率达99%。

关键词: 计算机应用技术, 藏语文本分析, 文本标准化, 语音合成, 特殊符号, 字音转换

Abstract:

In view of the complexity and nonstandard representation of modern Tibetan text， which affects the performance of speech synthesis system， this paper proposes a Tibetan text standardization method with the characteristics of easy maintenance and scalability. Firstly，a deep analysis was conducted on the different manifestations of Tibetan marker symbols and non Tibetan special symbols from other languages in Tibetan texts， and the special symbols were classified based on different features. Secondly， according to the different types of induction， the writing rules for converting 15 special symbols into Tibetan language were respectively established. Finally， using 13 490 sentences as the experimental data， the effectiveness of special symbols and Tibetan syllables in the text is identified and tested through the Tibetan grapheme-to-phoneme conversion test， and the sentences containing special symbols are standardized by the method of rule matching. The experimental results show that the omission rate of Tibetan phoneme transcription before standardization was as high as 4.69%， but after standardization， the omission rate of phoneme transcription was reduced to 0.01%， and the standardization accuracy rate of Tibetan text reached 99%.

Key words: computer application technology, Tibetan text analysis, text normalization, text-to-speech, special symbols, grapheme-to-phoneme

中图分类号:

TP391

拉巴顿珠,扎西多吉,珠杰. 藏语文本标准化方法[J]. 吉林大学学报(工学版), 2024, 54(12): 3577-3588.

Dondrub LHAKPA,Duoji ZHAXI,Jie ZHU. Tibetan text normalization method[J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(12): 3577-3588.

图/表 19

表1

表2

表3

表4

表5

表6

表7

表8

表9

表10

表11

常用关系运算符的规范书写格式"

符号	汉语含义	藏语含义	举例	藏语书写规则
<	小于	????????	$x < y$	x+???+y
>	大于	???????	$x > y$	x+??+y
=	等于	?????	$x = y$	x+?????+y

表11

表12

表13

表14

表15

图1

表16

标准化前藏语音素转写测试结果"

$Y T$	$Y F$	$T T$	$T F$	$O R$ /%
6 295	178	0	132	4.69

表16

表17

标准化后藏语音素转写测试结果"

是否人工介入	$Y T$	$Y F$	$T T$	$T F$	$O R$ /%
是	6 902	0	0	1	0.01
否	6 724	178	0	1	2.59

表17

表18

实验结果实例"

测试句子	标准化之前的测试结果	标准化之后的测试结果
测试句子	特殊符号检测结果（音素转写测试1）	文本标准化处理结果	标准的藏语音素序列文本（音素转写测试2）
???????????650????????????????????????	mxii ttej cxii 650 lxaa nnee lxen jqed gjuw rred	?????????????????????????????????????????????????????	mxii ttej cxii chug gjaa ngaa ccuw lxaa nnee lxen jqed gjuw rred
???1????10????????????????????“110??????????????????”???32??????	ddaa 1 tses 10 nxin nzii gjal yxoj gbii “110 chil zhaf nxin mxow ” ah 32 bvaa yxin	???????????????????????????????????????????????????????????????????????????????????????????????????	ddaa dtah bvow tses ccuw nxin nzii gjal yxoj gbii ccig ccig llad gvor chil zhaf nxin mxow ah ssum ccuw ssow njis bvaa yxin
????????????????????????????????????????????????????????????????????	ggee dqun qqos ppel nzii ccii lxow ???? lxoi ccii ddaa ? bvai tses ?? nxin cxuj	??????????????????????????????????????????????????????????????????????????????????????????????????????	ggee dqun qqos ppel nzii ccii lxow qqig dvoh gguw gjaa ccuw mxed ssum lxoi ccii ddaa xqii bvai tses nxii xxuw nxin cxuj
???????（1990~2022）?????????????????????	kkoh nzii （1990~2022）phar phod dtuw xquf bvaa rred	?????????????????????????????????????????????????????????????????????????????????????????	kkoh nzii qqig dvoh gguw gjaa gguw ccuw nzas nxis dvoh gjaa mxed nxii xxuw zzaa njis phar phod dtuw xquf bvaa rred
“119”??????????????????????????????	“119” ah chaj gkah jquh dtuw nnon mxii qqog	????????????????????????????????????????????	ccig ccig gguw ah chaj gkah jquh dtuw nnon mxii qqog
???????????????-15 ℃??????????	ppal qqer chod tsad -15 ℃ yxod bvaa ztaa	????????????????????????????????????????????????????	ppal qqer chod tsad llad gvor vvog gkii dvuv ccow ngaa yxod bvaa ztaa
?????????1+5=6 ???????????????????????	zzis xqii 1+5=6 llob qquh gkii nzah dton rred	??????????????????????????????????????????????????????????	zzis xqii ccig ddop ngaa tsuj chug llob qquh gkii nzah dton rred
???????????????500.6????	hid tsad lxaa mmis 500.6 dqug	????????????????????????????????????	jhid tsad lxaa mmis ngaa gjaa tseg chug dqug
??????????????：??????????	dtaa dvaa qquw tsod ??：?? rred dqug	???????????????????????????????????????	dtaa dvaa qquw tsod gguw bvaa dtah gvar mxaa ccuw rred dqug
??????????50%?????????????????	chaj bqor 50% phar mxah dtuw dvah yxod	?????????????????????????????????????????	chaj bqor gjaa qqaa ngaa ccuw phar mxah dtuw dvah yxod
????????1984??????????????????????????????????????????NBA????????????????	ccii lxow 1984 lxor ngos ssuw aa rrii lxag zzed bvow lxow ttun tsof NBA nzah ngos ssuw xquf	????????????????????????????????????????????????????????????????????????????????????NBA????????????????	ccii lxow qqig dvoh gguw gjaa gjad ccuw gtaa xqii lxor ngos ssuw aa rrii lxag zzed bvow lxow ttun tsof NBA nzah ngos ssuw xquf
???????????????（1991~2001）????????????????????????	kkoh nzii lxow nkow ??（1991~2001）lhaa ssar tsow phaa gbel phaa rred	?????????????????（??????????????????????????????????????????????????????????????????????）????????????????????????	kkoh nzii lxow nkow ccuw qqig dvoh gguw gjaa gguw ccuw gkow ccig tten nxis dvoh gjaa mxed ccuw mxed ccig lhaa ssar tsow phaa gbel phaa rred

表18

参考文献 17

1	Ren Y, Hu C, Tan X, et al. Fastspeech 2: fast and high-quality end-to-end text to speech[DB/OL].[2023-01-06].
2	王莉莉. 面向特定领域藏语统计参数语音合成的文本分析研究[D]. 兰州: 西北师范大学物理与电子工程学院, 2020.
	Wang Li-li. Text analysis of speech sythesis based on statical parameters of Tibetan language in specific fields[D]. Lanzhou:College of Physics and Electronic Engineering, Northwest Normal University, 2020.
3	张日培. 藏文文语转换系统关键技术研究[D]. 西宁: 青海师范大学计算机学院, 2018.
	Zhang Ri-pei. Research on key technologies of Tibetan text-to-speech system[D]. Xining: College of Computer,Qinghai Normal University, 2018.
4	拉巴顿珠, 欧珠, 祖漪清, 等. 藏语同形异音词的消歧方法研究[J]. 中文信息学报, 2018, 32(7):58-66.
	Lhakpa-Dondrub, Ngodrup, ZU Yi-qing, et al. Disambiguation of polyphonic words in Tibetan[J]. Journal of Chinese Information Processing, 2018, 32(7): 58-66.
5	庄暑楠. 基于深度学习的文本规范化的研究与实现[D]. 长春: 吉林大学计算机科学与技术学院, 2020.
	Zhuang Shu-nan. Research and implementation of text normalization based on deep learning [D]. Changchun: College of Computer Science and Technology,Jilin University, 2020.
6	Tyagi S, Bonafonte A, Lorenzo-Trueba J, et al. Proteno: text normalization with limited data for fast deployment in text to speech systems[DB/OL].[2023-01-08].
7	Tran O T, Bui V T. Neural text normalization in speech-to-text systems with rich features[J]. Applied Artificial Intelligence, 2021, 35(3): 193-205.
8	Zhang H, Sproat R, Ng A H, et al. Neural models of text normalization for speech applications[J]. Computational Linguistics, 2019, 45(2): 293-337.
9	Mansfield C, Sun M, Liu Y, et al. Neural text normalization with subword units[C]∥Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA,2019: 190-196.
10	Massimo L, Tatyana R, Anne G, et al. Encoder-decoder methods for text normalization[J/OL]. [2023-01-12].
11	Dai W L, Song C H, Li X, et al. An end-to-end Chinese text normalization model based on rule-guided flat-lattice transformer[C]∥Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, 2022: 7122-7126.
12	胥桂仙,刘兰寅,张廷,等.基于预训练模型和图神经网络的藏文文本分类研究[J].东北师大学报: 自然科学版,2023,55(1):52-64.
	Xu Gui-xian, Liu Lan-yin, Zhang Ting, et al.Tibetan text classification based on pre-trained model and graph neural network[J]. Joural of Northeast Normal University (Natural Science Edition), 2023,55(1):52-64.
13	艾金勇. 面向信息处理的藏文文本规范化方法研究[J]. 西北师范大学学报: 自然科学版, 2017, 53(2):52-56.
	Ai Jin-yong. Research on normalization method of Tibetan text for information processing[J]. Journal of Northwest Normal University (Natural Science), 2017, 53(2): 52-56.
14	贡保加, 才智杰, 才让卓玛, 等. 一种藏语语音识别中数字文本规范方法[J].高原科学研究, 2022, 6(3): 117-124.
	Gong Bao-jia, Cai Zhi-jie, Cairang-Zhuoma, et al. Study on a method of standardizing digital text in Tibetan speech recognition[J]. Plateau Science Research, 2022, 6(3): 117-124.
15	边巴嘉措. 现代藏语书面语语音结构分析[M]. 北京: 北京民族出版社, 2017.
16	邓戈. 藏语语音研究[M].拉萨: 西藏藏文古籍出版社, 2013.
17	珠杰. 藏文文本自动处理方法研究[M]. 2版.成都:西南交通大学出版社, 2022.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

功能	标记符号	功能	标记符号
起始符	?? ? ? ?	吟诵示意符	? ? ? ?
句末符	? ? ? ?	装饰符号	? ? ? ?
历算占星符	? ? ? ?	标点符号	? ? ? ?

项目	位数
项目	1	10	100	1 000	10 000	100 000	1 000 000	10 000 000	100 000 000
含义	个	十	百	千	万	十万	百万	千万	亿
藏语写法	????	???	????	????	???	???????或????	????????或???	????????或?????	????????

项目	单基数词
项目	0	1	2	3	4	5	6	7	8	9
含义	零	一	二	三	四	五	六	七	八	九
藏语写法	???????	????	????	????	???	??	????	????	?????	???

月份	文本中出现的形式		规范写法
1月	?????1	???1	??????????
2月	?????2	???2	???????????
3月	?????3	???3	???????????
4月	?????4	???4	??????????
5月	?????5	???5	?????????
6月	?????6	???6	???????????
7月	?????7	???7	???????????
8月	?????8	???8	????????????
9月	?????9	???9	??????????
10月	?????10	???10	??????????
11月	?????11	???11	???????????????
12月	?????12	???12	???????????????

序号	电话号码书写格式	含义	藏语书写规则
1	0+手机号	座机打异地号码	从左到右的顺序依次转写
2	+国家代码-地区号码-用户号码	国际电话号码
3	区号+普通号码	国内电话号码
4	3\|4\|5位数的号码	特殊电话号码