Journal of Jilin University(Earth Science Edition) ›› 2019, Vol. 49 ›› Issue (6): 1805-1814.doi: 10.13278/j.cnki.jjuese.20180346

Previous Articles    

Data Preprocessing Method of IoT Observation System in Coastal Wetland

Huang Gaixian, Tian Bo, Zhou Yunxuan, Yuan Qing   

  1. State Key Laboratory of Estuarine and Coastal Research, East China Normal University, Shanghai 200241, China
  • Received:2018-12-27 Published:2019-11-30
  • Supported by:
    Supported by National Key Research and Development Program of China (2016YFC0502704) and Scientific Research Project of Shanghai Science and Technology Commission (17DZ1201902, 18DZ1204802)

Abstract: Effective data preprocessing is essential to an online coastal wetland ecological internet of things (IoT) observation system. Outliers always occur due to the limitations of measuring methods and harsh environmental conditions, which challenge data applications. Based on the ecological observation data of Chongming Dongtan wetland in Shanghai, the outliers were divided into three types:abnormal values, abnormal fluctuation,and abnormal events. Integrating the interactions between indicators of coastal wetlands, we proposed a preprocessing method for the outliers of the coastal wetland ecological IoT system based on the residual probabilistic outlier detection algorithm, look-up table, and multi-indicator time series model. Compared with the traditional methods, this method can not only ensure the accuracy of outlier detection, but also better distinguish abnormal events from sensor problems to reduce false positives. Through the analysis of more than 50 000 data records of nine indicators, two abnormal events and 0.18%-8.12% abnormal values and abnormal fluctuations were detected with the threshold of 10-8-10-20. Through the analysis of the preprocessed data, we find that the observation principle and observation season will affect the stability of sensors, and the human activities in the observation area are the main factors causing abnormal events.

Key words: coastal wetlands, ecological internet of things, data preprocessing, multi-indicators time series model

CLC Number: 

  • P951
[1] Costanza R,d'Arge R, de Groot R, et al. The Value of the World's Ecosystem Services and Natural Capital[J]. Nature, 1997, 387:253-260.
[2] 宋庆丰,牛香,王兵,等. 基于大数据的森林生态系统服务功能评估进展[J]. 生态学杂志,2015,34(10):2914-2921. Song Qingfeng, Niu Xiang, Wang Bing, et al. Review on Forest Ecosystem Services Assessment Based on Big Data[J]. Chinese Journal of Ecology, 2015, 34(10):2914-2921.
[3] 崔洪亮,于淼,常天英,等. 应用于海洋环境和海洋工程的光纤传感技术[J]. 吉林大学学报(地球科学版),2017,47(1):279-293. Cui Hongliang, Yu Miao, Chang Tianying, et al. Fiber Optic Sensing Technology for Applications in Marine Environment and Marine Engineering[J]. Journal of Jilin University (Earth Science Edition), 2017, 47(1):279-293.
[4] Fiebrich C A, Morgan C R, Mccombs A G, et al. Quality Assurance Procedures for Mesoscale Meteorological Data[J]. Journal of Atmospheric and Oceanic Technology, 2010, 27(10):1565-1582.
[5] Byer D, Carlson K H. Real-Time Detection of Intentional Chemical Contamination in the Distribution System[J]. Journal American Water Works Association, 2005, 97(7):130-133.
[6] Wu H, Tang X, Wang Z, et al. Probabilistic Automatic Outlier Detection for Surface Air Quality Measurements from the China National Environmental Monitoring Network[J]. Advances in Atmospheric Sciences, 2018, 35(12):1522-1532.
[7] 魏媛,冯天恒,黄平捷,等. 管网水质多指标动态关联异常检测方法[J]. 浙江大学学报(工学版),2016,50(7):1402-1409. Wei Yuan, Feng Tianheng, Huang Pingjie, et al. Contamination Event Detection Method Based on Dynamic Correlation Analyis of Multiple Water Quality Parameters[J]. Journal of Zhejiang University (Engineering Science), 2016, 50(7):1402-1409.
[8] Breunig M M, Kriegel H P, Ng R T, et al. LOF:Identifying Density-Based Local Outliers[C]//Acm Sigmod International Conference on Management of Data. Dallas:ACM, 2000:93-104.
[9] Billor N, Hadi A S, Velleman P F. BACON:Blocked Adaptive Computationally Efficient Outlier Nominators[J]. Computational Statistics & Data Analysis, 2000, 34(3):279-298.
[10] Hochenbaum J, Vallis O S, Kejariwal A. Automatic Anomaly Detection in the Cloud via Statistical Learning[J/OL]. arXiv Preprint. (2017-04-24). http://arxiv.org/abs/1704.07706.
[11] Babin S M, Burkom H S, Mnatsakanyan Z R, et al. Drinking Water Security and Public Health Disease Outbreak Surveillance[J]. Johns Hopkins Apl Technical Digest, 2008, 27(4):403-411.
[12] Modaresi F, Araghinejad S. A Comparative Assessment of Support Vector Machines, Probabilistic Neural Networks, and K-Nearest Neighbor Algorithms for Water Quality Classification[J]. Water Resources Management, 2014, 28(12):4095-4111.
[13] Zhang Y, Meratnia N, Havinga P J M. Outlier Detection Techniques for Wireless Sensor Networks:A Survey[J]. IEEE Communications Surveys and Tutorials, 2010, 12(2):159-170.
[14] 地表水环境质量标准:GB 3838-2002[S]. 北京:国家环境保护总局,国家质量监督检验检疫总局,2002. Environmental Quality Standards for Surface Water:GB 3838-2002[S]. Beijing:State of Environmental Protection Agency, General Administration of Quality Supervision, Inspection and Quarantine, 2002.
[15] Feng S, Hu Q, Qian W. Quality Control of Daily Meteorological Data in China, 1951-2000:A New Dataset[J]. International Journal of Climatology, 2004, 24(7):853-870.
[16] Baty F, Ritz C, Charles S, et al. A Toolbox for Nonlinear Regression in R:The Package Nlstools[J]. Journal of Statistical Software, 2015, 66(5):1-21.
[17] Dunn R J H, Willett K M, Thorne P W, et al. HadISD:A Quality-Controlled Global Synoptic Report Database for Selected Variables at Long-Term Stations from 1973-2011[J]. Climate of the Past, 2012, 8(5):1649-1679.
[18] 海水水质标准:GB 3097-1997[S]. 北京:国家环境保护局,1997. Sea Water Quality Standard:GB 3097-1997[S]. Beijing:National Environmental Protection Agency, 1997.
[19] 翟世奎,张怀静,范德江,等. 长江口及其邻近海域悬浮物浓度和浊度的对应关系[J]. 环境科学学报,2005,25(5):693-699. Zhai Shikui, Zhang Huaijing, Fan Dejiang, et al. Corresponding Relationship Between Suspended Matter Concentration and Turbidity on Changjiang Estuary and Adjacent Sea Area[J]. Acta Scientiae Circumstantiae, 2005, 25(5):693-699.
[20] 李修竹,苏荣国,张传松,等. 基于支持向量机的长江口及其邻近海域叶绿素-a浓度预测模型[J]. 中国海洋大学学报(自然科学版),2019,49(1):69-76. Li Xiuzhu, Su Rongguo, Zhang Chuansong, et al. A Chl-a Prediction Model Based on Support Vector Machine in Yangtze River Estuaries and Its Adjacent Sea Area[J]. Periodical of Ocean University of China, 2019, 49(1):69-76.
[21] 王佳鹏,施润和,张超,等. 基于光谱分析的长江口湿地互花米草叶片叶绿素含量反演研究[J]. 遥感技术与应用,2017,32(6):1056-1063. Wang Jiapeng, Shi Runhe, Zhang Chao, et al. Study on the Inversion of Chlorophyll Content of Spartina Alterniflora Leaf in the Yangtze River Estuary Wetland Based on Spectral Analysis[J]. Remote Sensing Technology and Application, 2017, 32(6):1056-1063.
[22] 谢明媚,孙德勇,丘仲锋,等. 长江口水质MERIS卫星数据遥感反演研究[J]. 广西科学,2016,23(6):520-527. Xie Mingmei, Sun Deyong, Qiu Zhongfeng, el at. Water Quality Retrievals from MERIS Satellite Data in Yangtze Estuary[J]. Guangxi Sciences, 2016, 23(6):520-527.
[23] 崔莉凤,黄振芳,刘载文,等. 水华暴发叶绿素-a与表征指标溶解氧和pH的关系[J]. 给水排水,2008,44(增刊1):177-178. Cui Lifeng, Huang Zhenfang, Liu Zaiwen, et al. Relationship Between Chlorophyll-a and Characteristic Indicators of Dissolved Oxygen and pH in Blooms[J]. Water & Wastewater Engineering, 2008, 44(Sup. 1):177-178.
[1] LI Gui-dong,ZHOU Yun-xuan,TIAN Bo,LIU Zhi-guo,ZHENG Zong-sheng. Shanghai Estuarine and Coastal Wetlands Change Analysis Based on Remote Sensing and GIS [J]. J4, 2008, 38(2): 319-0323.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] QIN Li-juan, CAO Jian-feng,PING Jian-hua,JIANG Ji-yi, WANG Nan, SHEN Yuan-yuan, LI Sheng. Application of Fuzzy Mathematics in Evaluation of Water Resources Value in Zhengzhou City[J]. J4, 2005, 35(04): 487 -0490 .
[2] GE Yu-hui, SUN Chun-lin, LIU Mao-xiu. The Flora from the Middle Jurassic Yan’an Formation in the Northeastern Margin of Erdos Basin and Discussion on the Palaeoclimate[J]. J4, 2006, 36(02): 164 -0168 .
[3] DONG Chun-yan,MA Rui,CHI Xiao-guo,LIU Jian-feng,LI Guang-rong. An Experimental Study on the Relationship Between Stress and Rock Melt[J]. J4, 2006, 36(02): 177 -0182 .
[4] YANG Jun-peng, HU Ke, LIU Yu-ying. The Stable Carbon Isotopic Characteristics of Carbonates in Salini alkani Soil in the West Jilin Province[J]. J4, 2006, 36(02): 245 -0249 .
[5] CUI Jian,LIN Nian-feng,TANG Jie,JIANG Ling-ling,CAI Yu. The Dynamic Change and Trend Forecast of Land Use in the Lower Huolin River Basin[J]. J4, 2006, 36(02): 259 -0264 .
[6] ZOU Xin-ning,SUN Wei,ZHANG Meng-bo,WAN Yu-jun. The Application of Seismic Attributes Analysis to Lithologic Gas Reservoir Description[J]. J4, 2006, 36(02): 289 -0294 .
[7] DU Chun-guo, ZOU Hua-yao, SHAO Zhen-jun,ZHANG Jun. Formation Mechanism and Mode of Sand Lens Reservoirs[J]. J4, 2006, 36(03): 370 -376 .
[8] ZHU Hong-chen,WANG Hai-po, ZHANG Jiong-fei. Two Genetic Types of Gold Deposits in Sonid Zuoqi , Inner Mongolia[J]. J4, 2006, 36(05): 759 -766 .
[9] BAO Qing-zhong,ZHANG Chang-jie,WU Zhi-li,WANG Hong,LI Wei,SANG Jia-he,LIU Yong-sheng. SHRIMP U-Pb Zircon Geochronology of a Carboniferous Quartz-Diorite in Baiyingaole Area, Inner Mongolia and Its Implications[J]. J4, 2007, 37(1): 15 -0023 .
[10] GAO Hong-mei,GAO Fu-hong,FAN Fu, GAO Yu-qiao. Geochemical Characteristics of Solvable Organic Matter Extracted from Source Rocks of Early Cretaceous in Jixi Basin[J]. J4, 2007, 37(1): 86 -0090 .