Abstract: To ensure stable network access and reduce resource waste, an open-source big data load balancing optimization algorithm based on real-time node load is proposed. An open-source big data node computing capability model is established, timely feedback and adjustments based on the size of node load are provided, the next action based on the number of requests received by servers in the region is predicted, exponential smoothing method is used to calculate the predicted number of server requests per second, the lag deviation problem of first- order exponential smoothing method is improved, and the comprehensive server load is calculated. Add a load agent and load monitor on the node to balance the number of blocks and the load of sharded nodes, and place undeleted shards and blocks into the minimum unit candidate list to achieve load balancing optimization. Through experiments, it has been proven that the proposed algorithm can improve network resource utilization and load balancing, ensuring a more stable and secure network during access.

Key words: node real-time load, open source big data, load balancing optimization, regional server, node performance