Abstract:
Aiming at the characteristics of multi-period, multi-dependency, and backfill requirements for massive time-series data statistical tasks in smart grid and IoT fields, this paper proposes a master-slave scheduling framework for massive data statistical tasks. The framework adopts a master-slave architecture with master node scheduling management and slave node execution, supporting multi-period statistical configurations from minute to year. The framework introduces a pre-processing period mechanism, enabling large-period statistics to be efficiently completed based on small-period intermediate results; adopts task dependency scheduling strategy to ensure the execution order of tasks; designs a distributed parallel execution mode to dynamically partition large-scale statistical tasks to cluster nodes for parallel computing, fully utilizing cluster computing resources; designs partitioned storage and storage timestamp management mechanisms to support the persistence of massive statistical results; provides data backfill and re-statistics functions to solve the problem of historical data correction. The framework provides an efficient, reliable, and scalable solution for enterprise-level data statistical analysis.