高级检索

一种面向海量数据统计任务的主从调度框架设计

A Master-Slave Scheduling Framework Design for Massive Data Statistical Tasks

  • 摘要: 针对智能电网、物联网等领域海量时序数据统计任务的多周期、多依赖、需回补等特点,本文提出了一种面向海量数据统计任务的主从调度框架。该框架采用主节点调度管理与从节点执行计算的主从架构,支持分钟至年级等多周期统计配置。框架引入预处理周期机制,使大周期统计可基于小周期中间结果高效完成;采用任务依赖调度策略,确保统计任务的执行顺序;设计分布式并行执行模式,将大规模统计任务动态分片至集群节点并行计算,充分利用集群计算资源;设计分表存储与存储时标管理机制,支撑海量统计结果的持久化;提供数据回补与重统计功能,解决历史数据修正问题。该框架为企业级数据统计分析提供了一种高效、可靠、可扩展的解决方案。

     

    Abstract: Aiming at the characteristics of multi-period, multi-dependency, and backfill requirements for massive time-series data statistical tasks in smart grid and IoT fields, this paper proposes a master-slave scheduling framework for massive data statistical tasks. The framework adopts a master-slave architecture with master node scheduling management and slave node execution, supporting multi-period statistical configurations from minute to year. The framework introduces a pre-processing period mechanism, enabling large-period statistics to be efficiently completed based on small-period intermediate results; adopts task dependency scheduling strategy to ensure the execution order of tasks; designs a distributed parallel execution mode to dynamically partition large-scale statistical tasks to cluster nodes for parallel computing, fully utilizing cluster computing resources; designs partitioned storage and storage timestamp management mechanisms to support the persistence of massive statistical results; provides data backfill and re-statistics functions to solve the problem of historical data correction. The framework provides an efficient, reliable, and scalable solution for enterprise-level data statistical analysis.

     

/

返回文章
返回