WO2019223178A1 - Cross-platform task scheduling method and system, computer device, and storage medium - Google Patents
Cross-platform task scheduling method and system, computer device, and storage medium Download PDFInfo
- Publication number
- WO2019223178A1 WO2019223178A1 PCT/CN2018/104702 CN2018104702W WO2019223178A1 WO 2019223178 A1 WO2019223178 A1 WO 2019223178A1 CN 2018104702 W CN2018104702 W CN 2018104702W WO 2019223178 A1 WO2019223178 A1 WO 2019223178A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- essbase
- odi
- scheduling
- oracle
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Definitions
- the present application relates to the technical field of task scheduling, and in particular, to a cross-platform task scheduling method, system, computer device, and storage medium.
- ORACLE ODI Order Integrator
- ESSBASE is a multi-dimensional database of BI software hyperion. Unlike a relational database in the usual sense, ESSBASE divides data into "blocks", and each data defines a different dimension.
- a cross-platform task scheduling method includes:
- S1 configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
- S3 Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
- this application also provides a cross-platform task scheduling system, including:
- the configuration unit is configured to configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in a scheduling platform database;
- the deployment unit is configured to use the Oracle ODI or ESSBASE execution template to deploy the Oracle ODI or ESSBASE execution template to convert the SSH in the scheduling platform server to the Oracle ODI or ESSBASE server to execute the Oracle ODI or ESSBASE server.
- Task scheduling is configured to use the Oracle ODI or ESSBASE execution template to deploy the Oracle ODI or ESSBASE execution template to convert the SSH in the scheduling platform server to the Oracle ODI or ESSBASE server to execute the Oracle ODI or ESSBASE server.
- the execution unit is configured to capture task information serving ORACLE or ODI or ESSBASE scheduling conditions and stitch the parameter information into executable ODI or ODI or ESSBASE scheduling tasks, and execute ORACLE or ODI using SSH in ORACLE or ESSBASE server Or ESSBASE scheduling tasks, cross-platform task scheduling.
- the present application also provides a computer device including a memory and a processor.
- the memory stores computer-readable instructions.
- the processing causes the processing.
- the processor performs the following steps:
- S1 configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
- S3 Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
- the present application also provides a storage medium storing computer-readable instructions.
- the computer-readable instructions When the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:
- S1 configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
- S3 Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
- Parameter information and store the parameter information in the scheduling platform database; set up a database table in the scheduling platform database, use the scheduling platform server to read SSH account information, help SSH log in to the scheduling platform server, and install python on the scheduling platform Program, use it to write ORACLE ODI or ESSBASE execution modules, and then deploy ORACLE ODI or ESSBASE execution templates, and then use it to convert SSH in the scheduling platform server to SSH in ORACLE ODI or ESSBASE server to execute ORACLE ODI or ESSBASE server Task scheduling; the scheduling platform server continuously polls the scheduling platform configuration library through ORACLE ODI or ESSBASE execution modules, grabs task information that serves ORACLE ODI or ESSBASE scheduling conditions, and obtains ORACLE ODI or ESSBASE scheduling conditions from the scheduling platform database of Parameter information.
- the Oracle ODI or ESSBASE execution module will stitch the task information and parameter information into executable Oracle ODI or ESSBASE scheduling tasks.
- Various types of tasks can form a complete dependency flow and data processing flow, which greatly saves the development manpower on scheduling and cross-platform tool dependency, which greatly facilitates the operation and maintenance personnel's batch operation and task monitoring of the entire data flow.
- FIG. 1 is a flowchart of a cross-platform task scheduling method according to an embodiment of the present application
- FIG. 3 is a flowchart of a deployment execution module according to an embodiment of the present application.
- FIG. 5 is a schematic diagram of a cross-platform task scheduling system according to an embodiment of the present application.
- FIG. 6 is a schematic diagram of configuration parameter information in an embodiment of the present application.
- FIG. 7 is a schematic diagram of a deployment execution module in an embodiment of the present application.
- FIG. 8 is a schematic diagram of performing a scheduling task in an embodiment of the present application.
- this application provides a cross-platform task scheduling method, including:
- Step S1 Configure the parameter information of Oracle ODI or ESSBASE for task scheduling, and store the parameter information in the database of the scheduling platform;
- Step S2 Deploy the Oracle ODI or ESSBASE execution template and use the Oracle ODI or ESSBASE execution template to transfer the information in the scheduling platform server.
- SSH is converted into Oracle or ODI or ESSBASE server to perform task scheduling in ORACLE ODI or ESSBASE server;
- Step S3 Capture task information and parameter information serving ORACLE ODI or ESSBASE scheduling conditions and stitch them into executable ODI ODI Or ESSBASE scheduling tasks, and use Oracle ODI or SSH in the ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks to perform cross-platform task scheduling.
- ZooKeeper is an open and open source distributed application coordination service. It is an open source implementation of Google's Chubby and an important component of Hadoop and Hbase. ZooKeeper is a software that provides consistent services for distributed applications. The functions it can provide include configuration maintenance, domain name services, distributed synchronization, and group services. It can implement key services that are complex and error-prone, and will be easy to use. The interface and efficient and stable system are provided to users.
- an Oracle database may be used as the scheduling platform database.
- Oracle database is currently the world's leading relational database relational system. Its system is portable, easy to use, and powerful. It is suitable for various large, medium, small, and microcomputer environments. Storing the completed ODI or ODI or ESSBASE parameter information in the Oracle database can ensure the stability of ORACLE ODI or ESSBASE parameter information, and can adapt to the storage of parameter information of different reserves. When using the ORACLE ODI or ESSBASE execution module for grabbing, it can quickly locate the required ODILE or ESSBASE parameter information.
- the scheduling platform server in this embodiment may select a Linux server.
- the Linux server is a network server using the Linux system for business processing applications, such as network and system management, database management, and Web services. It is a server with high performance and open source.
- the Linux server has excellent system stability and security. Because it uses a multi-user operating system, it has good flexibility and scalability. It can accommodate resources well regardless of the processor or machine architecture.
- the software on the Linux server is free, so the Linux server can save costs in application development. Most user services can be developed more freely with Linux systems.
- the ORACLE ODI or ESSBASE task scheduling method enables various types of tasks in the entire project to form a complete dependency flow and data processing flow, which greatly saves the development manpower in scheduling and cross-platform tool dependency, which greatly facilitates operation. Maintenance personnel monitor the batch operations and tasks of the entire data flow.
- FIG. 2 is a flowchart of configuring parameter information according to an embodiment of the present application.
- step S1 parameter information of ODILE or ESSBASE for task scheduling is configured through a parameter template table and a parameter routing table.
- This embodiment Step S1 may include S101-S103:
- Step S101 Set the Oracle database on the scheduling platform, and set the parameter template table and parameter routing table of ODILE or ESSBASE in the parameter routing table.
- the parameter routing table is based on the historical record stored in the Oracle database, and one parameter corresponds to one record.
- the generated form as shown in Table 1 below, when generating the parameter template table, according to the historical records of parameters such as task name, date, and time type stored in the Oracle database, a parameter template table is generated by using one parameter corresponding to one record;
- the parameter routing table includes how many parameter items each parameter has and a path for obtaining information under each parameter item, and the parameter routing table is shown in Table 2 below;
- Step S102 Select the number of parameters and parameter names required by the Oracle ODI or ESSBASE task when scheduling, and configure according to the parameter routing table to obtain the parameter item corresponding to the parameter name; for example, the parameter name is "date ", Then in the parameter template table shown in Table 1 above," date "is positioned as the parameter item” inc_time ";
- Step S103 Configure the parameter information under the parameter item according to the parameter routing table; for example, the parameter item is "inc_time”, and according to the parameter routing table shown in Table 2 above, it can be known according to "inc_time” "Inc_time” contains two item categories, "inc_start” and "inc_end”.
- ORACLE ODI or ESSBASE parameter information provides preparation conditions for splicing into executable ODI OESS or ESSBASE scheduling tasks.
- FIG. 3 is a flowchart of deploying an execution module according to an embodiment of the present application.
- step S2 read the SSH account information, log in to SSH to the scheduling platform server, and use the python program on the scheduling platform to use It compiles Oracle ODI or ESSBASE execution modules, deploys ORACLE ODI or ESSBASE execution modules, and then uses the ORACLE ODI or ESSBASE execution modules to convert SSH in the scheduling platform into ORACLE ODI or ESSBASE servers.
- step S2 may include S201-S203;
- Step S201 setting a database table in the database of the scheduling platform, where the database table stores SSH account information, and the scheduling platform server reads the SSH account information from the database table to help SSH login to the scheduling platform server;
- the SSH account information includes ORACLE ODI or ESSBASE server IP information, user information, and password information.
- the database table is shown in Table 3 below:
- the dispatch platform server reads according to the information shown in Table 3 above, reads the corresponding dispatch platform server name according to "Server_name”, reads the IP information of the dispatch platform server, account based on "Server_ip", “Server_user” and “Server_pwd”. Information and password information.
- Step S202 Install a python program on the scheduling platform, and use the python program to write an Oracle ODI or ESSBASE execution module. After SSH logging in to the scheduling platform server, upload the ORACLE ODI or ESSBASE execution module program to the scheduling platform server using SSH Secure Shell. To complete the deployment of the Oracle ODI or ESSBASE execution module;
- the python program in this embodiment is free software.
- the source code and interpreter CPython follow the CPL protocol, and its syntax is concise and clear.
- Python has a rich and powerful library that can easily link various modules (especially C / C ++) made in other languages. Install a Python program on the scheduling platform. You can use this program to quickly and easily write ORACLE ODI or ESSBASE execution modules.
- Step S203 Use ORACLE ODI or ESSBASE execution module to convert SSH logged in to the scheduling platform server into ORACLE ODI or ESSBASE server.
- the Oracle ODI or ESSBASE execution module inherits the base class of the scheduling platform to implement functions such as Oracle ODI or ESSBASE task execution, abnormal return, log recording, error log extraction, and number of recorded logs.
- it can implement scheduling
- the SSH in the platform server is converted into OSSH in Oracle ODI or ESSBASE server to achieve the connection between ORACLE ODI and ESSBASE scheduling tasks.
- the SSH in the scheduling platform server is converted into the SSH in ORACLE ODI or ESSBASE server, and an execution component is provided for the next step of ORACLE ODI or ESSBASE task scheduling.
- FIG. 4 is a flowchart of executing a scheduling task according to an embodiment of the present application.
- the executable ODI or ESSBASE scheduling task is spliced and executed by using ORACLE or SSH in the ESSBASE server.
- step S3 includes S301-S303;
- Step S301 Fill in a deployment document of a scheduled ORACLE or ESSBASE task schedule on a scheduling platform.
- Step S302 Import the deployment document into a web interface through a scheduling platform, and store information in the imported deployment document into a scheduling platform configuration library.
- the deployment document in this embodiment is filled with an EXCEL-type deployment document, and records information such as the server name, program path, scheduling time, and scheduling frequency corresponding to the Oracle ODI or ESSBASE task. After filling in the deployment document, you can import it to the web interface and import it as a web page.
- Step S303 The scheduling platform server continuously polls the scheduling platform configuration library through the Oracle ODI or ESSBASE execution module, grabs task information serving the scheduling conditions of ORACLE ODI or ESSBASE, and obtains from the database of the scheduling platform that meets the ODI or ESSBASE scheduling conditions.
- the Oracle ODI or ESSBASE execution module will stitch the task information and parameter information into executable Oracle ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in the ESSBASE server to perform Oracle ODI or ESSBASE scheduling tasks.
- ORACLE ODI or SSH in the ESSBASE server to execute ORACLE ODI or ESSBASE scheduling tasks to achieve cross-platform task scheduling of ORACLE ODI and ESSBASE tasks.
- the embodiment of the present application also provides a cross-platform task scheduling system, as shown in FIG. 5, including: a configuration unit, configured to configure ORACLE ODI or ESSBASE parameter information for task scheduling, and parameter information Stored in the scheduling platform database; the deployment unit is set to deploy ORACLE ODI or ESSBASE execution templates and use it to convert SSH in the scheduling platform server to SSH in ORACLE ODI or ESSBASE server to perform tasks in ORACLE ODI or ESSBASE server Scheduling; execution unit, which is set to capture task information and parameter information that serve the scheduling conditions of Oracle ODI or ESSBASE and stitch them into executable Oracle ODI or ESSBASE scheduling tasks, and execute ORACLE ODI or ESSBASE by using ORACLE or SSH in the ESSBASE server Schedule tasks and perform cross-platform task scheduling.
- a configuration unit configured to configure ORACLE ODI or ESSBASE parameter information for task scheduling, and parameter information Stored in the scheduling platform database
- the deployment unit
- FIG. 6 is a schematic diagram of configuration parameter information according to an embodiment of the present application.
- the configuration unit includes a setting module configured to set an Oracle database on a scheduling platform and set a parameter template table of ORACLE ODI or ESSBASE therein. And parameter routing table.
- the parameter routing table is a table generated by using a parameter corresponding to a record according to the historical records stored in the Oracle database.
- the parameter routing table contains how many parameter items each parameter has and how many The path for obtaining information under each parameter; the parameter acquisition module is set to the number of parameters and parameter names required by the selected ODILE or ESSBASE task for scheduling, and is configured according to the parameter routing table to obtain the parameters corresponding to the parameter name Item; a configuration parameter information module configured to configure parameter information under the parameter item according to a parameter routing table.
- FIG. 7 is a schematic diagram of a deployment execution module according to an embodiment of the present application.
- the execution module includes a login module configured to set a database table in a scheduling platform database.
- the database table stores SSH account information. Read the SSH account information in the database table to help SSH login to the scheduling platform server; deploy the execution module, set to install a python program on the scheduling platform, and use the python program to write ORACLE ODI or ESSBASE execution modules, and SSH to log in to the scheduling platform server Then upload the ORACLE ODI or ESSBASE execution module program to the scheduling platform server using SSH Secure Shell to complete the deployment of the ODILE or ESSBASE execution module; the conversion module is set to use ORACLE ODI or ESSBASE execution module to log in to the dispatch platform server SSH into ODI or ODI or ESSBASE server.
- FIG. 8 is a schematic diagram of executing a scheduling task according to an embodiment of the present application. As shown in FIG. 8, it includes: a filling module configured to fill in a deployment file of a predetermined format type ORACLE ODI or ESSBASE task scheduling on a scheduling platform; import and save Module, set to import the deployment document to the web interface through the dispatch platform, and store the information in the imported deployment document to the dispatch platform configuration library; execute the dispatch task module, set to the dispatch platform server to execute the module via ORACLE ODI or ESSBASE Constantly poll the scheduling platform configuration library, grab the task information that serves the ODI or ESSBASE scheduling conditions, and obtain parameter information that meets the ODILE or ESSBASE scheduling conditions from the scheduling platform database.
- a filling module configured to fill in a deployment file of a predetermined format type ORACLE ODI or ESSBASE task scheduling on a scheduling platform
- import and save Module set to import the deployment document to the web interface through the dispatch platform, and store the information in the
- the ORACLE ODI or ESSBASE execution module will The task information and parameter information are spliced into executable Oracle ODI or ESSBASE scheduling tasks, and Oracle ODI or SSH in the ESSBASE server is used to perform ORACLE ODI or ESSBASE scheduling tasks.
- the scheduling platform adopts a distributed application coordination service ZooKeeper, a scheduling platform database is preset on ZooKeeper, and parameter information of ORACLE ODI or ESSBASE is stored.
- the computer device includes a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor.
- the processor implements the computer-readable instructions to implement the foregoing. Steps in the cross-platform task scheduling method in each embodiment.
- the present application also provides a storage medium storing computer-readable instructions.
- the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps in the foregoing embodiments. Steps in a platform task scheduling method.
- the computer program can be stored in a computer-readable storage medium.
- the storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (Random Access Memory, RAM).
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
本申请要求于2018年05月21日提交中国专利局、申请号为201810487527.X、发明名称为″跨平台任务调度方法、系统、计算机设备和存储介质″的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed on May 21, 2018 with the Chinese Patent Office, application number 201810487527.X, and the invention name "Cross-platform task scheduling method, system, computer equipment and storage medium", all of which The contents are incorporated herein by reference.
本申请涉及任务调度技术领域,特别是涉及跨平台任务调度方法、系统、计算机设备和存储介质。The present application relates to the technical field of task scheduling, and in particular, to a cross-platform task scheduling method, system, computer device, and storage medium.
ORACLE ODI(Oracle Data Integrator)是Oracle的数据集成类的工具,是使用E-LT的理念设计出来的数据抽取/数据转换工具。ESSBASE是BI软件hyperion的多维数据库,不同于通常意义上的关系数据库,ESSBASE把数据按照″块″进行划分,每个数据会定义不同的维度。现有仅有一种基于HIVE任务的调度方法及装置,涉及对ORACLE ODI和ESSBASE的调度方法,这种方法虽是ORACLE ODI的调度方法,但是该种方法依然存在诸多弊端,其具体如下:ORACLE ODI (Oracle Data Integrator) is a data integration tool of Oracle. It is a data extraction / data conversion tool designed using the concept of E-LT. ESSBASE is a multi-dimensional database of BI software hyperion. Unlike a relational database in the usual sense, ESSBASE divides data into "blocks", and each data defines a different dimension. There is only one scheduling method and device based on HIVE tasks, which involves the scheduling methods of ORACLE ODI and ESSBASE. Although this method is the ORACLE ODI scheduling method, this method still has many disadvantages, which are as follows:
1、现有业内工具仅是单独实现ORACLE ODI或ESSBASE的调度方法,没有专门的工具足以实现ORACLE ODI和ESSBASE任务的跨平台、跨工具的调度方法;1. The existing tools in the industry only implement the scheduling methods of ORACLE ODI or ESSBASE separately. There is no special tool sufficient to implement the cross-platform, cross-tool scheduling methods of ORACLE ODI and ESSBASE tasks.
2、现有实现ORACLE ODI或ESSBASE的调度方法无法实现与现有大数据SQOOP任务和HIVE任务进行依赖;2. The existing scheduling methods that implement Oracle ODI or ESSBASE cannot achieve dependence on existing big data SQOOP tasks and HIVE tasks;
3、现有实现ORACLE ODI或ESSBASE的调度方法也无法形成直观、易于部署和管理的完整数据处理流。3. Existing scheduling methods that implement Oracle ODI or ESSBASE cannot form a complete data processing flow that is intuitive and easy to deploy and manage.
发明内容Summary of the Invention
基于此,有必要针对在进行跨平台任务调度时,无法实现ORACLE ODI或ESSBASE的任务调度,也无法形成直观、易于部署和管理的完整数据处理流等 一系列问题,提供跨平台任务调度方法、系统、计算机设备和存储介质。Based on this, it is necessary to provide cross-platform task scheduling methods for a series of problems such as the inability to implement ODILE or ESSBASE task scheduling and the formation of an intuitive, easy-to-deploy complete data processing flow. Systems, computer equipment, and storage media.
一种跨平台任务调度方法,包括:A cross-platform task scheduling method includes:
S1:配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将所述参数信息存储至调度平台数据库中;S1: configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
S2:部署ORACLE ODI或ESSBASE执行模板后利用所述ORACLE ODI或ESSBASE执行模板将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;S2: After deploying the ODI or ODI or ESSBASE execution template, use the ODI or ODI or ESSBASE execution template to convert SSH in the scheduling platform server to ORACLE or ODI or ESSBASE server to perform task scheduling in ORACLE ODI or ESSBASE server;
S3:抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息和所述参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。S3: Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
基于相同的技术构思,本申请还提供一种跨平台任务调度系统,包括:Based on the same technical concept, this application also provides a cross-platform task scheduling system, including:
所述配置单元,设置为配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将所述参数信息存储至调度平台数据库中;The configuration unit is configured to configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in a scheduling platform database;
所述部署单元,设置为部署ORACLE ODI或ESSBASE执行模板后利用所述ORACLE ODI或ESSBASE执行模板将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;The deployment unit is configured to use the Oracle ODI or ESSBASE execution template to deploy the Oracle ODI or ESSBASE execution template to convert the SSH in the scheduling platform server to the Oracle ODI or ESSBASE server to execute the Oracle ODI or ESSBASE server. Task scheduling
所述执行单元,设置为抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息和所述参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。The execution unit is configured to capture task information serving ORACLE or ODI or ESSBASE scheduling conditions and stitch the parameter information into executable ODI or ODI or ESSBASE scheduling tasks, and execute ORACLE or ODI using SSH in ORACLE or ESSBASE server Or ESSBASE scheduling tasks, cross-platform task scheduling.
基于相同的技术构思,本申请还提供一种计算机设备,包括存储器和处理器,所述存储器中存储有计算机可读指令,所述计算机可读指令被所述处理器执行时,使得所述处理器执行以下步骤:Based on the same technical concept, the present application also provides a computer device including a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processing causes the processing. The processor performs the following steps:
S1:配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将所述参数信息存储至调度平台数据库中;S1: configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
S2:部署ORACLE ODI或ESSBASE执行模板后利用所述ORACLE ODI或ESSBASE执行模板将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;S2: After deploying the ODI or ODI or ESSBASE execution template, use the ODI or ODI or ESSBASE execution template to convert SSH in the scheduling platform server to ORACLE or ODI or ESSBASE server to perform task scheduling in ORACLE ODI or ESSBASE server;
S3:抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息和所述参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。S3: Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
基于相同的技术构思,本申请还提供一种存储有计算机可读指令的存储介质,所述计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行以下步骤:Based on the same technical concept, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:
S1:配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将所述参数信息存储至调度平台数据库中;S1: configure ORACLE ODI or ESSBASE parameter information for task scheduling, and store the parameter information in the scheduling platform database;
S2:部署ORACLE ODI或ESSBASE执行模板后利用所述ORACLE ODI或ESSBASE执行模板将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;S2: After deploying the ODI or ODI or ESSBASE execution template, use the ODI or ODI or ESSBASE execution template to convert SSH in the scheduling platform server to ORACLE or ODI or ESSBASE server to perform task scheduling in ORACLE ODI or ESSBASE server;
S3:抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息和所述参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。S3: Capture task information serving ORACLE ODI or ESSBASE scheduling conditions and the parameter information to splice into executable ODI ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks, Perform cross-platform task scheduling.
上述跨平台任务调度方法、系统、计算机设备和存储介质,通过在调度平台数据库中设置参数模板表和参数路由表,根据所述参数模板表和参数路由表,配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将所述参数信息存储至调度平台数据库中;在调度平台数据库中设置数据库表,利用调度平台服务器读取SSH账号信息,帮SSH登录至调度平台服务器中,在调度平台上安装python程序,利用其进行编写ORACLE ODI或ESSBASE执行模块,再部署ORACLE ODI或ESSBASE执行模板后利用其将调度平台服务器中的SSH转换 成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;调度平台服务器通过ORACLE ODI或ESSBASE执行模块不断轮询调度平台配置库,抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息,并从调度平台数据库中获取符合ORACLE ODI或ESSBASE调度条件的参数信息,ORACLE ODI或ESSBASE执行模块会将所述任务信息和参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,使得整个项目各种类型的任务都能形成完整的依赖流和数据处理流,大大节省了在调度和跨平台工具依赖上的开发人力,极大的方便运维人员对整个数据流的批量操作和任务监控。The above-mentioned cross-platform task scheduling method, system, computer equipment and storage medium, by setting a parameter template table and a parameter routing table in a scheduling platform database, and configuring ORACLE ODI or ESSBASE for task scheduling according to the parameter template table and parameter routing table. Parameter information, and store the parameter information in the scheduling platform database; set up a database table in the scheduling platform database, use the scheduling platform server to read SSH account information, help SSH log in to the scheduling platform server, and install python on the scheduling platform Program, use it to write ORACLE ODI or ESSBASE execution modules, and then deploy ORACLE ODI or ESSBASE execution templates, and then use it to convert SSH in the scheduling platform server to SSH in ORACLE ODI or ESSBASE server to execute ORACLE ODI or ESSBASE server Task scheduling; the scheduling platform server continuously polls the scheduling platform configuration library through ORACLE ODI or ESSBASE execution modules, grabs task information that serves ORACLE ODI or ESSBASE scheduling conditions, and obtains ORACLE ODI or ESSBASE scheduling conditions from the scheduling platform database of Parameter information. The Oracle ODI or ESSBASE execution module will stitch the task information and parameter information into executable Oracle ODI or ESSBASE scheduling tasks. Use ORACLE ODI or SSH in the ESSBASE server to execute Oracle ODI or ESSBASE scheduling tasks to make the entire project Various types of tasks can form a complete dependency flow and data processing flow, which greatly saves the development manpower on scheduling and cross-platform tool dependency, which greatly facilitates the operation and maintenance personnel's batch operation and task monitoring of the entire data flow.
通过阅读下文优选实施方式的详细描述,各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的,而并不认为是对本申请的限制。Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the detailed description of the preferred embodiments below. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present application.
图1为本申请一个实施例中一种跨平台任务调度方法的流程图;1 is a flowchart of a cross-platform task scheduling method according to an embodiment of the present application;
图2为本申请一个实施例中配置参数信息的流程图;2 is a flowchart of configuring parameter information in an embodiment of the present application;
图3为本申请一个实施例中部署执行模块的流程图;3 is a flowchart of a deployment execution module according to an embodiment of the present application;
图4为本申请一个实施例中执行调度任务的流程图;4 is a flowchart of performing a scheduling task in an embodiment of the present application;
图5为本申请一个实施例中一种跨平台任务调度系统的示意图;5 is a schematic diagram of a cross-platform task scheduling system according to an embodiment of the present application;
图6为本申请一个实施例中配置参数信息的示意图;6 is a schematic diagram of configuration parameter information in an embodiment of the present application;
图7为本申请一个实施例中部署执行模块的示意图;7 is a schematic diagram of a deployment execution module in an embodiment of the present application;
图8为本申请一个实施例中执行调度任务的示意图。FIG. 8 is a schematic diagram of performing a scheduling task in an embodiment of the present application.
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅仅用以解释本申请,并不用于限定本申请。In order to make the purpose, technical solution, and advantages of the present application clearer, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.
本技术领域技术人员可以理解,除非特意声明,这里使用的单数形式″一″、″一个″、″所述″和″该″也可包括复数形式。应该进一步理解的是,本申请的说明书中使用的措辞″包括″是指存在所述特征、整数、步骤、操作、元件和/或组件,但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" may include plural forms unless specifically stated otherwise. It should be further understood that the wording "including" used in the specification of the present application refers to the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and / or groups thereof.
如图1所示,本申请提供一种跨平台任务调度方法,包括:As shown in FIG. 1, this application provides a cross-platform task scheduling method, including:
步骤S1:配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将参数信息存储至调度平台数据库中;步骤S2:部署ORACLE ODI或ESSBASE执行模板后利用ORACLE ODI或ESSBASE执行模板将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;步骤S3:抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息和参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。Step S1: Configure the parameter information of Oracle ODI or ESSBASE for task scheduling, and store the parameter information in the database of the scheduling platform; Step S2: Deploy the Oracle ODI or ESSBASE execution template and use the Oracle ODI or ESSBASE execution template to transfer the information in the scheduling platform server. SSH is converted into Oracle or ODI or ESSBASE server to perform task scheduling in ORACLE ODI or ESSBASE server; Step S3: Capture task information and parameter information serving ORACLE ODI or ESSBASE scheduling conditions and stitch them into executable ODI ODI Or ESSBASE scheduling tasks, and use Oracle ODI or SSH in the ESSBASE server to perform ORACLE ODI or ESSBASE scheduling tasks to perform cross-platform task scheduling.
本实施例中的调度平台可以选定ZooKeeper。ZooKeeper是一个开放式的、开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。ZooKeeper是一个为分布式应用提供一致性服务的软件,所能提供的功能包括配置维护、域名服务、分布式同步、组服务等,能够实现将封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。在ZooKeeper调度平台上设置调度平台数据库,保存ORACLE ODI或ESSBASE的配置信息,部署ORACLE ODI或ESSBASE执行模块以及将转换SSH,为实现利用SSH调度ORACLE ODI或ESSBASE任务提供稳定的运行环境。The scheduling platform in this embodiment may select ZooKeeper. ZooKeeper is an open and open source distributed application coordination service. It is an open source implementation of Google's Chubby and an important component of Hadoop and Hbase. ZooKeeper is a software that provides consistent services for distributed applications. The functions it can provide include configuration maintenance, domain name services, distributed synchronization, and group services. It can implement key services that are complex and error-prone, and will be easy to use. The interface and efficient and stable system are provided to users. Set the scheduling platform database on the ZooKeeper scheduling platform, save the configuration information of ORACLE ODI or ESSBASE, deploy ORACLE ODI or ESSBASE execution modules, and convert SSH to provide a stable operating environment for scheduling ORACLE ODI or ESSBASE tasks using SSH.
本实施例中调度平台数据库可以选用Oracle数据库。Oracle数据库是目前世界上处于领先地位的关系数据库关系系统,其系统可移植性好、使用方便、功能强,适用于各类大、中、小、微机环境。将配置完成的ORACLE ODI或ESSBASE参数信息存储至Oracle数据库中,能够保证ORACLE ODI或ESSBASE参数信息的稳定性,且能够适应不同储量的参数信息的保存。在利用 ORACLE ODI或ESSBASE执行模块进行抓取时,能够快速定位所需的ORACLE ODI或ESSBASE参数信息。In this embodiment, an Oracle database may be used as the scheduling platform database. Oracle database is currently the world's leading relational database relational system. Its system is portable, easy to use, and powerful. It is suitable for various large, medium, small, and microcomputer environments. Storing the completed ODI or ODI or ESSBASE parameter information in the Oracle database can ensure the stability of ORACLE ODI or ESSBASE parameter information, and can adapt to the storage of parameter information of different reserves. When using the ORACLE ODI or ESSBASE execution module for grabbing, it can quickly locate the required ODILE or ESSBASE parameter information.
本实施例中的调度平台服务器可以选定Linux服务器。Linux服务器是一种采用Linux系统的网络服务器,用于业务处理应用,如网络和系统管理、数据库管理和Web服务等,是具备高性能和开源性的一种服务器。Linux服务器具有极佳的系统稳定性、安全性,由于采用多用户操作系统,因此具有好的灵活性和可扩展性,无论处理器或机器架构都能很好地进行资源上的容纳。Linux服务器上的软件是免费,因而Linux服务器在应用开发上更能节约成本。大多数的用户业务,都可以用Linux系统对业务进行更加自由的开发。The scheduling platform server in this embodiment may select a Linux server. The Linux server is a network server using the Linux system for business processing applications, such as network and system management, database management, and Web services. It is a server with high performance and open source. The Linux server has excellent system stability and security. Because it uses a multi-user operating system, it has good flexibility and scalability. It can accommodate resources well regardless of the processor or machine architecture. The software on the Linux server is free, so the Linux server can save costs in application development. Most user services can be developed more freely with Linux systems.
上述ORACLE ODI或ESSBASE调度任务的方式,使得整个项目各种类型的任务都能形成完整的依赖流和数据处理流,大大节省了在调度和跨平台工具依赖上的开发人力,极大的方便运维人员对整个数据流的批量操作和任务监控。The ORACLE ODI or ESSBASE task scheduling method enables various types of tasks in the entire project to form a complete dependency flow and data processing flow, which greatly saves the development manpower in scheduling and cross-platform tool dependency, which greatly facilitates operation. Maintenance personnel monitor the batch operations and tasks of the entire data flow.
图2为本申请一个实施例中配置参数信息的流程图,如图2所示,在步骤S1中,通过参数模板表和参数路由表配置ORACLE ODI或ESSBASE进行任务调度的参数信息,本实施例中,步骤S1可以包括S101-S103:FIG. 2 is a flowchart of configuring parameter information according to an embodiment of the present application. As shown in FIG. 2, in step S1, parameter information of ODILE or ESSBASE for task scheduling is configured through a parameter template table and a parameter routing table. This embodiment Step S1 may include S101-S103:
步骤S101:在调度平台上设置Oracle数据库,并在其中设置ORACLE ODI或ESSBASE的参数模板表和参数路由表,参数路由表是根据存储在Oracle数据库中的历史记录,采用一个参数对应一条记录的方式而生成的表格;如下表1所示,在生成参数模板表时根据存储在Oracle数据库中task名称,日期、时间类型等参数的历史记录,采用一个参数对应一条记录的方式生成的参数模板表;Step S101: Set the Oracle database on the scheduling platform, and set the parameter template table and parameter routing table of ODILE or ESSBASE in the parameter routing table. The parameter routing table is based on the historical record stored in the Oracle database, and one parameter corresponds to one record. The generated form; as shown in Table 1 below, when generating the parameter template table, according to the historical records of parameters such as task name, date, and time type stored in the Oracle database, a parameter template table is generated by using one parameter corresponding to one record;
表1Table 1
所述参数路由表包含有每个参数其有多少个参数项以及每个参数项下信息的获取路径,所述参数路由表如下表2所示;The parameter routing table includes how many parameter items each parameter has and a path for obtaining information under each parameter item, and the parameter routing table is shown in Table 2 below;
表2Table 2
步骤S102:选定ORACLE ODI或ESSBASE任务在进行调度时需要的参数个数和参数名称,并根据所述参数路由表进行配置,获取参数名称对应的参数项;例如,所述参数名称为″日期″,则在上述表1所示的参数模板表中,将″日期″定位为参数项″inc_time″;Step S102: Select the number of parameters and parameter names required by the Oracle ODI or ESSBASE task when scheduling, and configure according to the parameter routing table to obtain the parameter item corresponding to the parameter name; for example, the parameter name is "date ", Then in the parameter template table shown in Table 1 above," date "is positioned as the parameter item" inc_time ";
步骤S103:根据所述参数路由表,配置所述参数项下的参数信息;例如,参数项为″inc_time″,根据上述表2所展示的参数路由表,即可根据″inc_time″,即可知晓″inc_time″包含两个项目类别,分别为″inc_start″和″inc_end″。Step S103: Configure the parameter information under the parameter item according to the parameter routing table; for example, the parameter item is "inc_time", and according to the parameter routing table shown in Table 2 above, it can be known according to "inc_time" "Inc_time" contains two item categories, "inc_start" and "inc_end".
上述通过配置ORACLE ODI或ESSBASE参数信息,为拼接成可执行的ORACLE ODI或ESSBASE调度任务提供准备条件。The foregoing configuration of ORACLE ODI or ESSBASE parameter information provides preparation conditions for splicing into executable ODI OESS or ESSBASE scheduling tasks.
图3为本申请一个实施例中部署执行模块的流程图,如图3所示,在步骤S2中,读取SSH账号信息,将SSH登录至调度平台服务器,在调度平台上安装python程序后利用其进行ORACLE ODI或ESSBASE执行模块的编写,并部署ORACLE ODI或ESSBASE执行模块,再利用所述ORACLE ODI或ESSBASE执行模块将调度平台中的SSH转换成ORACLE ODI或ESSBASE服务器中的 SSH。本实施中,步骤S2可以包括S201-S203;FIG. 3 is a flowchart of deploying an execution module according to an embodiment of the present application. As shown in FIG. 3, in step S2, read the SSH account information, log in to SSH to the scheduling platform server, and use the python program on the scheduling platform to use It compiles Oracle ODI or ESSBASE execution modules, deploys ORACLE ODI or ESSBASE execution modules, and then uses the ORACLE ODI or ESSBASE execution modules to convert SSH in the scheduling platform into ORACLE ODI or ESSBASE servers. In this implementation, step S2 may include S201-S203;
步骤S201:在调度平台数据库中设置数据库表,所述数据库表中存储有SSH账号信息,调度平台服务器从所述数据库表中读取SSH账号信息,帮助SSH登录至调度平台服务器中;Step S201: setting a database table in the database of the scheduling platform, where the database table stores SSH account information, and the scheduling platform server reads the SSH account information from the database table to help SSH login to the scheduling platform server;
在一个实施例中,所述SSH账号信息包括ORACLE ODI或ESSBASE服务器IP信息、用户信息和密码信息,其中所述数据库表如下表3所示:In one embodiment, the SSH account information includes ORACLE ODI or ESSBASE server IP information, user information, and password information. The database table is shown in Table 3 below:
表3table 3
调度平台服务器根据上述表3中显示的信息进行读取,根据″Server_name″读取对应的调度平台服务器名称,根据″Server_ip″、″Server_user″和″Server_pwd″读取调度平台服务器的IP信息,账户信息和密码信息。The dispatch platform server reads according to the information shown in Table 3 above, reads the corresponding dispatch platform server name according to "Server_name", reads the IP information of the dispatch platform server, account based on "Server_ip", "Server_user" and "Server_pwd". Information and password information.
步骤S202:在调度平台安装python程序,并利用所述python程序进行编写ORACLE ODI或ESSBASE执行模块,SSH登录至调度平台服务器后利用SSH Secure Shell将ORACLE ODI或ESSBASE执行模块的程序上传至调度平台服务器中,完成ORACLE ODI或ESSBASE执行模块的部署;Step S202: Install a python program on the scheduling platform, and use the python program to write an Oracle ODI or ESSBASE execution module. After SSH logging in to the scheduling platform server, upload the ORACLE ODI or ESSBASE execution module program to the scheduling platform server using SSH Secure Shell. To complete the deployment of the Oracle ODI or ESSBASE execution module;
本实施例中的python程序是自由软件,源代码和解释器CPython遵循CPL协议,其使用的语法简洁清晰。Python具有丰富和强大的库,能够将其他语言制作的各种模块(尤其是C/C++)很轻松地联结在一起。在调度平台上安装Python程序,能够利用该程序快速方便的编写ORACLE ODI或ESSBASE执行模块。The python program in this embodiment is free software. The source code and interpreter CPython follow the CPL protocol, and its syntax is concise and clear. Python has a rich and powerful library that can easily link various modules (especially C / C ++) made in other languages. Install a Python program on the scheduling platform. You can use this program to quickly and easily write ORACLE ODI or ESSBASE execution modules.
步骤S203:利用ORACLE ODI或ESSBASE执行模块将登录至调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH。Step S203: Use ORACLE ODI or ESSBASE execution module to convert SSH logged in to the scheduling platform server into ORACLE ODI or ESSBASE server.
本实施例中的ORACLE ODI或ESSBASE执行模块通过继承调度平台的基类,实现ORACLE ODI或ESSBASE任务的执行、异常返回、日志记录、错误日志提取、记录日志条数等功能,同时能够实现将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,实现ORACLE ODI和 ESSBASE调度任务的对接。In this embodiment, the Oracle ODI or ESSBASE execution module inherits the base class of the scheduling platform to implement functions such as Oracle ODI or ESSBASE task execution, abnormal return, log recording, error log extraction, and number of recorded logs. At the same time, it can implement scheduling The SSH in the platform server is converted into OSSH in Oracle ODI or ESSBASE server to achieve the connection between ORACLE ODI and ESSBASE scheduling tasks.
上述将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,为下一步进行ORACLE ODI或ESSBASE任务调度提供执行部件。In the above, the SSH in the scheduling platform server is converted into the SSH in ORACLE ODI or ESSBASE server, and an execution component is provided for the next step of ORACLE ODI or ESSBASE task scheduling.
图4为本申请一个实施例中执行调度任务的流程图,如图4所示,在步骤S3中,通过拼接成可执行的ORACLE ODI或ESSBASE调度任务,利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE的调度任务。本实施例中,步骤S3包括S301-S303;FIG. 4 is a flowchart of executing a scheduling task according to an embodiment of the present application. As shown in FIG. 4, in step S3, the executable ODI or ESSBASE scheduling task is spliced and executed by using ORACLE or SSH in the ESSBASE server. ORACLE, ODI or ESSBASE scheduling task. In this embodiment, step S3 includes S301-S303;
步骤S301:在调度平台上填写预设格式类型的ORACLE ODI或ESSBASE任务调度的部署文档。Step S301: Fill in a deployment document of a scheduled ORACLE or ESSBASE task schedule on a scheduling platform.
步骤S302:通过调度平台将所述部署文档导入至Web界面上,并将导入的部署文档中的信息存储至调度平台配置库中。Step S302: Import the deployment document into a web interface through a scheduling platform, and store information in the imported deployment document into a scheduling platform configuration library.
本实施例中的部署文档是通过填写EXCEL类型的部署文档,记录ORACLE ODI或ESSBASE任务对应的服务器名称、程序路径、调度时间、调度频度等信息。部署文档在填写完成之后,即可导入Web界面上,以网页的形式进行导入。The deployment document in this embodiment is filled with an EXCEL-type deployment document, and records information such as the server name, program path, scheduling time, and scheduling frequency corresponding to the Oracle ODI or ESSBASE task. After filling in the deployment document, you can import it to the web interface and import it as a web page.
步骤S303:调度平台服务器通过ORACLE ODI或ESSBASE执行模块不断轮询调度平台配置库,抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息,并从调度平台数据库中获取符合ORACLE ODI或ESSBASE调度条件的参数信息,ORACLE ODI或ESSBASE执行模块会将所述任务信息和参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务。Step S303: The scheduling platform server continuously polls the scheduling platform configuration library through the Oracle ODI or ESSBASE execution module, grabs task information serving the scheduling conditions of ORACLE ODI or ESSBASE, and obtains from the database of the scheduling platform that meets the ODI or ESSBASE scheduling conditions. For parameter information, the Oracle ODI or ESSBASE execution module will stitch the task information and parameter information into executable Oracle ODI or ESSBASE scheduling tasks, and use ORACLE ODI or SSH in the ESSBASE server to perform Oracle ODI or ESSBASE scheduling tasks.
上述调用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务实现ORACLE ODI和ESSBASE任务的跨平台任务调度。The above calls ORACLE ODI or SSH in the ESSBASE server to execute ORACLE ODI or ESSBASE scheduling tasks to achieve cross-platform task scheduling of ORACLE ODI and ESSBASE tasks.
基于相同的技术构思,本申请实施例还提供了一种跨平台任务调度系统,如图5所示,包括:配置单元,设置为配置ORACLE ODI或ESSBASE进行任务调度的参数信息,并将参数信息存储至调度平台数据库中;部署单元,设置为部署ORACLE ODI或ESSBASE执行模板后利用其将调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH,以执行ORACLE ODI或ESSBASE服务器中的任务调度;执行单元,设置为抓取服务于ORACLE ODI 或ESSBASE调度条件的任务信息和参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,并利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务,进行跨平台任务调度。Based on the same technical concept, the embodiment of the present application also provides a cross-platform task scheduling system, as shown in FIG. 5, including: a configuration unit, configured to configure ORACLE ODI or ESSBASE parameter information for task scheduling, and parameter information Stored in the scheduling platform database; the deployment unit is set to deploy ORACLE ODI or ESSBASE execution templates and use it to convert SSH in the scheduling platform server to SSH in ORACLE ODI or ESSBASE server to perform tasks in ORACLE ODI or ESSBASE server Scheduling; execution unit, which is set to capture task information and parameter information that serve the scheduling conditions of Oracle ODI or ESSBASE and stitch them into executable Oracle ODI or ESSBASE scheduling tasks, and execute ORACLE ODI or ESSBASE by using ORACLE or SSH in the ESSBASE server Schedule tasks and perform cross-platform task scheduling.
图6为本申请一个实施例中配置参数信息的示意图,如图6所示,配置单元包括:设置模块,设置为在调度平台上设置Oracle数据库,并在其中设置ORACLE ODI或ESSBASE的参数模板表和参数路由表,参数路由表是根据存储在所述Oracle数据库中的历史记录,采用一个参数对应一条记录的方式而生成的表格,参数路由表包含有每个参数其有多少个参数项以及每个参数项下信息的获取路径;获取参数项模块,设置为选定ORACLE ODI或ESSBASE任务在进行调度时需要的参数个数和参数名称,并根据参数路由表进行配置,获取参数名称对应的参数项;配置参数信息模块,设置为根据参数路由表,配置参数项下的参数信息。FIG. 6 is a schematic diagram of configuration parameter information according to an embodiment of the present application. As shown in FIG. 6, the configuration unit includes a setting module configured to set an Oracle database on a scheduling platform and set a parameter template table of ORACLE ODI or ESSBASE therein. And parameter routing table. The parameter routing table is a table generated by using a parameter corresponding to a record according to the historical records stored in the Oracle database. The parameter routing table contains how many parameter items each parameter has and how many The path for obtaining information under each parameter; the parameter acquisition module is set to the number of parameters and parameter names required by the selected ODILE or ESSBASE task for scheduling, and is configured according to the parameter routing table to obtain the parameters corresponding to the parameter name Item; a configuration parameter information module configured to configure parameter information under the parameter item according to a parameter routing table.
图7为本申请一个实施例中部署执行模块的示意图,如图7所示,包括:登录模块,设置为在调度平台数据库中设置数据库表,数据库表中存储有SSH账号信息,调度平台服务器从数据库表中读取SSH账号信息,帮助SSH登录至调度平台服务器中;部署执行模块,设置为在调度平台安装python程序,并利用python程序进行编写ORACLE ODI或ESSBASE执行模块,SSH登录至调度平台服务器后利用SSH Secure Shell将ORACLE ODI或ESSBASE执行模块的程序上传至调度平台服务器中,完成ORACLE ODI或ESSBASE执行模块的部署;转换模块,设置为利用ORACLE ODI或ESSBASE执行模块将登录至调度平台服务器中的SSH转换成ORACLE ODI或ESSBASE服务器中的SSH。FIG. 7 is a schematic diagram of a deployment execution module according to an embodiment of the present application. As shown in FIG. 7, the execution module includes a login module configured to set a database table in a scheduling platform database. The database table stores SSH account information. Read the SSH account information in the database table to help SSH login to the scheduling platform server; deploy the execution module, set to install a python program on the scheduling platform, and use the python program to write ORACLE ODI or ESSBASE execution modules, and SSH to log in to the scheduling platform server Then upload the ORACLE ODI or ESSBASE execution module program to the scheduling platform server using SSH Secure Shell to complete the deployment of the ODILE or ESSBASE execution module; the conversion module is set to use ORACLE ODI or ESSBASE execution module to log in to the dispatch platform server SSH into ODI or ODI or ESSBASE server.
图8为本申请一个实施例中执行调度任务的示意图,如图8所示,包括:填写模块,设置为在调度平台上填写预设格式类型的ORACLE ODI或ESSBASE任务调度的部署文档;导入保存模块,设置为通过调度平台将部署文档导入至Web界面上,并将导入的部署文档中的信息存储至调度平台配置库中;执行调度任务模块,设置为调度平台服务器通过ORACLE ODI或ESSBASE执行模块不断轮询调度平台配置库,抓取服务于ORACLE ODI或ESSBASE调度条件的任务信息,并从调度平台数据库中获取符合ORACLE ODI或ESSBASE调度条件的参数信息, ORACLE ODI或ESSBASE执行模块会将所述任务信息和参数信息拼接成可执行的ORACLE ODI或ESSBASE调度任务,利用ORACLE ODI或ESSBASE服务器中的SSH执行ORACLE ODI或ESSBASE调度任务。FIG. 8 is a schematic diagram of executing a scheduling task according to an embodiment of the present application. As shown in FIG. 8, it includes: a filling module configured to fill in a deployment file of a predetermined format type ORACLE ODI or ESSBASE task scheduling on a scheduling platform; import and save Module, set to import the deployment document to the web interface through the dispatch platform, and store the information in the imported deployment document to the dispatch platform configuration library; execute the dispatch task module, set to the dispatch platform server to execute the module via ORACLE ODI or ESSBASE Constantly poll the scheduling platform configuration library, grab the task information that serves the ODI or ESSBASE scheduling conditions, and obtain parameter information that meets the ODILE or ESSBASE scheduling conditions from the scheduling platform database. The ORACLE ODI or ESSBASE execution module will The task information and parameter information are spliced into executable Oracle ODI or ESSBASE scheduling tasks, and Oracle ODI or SSH in the ESSBASE server is used to perform ORACLE ODI or ESSBASE scheduling tasks.
在一个实施例中,调度平台采用分布式应用程序协调服务ZooKeeper,在ZooKeeper上预设调度平台数据库,保存ORACLE ODI或ESSBASE的参数信息。In one embodiment, the scheduling platform adopts a distributed application coordination service ZooKeeper, a scheduling platform database is preset on ZooKeeper, and parameter information of ORACLE ODI or ESSBASE is stored.
基于相同的技术构思,本申请提出了一种计算机设备,计算机设备包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机可读指令,处理器执行计算机可读指令时实现上述各实施例里跨平台任务调度方法中的步骤。Based on the same technical concept, this application proposes a computer device. The computer device includes a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor. The processor implements the computer-readable instructions to implement the foregoing. Steps in the cross-platform task scheduling method in each embodiment.
基于相同的技术构思本申请还提供了一种存储有计算机可读指令的存储介质,计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行上述各实施例里跨平台任务调度方法中的步骤。Based on the same technical concept, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps in the foregoing embodiments. Steps in a platform task scheduling method.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,该计算机程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。其中,存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)等非易失性存储介质,或随机存储记忆体(Random Access Memory,RAM)等。A person of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be implemented by using a computer program to instruct related hardware. The computer program can be stored in a computer-readable storage medium. When executed, the processes of the embodiments of the methods described above may be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (Random Access Memory, RAM).
以上所述实施例的各技术特征可以进行任意的组合,为使描述简洁,未对上述实施例中的各个技术特征所有可能的组合都进行描述,然而,只要这些技术特征的组合不存在矛盾,都应当认为是本说明书记载的范围。The technical features of the embodiments described above can be arbitrarily combined. In order to simplify the description, all possible combinations of the technical features in the above embodiments have not been described. However, as long as there is no contradiction in the combination of these technical features, It should be considered as the scope described in this specification.
以上所述实施例仅表达了本申请的几种实施方式,其描述较为具体和详细,但并不能因此而理解为对本申请专利范围的限制。应当指出的是,对于本领域的普通技术人员来说,在不脱离本申请构思的前提下,还可以做出若干变形和改进,这些都属于本申请的保护范围。因此,本申请专利的保护范围应以所附权利要求为准。The above-mentioned embodiments only express several implementation manners of the present application, and their descriptions are more specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of this application patent shall be subject to the appended claims.
Claims (20)
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810487527.XA CN108804241B (en) | 2018-05-21 | 2018-05-21 | Cross-platform task scheduling method, system, computer equipment and storage medium |
| CN201810487527.X | 2018-05-21 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2019223178A1 true WO2019223178A1 (en) | 2019-11-28 |
Family
ID=64091215
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2018/104702 Ceased WO2019223178A1 (en) | 2018-05-21 | 2018-09-08 | Cross-platform task scheduling method and system, computer device, and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108804241B (en) |
| WO (1) | WO2019223178A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111506406A (en) * | 2020-04-10 | 2020-08-07 | 深圳前海微众银行股份有限公司 | Workflow scheduling method, device and system and computer readable storage medium |
| CN113010276A (en) * | 2020-06-11 | 2021-06-22 | 深圳市科脉技术股份有限公司 | A task scheduling method, device, terminal device and storage medium |
| CN114398028A (en) * | 2022-01-13 | 2022-04-26 | 平安普惠企业管理有限公司 | Task batch processing method and device, computer equipment and storage medium |
| CN115256381A (en) * | 2022-07-19 | 2022-11-01 | 浙江国自机器人技术股份有限公司 | Robot scheduling device, system and method |
| CN115373848A (en) * | 2022-08-26 | 2022-11-22 | 中国工商银行股份有限公司 | Batch operation and maintenance method and device for servers |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110428453B (en) * | 2019-07-30 | 2020-12-15 | 深圳云天励飞技术有限公司 | Data processing method, data processing device, data processing equipment and storage medium |
| CN111651429B (en) * | 2020-05-26 | 2023-10-24 | 中国平安财产保险股份有限公司 | Big data scheduling parameter management method, device and distributed platform |
| CN113553569B (en) * | 2021-07-06 | 2022-12-09 | 猪八戒股份有限公司 | Single sign-on method, system and terminal of Syngnathus system based on proxy server |
| CN114780216B (en) * | 2022-04-18 | 2023-06-16 | 湖南长银五八消费金融股份有限公司 | Task scheduling method, device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102918530A (en) * | 2010-05-27 | 2013-02-06 | 甲骨文国际公司 | Data mart automation |
| CN106302778A (en) * | 2016-08-25 | 2017-01-04 | 广东亿迅科技有限公司 | A kind of distributed flow process automotive engine system |
| CN108037973A (en) * | 2017-12-20 | 2018-05-15 | 苏州优圣美智能系统有限公司 | A kind of data flow modeling interacted with data processing tools and processing system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8880668B2 (en) * | 2011-02-28 | 2014-11-04 | Verizon Patent And Licensing Inc. | Method and system for integrating data from multiple sources |
| CN105119757A (en) * | 2015-09-14 | 2015-12-02 | 沈文策 | Method and system for operation and maintenance automation of enterprise servers |
| CN107870807A (en) * | 2016-09-26 | 2018-04-03 | 平安科技(深圳)有限公司 | A kind of cross-platform method for scheduling task and device |
| CN107908465B (en) * | 2017-10-19 | 2020-11-03 | 深圳索信达数据技术有限公司 | Task scheduling method for big data platform |
-
2018
- 2018-05-21 CN CN201810487527.XA patent/CN108804241B/en active Active
- 2018-09-08 WO PCT/CN2018/104702 patent/WO2019223178A1/en not_active Ceased
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102918530A (en) * | 2010-05-27 | 2013-02-06 | 甲骨文国际公司 | Data mart automation |
| CN106302778A (en) * | 2016-08-25 | 2017-01-04 | 广东亿迅科技有限公司 | A kind of distributed flow process automotive engine system |
| CN108037973A (en) * | 2017-12-20 | 2018-05-15 | 苏州优圣美智能系统有限公司 | A kind of data flow modeling interacted with data processing tools and processing system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111506406A (en) * | 2020-04-10 | 2020-08-07 | 深圳前海微众银行股份有限公司 | Workflow scheduling method, device and system and computer readable storage medium |
| CN113010276A (en) * | 2020-06-11 | 2021-06-22 | 深圳市科脉技术股份有限公司 | A task scheduling method, device, terminal device and storage medium |
| CN114398028A (en) * | 2022-01-13 | 2022-04-26 | 平安普惠企业管理有限公司 | Task batch processing method and device, computer equipment and storage medium |
| CN115256381A (en) * | 2022-07-19 | 2022-11-01 | 浙江国自机器人技术股份有限公司 | Robot scheduling device, system and method |
| CN115373848A (en) * | 2022-08-26 | 2022-11-22 | 中国工商银行股份有限公司 | Batch operation and maintenance method and device for servers |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108804241B (en) | 2023-06-02 |
| CN108804241A (en) | 2018-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019223178A1 (en) | Cross-platform task scheduling method and system, computer device, and storage medium | |
| US12099465B2 (en) | Cross-platform communication for facilitation of data sharing | |
| US10778810B2 (en) | Staging and deployment to multiple service clouds | |
| CN102542007B (en) | Method and system for synchronization of relational databases | |
| US8874755B1 (en) | Provisioning of cloud networks with services | |
| US9311326B2 (en) | Virtual file system for automated data replication and review | |
| EP2675127B1 (en) | Method and device for automatically migrating system configuration item | |
| WO2021017301A1 (en) | Management method and apparatus based on kubernetes cluster, and computer-readable storage medium | |
| CN107864131A (en) | A kind of method and system for realizing Kubernetes cluster multi-tenant Network Isolations | |
| CN103780700A (en) | Application system and method for achieving compatibility and sharing among multi-source heterogeneous systems | |
| CN106533713B (en) | Application deployment method and device | |
| TW201737126A (en) | Method and device for executing data recovery operation | |
| US11108829B2 (en) | Managing network connections based on their endpoints | |
| CN108600282A (en) | Micro services dissemination method, device, storage medium and computer equipment | |
| CN106022007A (en) | Cloud platform system and method oriented to biological omics big data calculation | |
| WO2022193855A1 (en) | Task state updating method and apparatus, device, and medium | |
| CN105843871B (en) | Control and Management System of Distributed Application File | |
| CN111382201A (en) | Heterogeneous database synchronization method, device, computer equipment and storage medium | |
| CN109800081A (en) | A kind of management method and relevant device of big data task | |
| CN110008006B (en) | Container-based big data tool deployment method and system | |
| CN111897877A (en) | High-performance and high-reliability data sharing system and method based on distributed idea | |
| CN112612802B (en) | A real-time data processing method, device and platform | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| WO2015139565A1 (en) | Heterogeneous logging system management configuration | |
| CN108021411A (en) | System and method for discovering service call relationship |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18919534 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 18919534 Country of ref document: EP Kind code of ref document: A1 |