WO2011061034A1

WO2011061034A1 - A method and system for job scheduling in a data processing system with virtual environment

Info

Publication number: WO2011061034A1
Application number: PCT/EP2010/065769
Authority: WO
Inventors: Giuseppe Longobardi; Scot Maclellan
Original assignee: Compagnie IBM France SAS; International Business Machines Corp
Current assignee: Compagnie IBM France SAS; International Business Machines Corp
Priority date: 2009-11-23
Filing date: 2010-10-20
Publication date: 2011-05-26
Anticipated expiration: 2012-05-23

Abstract

A job scheduling system for executing work units in a data processing system wherein some of the work units require the availability of a virtual environment before the execution can start. The method of the present invention provides a job scheduling system which, when defining the scheduling plan, verifies if any virtual environment is required for the execution of one or more of the jobs. If this is the case, a repository containing all the necessary information for the set up and implementation of the Virtual Environment is accessed and the scheduling plan is modified to ensure that, at the scheduled time, the requested Virtual Environment is installed and ready for the execution of that one or more job.

Description

A METHOD AND SYSTEM FOR JOB SCHEDULING IN A DATA PROCESSING SYSTEM WITH VIRTUAL ENVIRONMENT TECHNICAL FIELD

The present invention relates to the field of data processing, more particularly of system for job scheduling in distributed data processing systems using virtual environments.

BACKGROUND OF INVENTION

Scheduling methods are commonly used in a data processing system to control submission of different work units to be executed (for example, jobs in a batch processing) . For this purpose, several types of workload schedulers have been proposed in the last years to automate the submission of large quantities of jobs. An example of scheduler is the IBM Tivoli Workload Scheduler for z/OS, by International Business Machines Corporation. A description of such scheduler can be found e.g. in "Best Practices: End-to-end and mainframe scheduling", SG24- 7156-01 or "Getting started with IBM; Tivoli Workload Scheduler V8.3", SG24-7237-00 Redbooks, published 30 October 2006.

A scheduler submits jobs according to a predefined plan. The plan establishes a flow of execution of the jobs according to several factors; typically, the factors affecting the flow of execution include temporal values (such as date, time, day of the week) and dependencies (such as completion of predecessor jobs or system resource availability) . The schedulers known in the art are very sophisticated in handling temporal and predecessor constraints. However, a very basic support is available for managing the problems relating to the availability of the resources that are used by the different jobs.

Another lack of commercially available scheduling systems is the capability to deal with virtual environments. In state of the art distributed systems there is an increasing interest and use of virtualized environment which offer several advantages in term of flexibility, security and scalability. There are several applications which provide management of Virtual Environments and which maintain repositories of Virtual machines/systems. However state of the art scheduling systems do not normally provide the possibility of scheduling jobs (e.g. a script) on a virtual machine with specific characteristics, e.g. a Linux RedHat x.x with a specific application, taken from a repository, creating an association between the scheduler workload and the required specific environment. It would be desirable to exploit the potentiality of Virtual Environments in job scheduling system, by allowing a suitable Virtual Environment being selected and implemented for the running of a specific job added to the scheduler plan. Also the synchronization of the scheduling plan and the Virtual Environment activation should be taken into consideration.

It is an object of the present invention to provide a technique which alleviates the above drawback of the prior art.

SUMMARY OF THE INVENTION In a preferred embodiment, the present invention provides a method of creating a plan of execution for scheduling a plurality of work units for execution on a data processing system, according to appended claim 1.

The method of the present invention can help to solve the problem of the prior art by providing a job scheduling system which, when defining the scheduling plan, verifies if any virtual environment is required for the execution of one or more of the jobs. If this is the case, a repository containing all the necessary information for the activation and the set of the Virtual Environment is accessed and the scheduling plan is modified with additional work units to ensure that, at the scheduled time, the requested Virtual Environment is installed (i.e. loaded) and ready for the execution of that one or more job requiring it.

In a further embodiment of the present invention it is provided a system comprising components adapted to implement the method above.

In another embodiment a computer program is provided which realizes the method above when run on a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example only, by reference to the accompanying drawings, in which:

Figure 1 shows an example of a computer network to which the method of the present invention is applied; Figure 2 is a diagram of a general computer system adapted to support the job scheduling method of the preferred embodiment;

Figure 3 is a block diagram of a software architecture of a scheduling system of the preferred embodiment of the present invention;

Figure 4 shows a flowchart representing the steps to perform a method according to a preferred embodiment of the present invention.

DESCRIPTION OF A PREFERRED EMBODIMENT

As shown in Figure 1, the preferred embodiment 100 comprises a scheduling system 101 which is connected through a network (e.g. a LAN) to a number of system resources servers 105 which control the resource access according to system requests to schedule execution of a plurality of jobs. The scheduling system 101 can be for example a Tivoli Workload Scheduler for z/OS controller application handling the planning and the control of the workload to be run on the system resources (holding info such as calendar and time schedules definitions, workload and resources properties, job running attributes, and the like) , while the system resources servers 105 can be the CPUS (z/OS system or workstations) where the unit of work (jobs) are executed. The method according to a preferred embodiment of the present invention leverages on virtual machines repositories, and let a scheduler associating to a specific task planned for execution, also the required specific virtual environment to use from the repository. When building the scheduler plan, it will be defined the job along with the virtual environment it needs to use. At scheduled time this request is processed by the scheduler and the pre-packaged environment is retrieved and started in advance to allow the workload to be executed according to the plan. Info like the normal (or average, max) time to download the virtual environment from the repository and the time to bring up (or to deploy) the virtual system environment should be made available in order to properly have all ready when needed. The advantage is to use the virtualization technology integrated with the scheduler, to improve the scheduler efficiency and virtual resource use. The specific required environment is made available just for the time it is requested by the specific workload which is requiring it to run on, according to the scheduling plan.

With reference to Figure 2, a generic computer of the system (e.g. computer, Internet server, router, remote servers) is denoted with 250. The computer 250 is formed by several units that are connected in parallel to a system bus 253. In detail, one or more microprocessors 256 control operation of the computer 250; a RAM 259 is directly used as a working memory by the microprocessors 256, and a ROM 262 stores basic code for a bootstrap of the computer 250. Peripheral units are clustered around a local bus 265 (by means of respective interfaces) . Particularly, a mass memory consists of a hard-disk 268 and a drive 271 for reading CD-ROMs 274. Moreover, the computer 250 includes input devices 277 (for example, a keyboard and a mouse) , and output devices 280 (for example, a monitor and a printer) . A Network Interface Card 283 is used to connect the computer 250 to the network. A bridge unit 286 interfaces the system bus 253 with the local bus 265. Each microprocessor 256 and the bridge unit 286 can operate as master agents requesting an access to the system bus 253 for transmitting information. An arbiter 289 manages the granting of the access with mutual exclusion to the system bus 253. Similar considerations apply if the system has a different topology, or it is based on other networks. Alternatively, the computers have a different structure, include equivalent units, or consist of other data processing entities (such as PDAs, mobile phones, and the like) .

Moving to Figure 3, the main software components that can be used to practice the method of the invention are depicted. The information (programs and data) is typically stored on hard-disks and loaded (at least partially) into the corresponding working memories when the programs are running, together with an operating system and other application programs (not shown in the figure) . The programs are normally initially installed onto the hard disks from CD-ROMs.

Particularly, a central scheduling application controls the execution of different jobs; for example jobs can consist of payroll programs, cost analysis applications and the like. The central scheduling application is composed of a job scheduler 301 for managing execution of a series of non-interactive jobs (typically during the evening); for example, the jobs can be payroll programs, cost analysis applications, and the like. The scheduler 301 accesses a workload database 303, which stores information about the different jobs to be executed. For each job, the workload database 303 includes a description of the corresponding steps and any dependency from other jobs or resources of the system as well as any requirements of a specific Virtual Environment (including for example possible "virtual appliances, or disk images, e.g. proprietary .vmx or .vmdx images); moreover, the workload database 303 stores a record indicating an estimated duration of the job. The scheduler 301 also includes a builder module 305 which creates a Scheduling Plan 307. The Scheduling Plan 307 has the function of dictating the succession of execution of jobs. According to a preferred embodiment of the present invention, the Scheduling Plan 307 is created by the builder module 305 taking into consideration the possible Virtual Environments required by one or more jobs included in the plan. When defining the Scheduling Plan 307, in case a special virtual environment is requested to execute a specific job, the builder module 305, through a Virtual Environment Request Handler 309, selects the suitable Virtual Environment from a Repository 311 which contains all necessary information to enable the scheduler setting up the correct environment for the job requesting such Virtual Environment. One of the information available in the Repository 311 is the time necessary to install and make it ready for execution each Virtual Environment. In this way the Builder 305 can organise the Scheduling Plan 307 with additional entries (i.e. the "shadow plan") so that all the activities necessary to install and run the Virtual Environment are performed in good time in order to be ready when the job needing that Virtual Environment is scheduled for execution. This is important to properly achieve the planned workload start date/time. As an example let's suppose a script A has to run on a special Linux virtual system (at a certain OS Kernel level which includes some special applications), at 5.00 PM; the scheduler determines from the information available on the Repository 311 that e.g. 15 minutes are necessary to have the environment in place. When building the Scheduling Plan, the builder module will arrange for the Linux virtual system set up program to start in advance (in our example at least 15 minutes) to ensure that at 5:00 PM all is ready to run the script A. To avoid that a delayed activity could prevent the Virtual Environment to be ready, a buffer time period could be added to the scheduled time, for example 10 min. According to the present example the scheduled time to start the preparation of the Linux Virtual Environment would be 4:35 PM (i.e. 5:00 PM - 15 min -10 min). In this way, at 5:00 PM all is running and ready and the script A can be executed without any further delays. If there are no other requests planned for the same Virtual Environment it is closed and uninstalled soon after (or it might be kept active for a certain pre-defined window to allow re-runs for example) . The evaluation of the additional buffer time, as mentioned above, may be based on a percentage of the requested time. For example defining the additional buffer as 50% of the requested time, results in an increase of the time needed by 50%: if the requested time from the repository is 30 min, it is increased by 15 min, so that the requested time to be ready results in a total time of 45 min.

In case the virtual environment repository is not accessible from the system where it has to run (for architecture, for safety reasons, or in case the link might go down) , the requested Virtual Environment system ZIP file, normally downloaded by the scheduler from the repository, can be directly attached to the plan (as the work units are) at plan creation time, and sent to the real system together with the workload (e.g. the script) and with the plan, in order to realize a fault tolerant scheduling architecture. Since virtual system environments are usually big and heavy in terms of memory and download times, the link from the repository may be stored, instead of the real source machine ZIP. As part of the environment info from the repository there must also be the info useful to evaluate how much time it is requested to retrieve that virtual environment from the library, and the time requested to let it start.

According to a preferred embodiment of the present invention the Scheduling Plan 307 contains a sort of "shadow plan". When adding into the scheduling plan an entry for a workload requiring a "special environment", when the plan is build a new additional entry is automatically added to the scheduler plan. It will schedule all the activities related to the start up and set up of the special environment. This considering the time requested for these preliminary operations to complete, in order to ensure that when it is time to run the job workload, everything is ready and properly set. It is a parallel plan ("shadow plan") created to automatically handle the special environment. Optionally another entry can be added to shut down the special environment when it is no more needed.

Of course there is a dependency between the "shadow plan" and the normal workload plan that has originated this additional plan entry. The following "normal" run is dependent on the completion of the preliminary start up and set up operations. This dependency is automatically set when the plan is originally created with the "special environment" option. The verification of the dependencies in the scheduler will assure that all is set in place when needed. As mentioned above the plan can also be arranged in order to remove the virtual environment when no more needed. The deletion of the virtual environment will be dependent on the completion of the "normal" runs requiring the environment.

Figure 4 schematically shows the method steps according to a preferred embodiment of the present invention. The process starts at step 401 and goes to step 403 where the scheduler start building the scheduling plan as described above. During the creation of the scheduling plan, the builder module (see description with reference to Fig 3) accesses a database where all the requirements of each job are defined. If such requirements include the need of a specific environment and more particularly a virtual environment (step 405) , the builder retrieves the information about the specific virtual environment in a Virtual Machine Repository (step 407) . The information which is useful to the builder module includes all the system specification and also an indication of the time required for such environment to be correctly installed and set up. At step 409, according to the retrieved information, the plan is customized: the builder creates a "shadow plan" which is used to adjust the plan, so that when a job requiring a virtual environment is due to start the necessary virtual environment is ready for use and there is no delay in starting the job. Then the process goes to step 411 where it is verified if any more jobs require the setting up of specific environments and, if this is the case, the process goes back to step 407. Otherwise the plan can be executed, when all other conditions for execution will be satisfied (e.g. a specific moment in time) . The method according to a preferred embodiment of the present invention has been described with reference to virtual environment, however other kind of specific requirement might be necessary for a job to properly execute: in such case, instead of information about a virtual environment, other kind of information could be retrieved by the builder and the shadow plan could be created according to these information .

Alterations and modifications may be made to the above without departing from the scope of the invention. Naturally, in order to satisfy local and specific requirements, a person skilled in the art may apply to the solution described above many modifications and alterations. Particularly, although the present invention has been described with a certain degree of particularity with reference to preferred embodiment ( s ) thereof, it should be understood that various omissions, substitutions and changes in the form and details as well as other embodiments are possible; moreover, it is expressly intended that specific elements and/or method steps described in connection with any disclosed embodiment of the invention may be incorporated in any other embodiment as a general matter of design choice.

For example, similar considerations apply if the computers have different structure or include equivalent units; in any case, it is possible to replace the computers with any code execution entity (such as a PDA, a mobile phone, and the like) .

Similar considerations apply if the program (which may be used to implement each embodiment of the invention) is structured in a different way, or if additional modules or functions are provided; likewise, the memory structures may be of other types, or may be replaced with equivalent entities (not necessarily consisting of physical storage media) . Moreover, the proposed solution lends itself to be implemented with an equivalent method (having similar or additional steps, even in a different order) . In any case, the program may take any form suitable to be used by or in connection with any data processing system, such as external or resident software, firmware, or microcode (either in object code or in source code) . Moreover, the program may be provided on any computer-usable medium; the medium can be any element suitable to contain, store, communicate, propagate, or transfer the program. Examples of such medium are fixed disks (where the program can be pre^¬ loaded) , removable disks, tapes, cards, wires, fibres, wireless connections, networks, broadcast waves, and the like; for example, the medium may be of the electronic, magnetic, optical, electromagnetic, infrared, or semiconductor type.

In any case, the solution according to the present invention lends itself to be carried out with a hardware structure (for example, integrated in a chip of semiconductor material) , or with a combination of software and hardware .

Claims

1. A method of creating a plan of execution for scheduling a plurality of work units for execution on a data processing system, at least one of the plurality of work units requiring a set of system conditions to be satisfied before the execution can start, the method including the steps of:

- maintaining in a repository information about the set of the system conditions for the at least one work unit requiring a set of system conditions to be satisfied before the execution can start, the information including an indication of the time required for performing those plurality of activities necessary for satisfying the system conditions;

- for each of the at least one of the plurality of work units requiring a set of system conditions to be satisfied before the execution can start:

- retrieving the information in the repository;

- customizing the execution plan according to the indication of time, so that at the scheduled time for execution of the work unit, the system conditions are satisfied;

- submitting the plan for execution.

2. The method of claim 1 wherein the set of system conditions include the availability of a virtual environment .

3. The method of any preceding claim, wherein the plurality of activities include installing a set of software applications.

4. The method of claim 3, wherein the step of customizing the execution plan includes:

- adding a shadow plan comprising additional steps for installing the set of software applications.

5. The method of claim 3 and 4, wherein the step of customizing the execution plan includes:

adjusting the execution plan, so that after execution of the work unit the set of software applications is un-installed .

6. The method of any preceding claim, wherein the step of customizing the execution plan includes:

adjusting the execution plan, so that at a target time before the scheduled time for execution of the work unit, the system conditions are satisfied, wherein the target time is the scheduled time minus a predetermined period of time.

7. System for creating a plan of execution for scheduling a plurality of work units for execution on a data processing system, wherein the system comprises a one or more components adapted to perform the method of any claim 1 to 6.

8. Computer program comprising instructions for carrying out the method of any claim 1 to 6 when said computer program is executed on a computer system.

9. A service deployed in a data processing system for implementing the method of any claim 1 to 6.