RU2411574C2 - Intellectual grid-system for highly efficient data processing - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: core of intellectual Grid-system is a server, which selects user programs from relation database, starts these programs at remote highly efficient computer resources, saves results of programs operation in the form of files in database for interaction with server and processing of its commands at each remote computer resource, tracks and notifies the server about the resource status and programs started on it, provides for storage of files with design programs, initial data and results, maintains queues of tasks on the basis of user criteria, tracks conditions of accessible computer resources, transfers user programs to computer systems, reception and preservation of files with produced results.

EFFECT: higher efficiency of functioning of parallel calculations in solution of complex resource-intensive tasks.

12 dwg

Description

The invention relates to computer tools for high-performance data processing in solving complex problems of analysis and interpretation of information in the conditions of separation of computing and information resources based on an open intelligent Grid system.

Known patent JP 9114980, 1997.06.02 "Method and program for selecting GRID executor via neural", which proposes a method for grid-execution of operations using a neural network model, the training of which is based on structured data received at the input and output of a neural network, and A trained neural network makes a selective choice of the sequence of functioning of the Grid-network and at the output it gives out the intensity of service.

An analogue of the Grid system under consideration is the program “Remote control system for distributed computing resources” (see certificate No.2008611206 (03/07/2008). The remote control system is designed to combine geographically distributed high-performance computing systems (computer clusters) into a single system that provides ease of administration and the use of these resources.The core of the system is a server that selects user programs from a relational database, launches this programs on remote high-performance computing resources and saves the results of programs in the form of files in a database for interacting with the server and processing its commands on each remote computing resource, a software agent is installed that monitors and reports to the server the status of the resource and the programs running on it. storing files with calculation programs, initial data and results; support for the job queue based on user criteria; monitoring the status of available computing resources; transfer of user programs to computer systems; receiving and saving files with results.

The disadvantage of this environment is the lack of effective procedural and logical components for use in the Grid environment, which differ significantly from the traditional approaches of cluster systems.

A closer analogue of the system under consideration is the “PEG-2” toolkit for designing high-performance applications for Grid architectures (see certificate No. 20088614623 09/25/08), which considers the resource interpretation of the Grid application as an environment that provides the user with access to subject-oriented resources in the form of applied grid services. A distinctive feature of the design and development of grid services is the orientation to the use of resources such as off-the-shelf components with a description of their interaction in the notation of the work flow. In contrast to the flowcharts of the algorithms in the PEG-2 system, the sequence of operations is implicitly set and obeys the principle of transferring control by the availability of source data.

The disadvantage of the PEG-2 tool shell is the limited functioning within the corporate Grid environment due to the imperfect organization of the functioning logic, as well as the absence of a control program that provides intelligent decision support in a complex dynamic environment in the face of heterogeneous computing resources, uncertainty in the characteristics of the task, and incomplete initial information.

To ensure the availability of software tools for a distributed Grid system under the growing complexity of the infrastructure and the stochastic variability of the parameters of communication links and computer systems, it becomes necessary to develop an application that provides the integration and synchronization of a large number of computer systems, the construction of applied Grid services and the management of the computing process based on intelligent technologies .

The technical result of the invention is to increase the efficiency of parallel computing in solving complex demanding tasks using functional blocks that implement a logical inference mechanism based on the principle of adaptive resonance, search and retrieve "hidden" knowledge and patterns, a dynamic knowledge model in the organization of competing computing technologies and interaction with by user using an intelligent interface.

The indicated technical result is achieved by constructing an intellectual problem-oriented Grid system based on system integration of computing and information components. The formalized logic for managing the solution of resource-intensive tasks in the study of complex phenomena and patterns of dynamic systems, as well as the functioning of virtual organizations and polygons, allows you to perform calculations and modeling based on an expert system of intelligent decision support, an adaptation unit that provides the choice of the preferred computing technology for data processing, setting logical models for the perception of new information and the extraction of "hidden" knowledge, an intelligent interface and supporting user interaction with the computing environment intelligent grid system.

Functional diagram of the system is presented in figure 1.

The system includes 4 main blocks: a block of intellectual support for the functioning of the Grid system 1, a program control block 2, a block of human-computer interaction 3, a block of applied Grid services 4.

The block of intellectual support 1 (figure 2) is the main element that ensures the functioning of the computer complex. The block includes an expert system 5, a composite application generator 6, and an adaptation block 7.

The program control unit 2 (Fig. 3) implements the functions of intelligent decision support when performing high-performance computing in an intelligent Grid system. Based on the initial information received to complete a complex task, block 2 performs a semantic search for the description of task 8 and, interacting with the knowledge base of the expert system 5, receives information about the available services and their ontological descriptions 4 and calls the interpreter 9, balancing algorithms 10 and Grid Application Runtime Prediction Element 11.

The unit of human-computer interaction 3 (Fig. 4) performs the functions of input and output of information and contains four basic components: an intelligent interface unit 12, which carries out the development and launch of a task, an information presentation unit 13, a visualization unit 14, and a documentation unit 15.

The block of applied grid services 4 (Fig. 5) provides the execution of a user task on remote target systems and includes additional functional elements: a virtual design shell 16, a presentation element (notation) 17 in the form of a set of blocks corresponding to the main operations of applied grid services and relationships between them, which determine the exchange of data, the element of balancing and scheduling 18, the scheduler 19, which, based on knowledge of services and source data, creates a set of alternative methods for posting oeniya choices and decisions and schedules.

The expert system of subject areas 5 (Fig. 6) ensures the functioning of the Grid system in a given computing environment and includes the following knowledge bases: a knowledge base of a subject-oriented description of services 20, a knowledge base of performance characteristics of computing services 21, a knowledge base of performance characteristics of basic operations models and interactions between them 22, the knowledge base of the characteristics of the reliability of the interpreted knowledge 23, the inference mechanism based on the principle of adaptive resonance 24, the system explanations 25, the system-wide database 26. The knowledge bases of the expert system 5 contain sets of concepts of the subject area in the form of frame-production models, and the model for representing subject knowledge is implemented using an ontology in the form of a semantic network, the set of nodes of which corresponds to the facts, and the set of arcs reflects logical connections between them.

The logical conclusion based on the principle of adaptive resonance (block 24) implements the block diagram shown in Fig.7. A dynamic knowledge model using this principle is focused on the restructuring of logical models and the formalization of information contained in knowledge bases 20-23, under conditions of uncertainty of input data, stochastic parameters of the Grid environment, and weak formalization of the problem statement. In the process of functioning of block 24, a check is made of the correspondence of the initial data to the logical system of frame-production models and the subsequent correction of the logical rules of the expert system 5, associated with the modification of existing rules, or the construction of new rules corresponding to the initial data.

The adaptation unit 7 (Fig. 8) implements adaptive learning procedures due to the ability to control the computing process with dynamically changing information. This ensures interaction with the database 26 of the expert system 5, containing applied computer programs for calculating and modeling, and the block of applied grid services 4. In addition, the adaptation block 7 implements the principle of competition when choosing the preferred computing technology 27, supporting the search and retrieval process " hidden "knowledge and patterns 28 (data mining procedures - Data Mining), especially the integration of knowledge in the face of heterogeneous computing resources, is stochastic the variability of the parameters of communication networks and computer systems, the uncertainty of the characteristics of the problem and the incompleteness of the initial information.

The algorithm for implementing the principle of competition (block 27) in solving complex problems in the process of functioning of the intellectual grid system consists in performing the following steps:

Step 1. Generation of alternative solutions to the problem using the composite application generator 6.

Step 2. The selection of competing computing technologies based on semantic search for the description of task 8 in the form of standard algorithms (block 17) and neural network models (block 35).

Step 3. Parallel solution of the problem using the interpreter 9, balancing algorithms 10 and intelligent interface 12.

Step 4. Analysis of alternatives and the choice of the preferred solution using the scheduler 19 and block 24, which implements the inference mechanism based on the principle of adaptive resonance.

The operations to perform steps 1-4 are carried out in block 36 using an expert system 5 and an intelligent interface 12.

Another operation performed in adaptation block 7 is associated with revealing “hidden” knowledge and patterns 28 (Fig. 9) based on statistical analysis 29, Bayesian network 30, hierarchy method 31, decision trees 32, reduction principle 33, ontology 34, artificial neural networks 35, formal conceptual analysis 36.

The intelligent interface unit 12 (Fig. 10) is a software-analytical complex that provides "transparency" of the meaning of access to information while maintaining user interaction using the program control unit 2, adaptation unit 7 and documenting unit 15. The intelligent interface 12 is focused on formalizing concepts subject area and includes four main modules: a parser 37 that performs the procedures of morphological and syntactic analysis of sentences of input information , a semantic analyzer 38, using the results of parsing to formalize the information content of the input data, a conceptual dictionary processor 39, which is a library of functions that compute the characteristics presented in the conceptual dictionary and a complete set of relations between them, as well as a fixed set of associative relations of the form " part - the whole "," device - function ", a processor of the database reference guide 40, which characterizes the models of subject areas that determine the functional Contents Intellectual Grid system using adaptation means 7 and the software of the control unit 2. In complex situations through an intelligent interface 12 is implemented relationship adaptation unit 7 with intellectual support unit 1, providing functionality Intellectual grid system.

The modular principle of constructing an intelligent Grid system of high-performance data processing is implemented on autonomous storage media using independently functioning units: an intellectual support unit of the Grid system 1, a program control unit 2, a human-computer interaction unit 3, an application Grid services unit 4. Tool environment intellectual Grid system, as a set of computing resources using problem-oriented Grid services 4, provides formalism the process of designing composite computing applications that achieve optimal performance using intelligent technologies.

Implementation of an intelligent Grid system for high-performance data processing is carried out using standard technical means of high-performance computing using the Jawa 5 programming language. The PEGI shell implements the main functionality that defines the interaction with the infrastructure of an intelligent Grid system based on Intel Grid Programming Emvironment (GPE). The procedure uses the GPE API libraries version 1.5, which are an add-on for the Globus environment. This solution allows you to isolate the implementation of the instrumental shell from the architectural features of the intellectual grid system. The execution element in the PEG2 shells provides the implementation of user task 4 on remote target systems: by calling the interpreter 9 WF (workflow), balancing algorithms 10 and monitoring using the smart interface 12. Additional functional elements are introduced into the tool shell, in particular, the visual design shell 14 , the presentation element WF 13, the monitoring element of the intelligent Grid system 12 and the element for predicting the runtime of applications 11. Visual olochka 3, 14 implements a process of human-computer interaction in graphical form. The WF element is essentially an interpreter 9 of the user-defined problem statement 12 for further use by other elements of the software package 1-4. Elements of monitoring the environment of the intelligent Grid system 1 and predicting the runtime of the composite application 11 are used by the balancing and scheduling element 10 to create a specific workflow - CWF that has the highest performance.

A generalized scheme of the functioning of the intellectual grid system is presented in Fig. 11. At the design stage, the user through the visual shell interprets the composite application in the form of MWF (meta workflow). In this case, knowledge about the corresponding set of available Grid application services 4, which are provided by the knowledge management system Grid Knowledge Manager 2, is used. This description is transmitted to the Workflow Planner 19, which, based on knowledge about services 20, 21, 22, 23 and data 26, creates a set Alternative Workflow - AWF. Each AWF is a consistent computing process with a fixed application grid service and raw data. Based on the AWF set, the Workflow Scheduler 18 builds a set of specific execution schedules (CWFs). The schedule that most closely matches the user criteria is finalized and submitted for execution. Upon completion of the tasks on the target Workflow Executor systems using the expert system 5 and the intelligent interface 12, the work results are collected and the final result is provided in the form of a data file 13. If unusual situations arise when solving large problems in a complex dynamic environment under heterogeneous conditions computing resources, the sequence of functioning of the intellectual grid system described above is supplemented by procedures that implement the logic conclusion based on the principle of adaptive resonance 24 and procedures for identifying “hidden” knowledge 28 with the support of an intelligent interface 12.

As an example of the practical use of the developed intellectual technology, one of their complex and time-consuming tasks is considered below on the example of processing geophysical information - analysis of the climatic spectra of sea waves. The results of the functioning of the intelligent Grid system in solving this problem is illustrated in Fig. 12.

Here are the characteristics of the processing time of an array of 8 thousand spectra obtained on an intelligent grid system consisting of 24 target systems. In the process of solving the problem, a statistical generalization of the given functions of the distribution of wave energy in frequencies and directions at a fixed point in space over a long period of time was carried out. The data shown in Fig. 13 illustrates the dependence of the task execution time on the number of calculators in the form of nuclear estimates of the distribution density of the operating time for three competing parallel computing technologies (schedule schemes): uniform circuit 1, direct 2, and inverse 3 cascade schemes. It can be seen that with an increase in the number of calculators (p = 2 ÷ 4), the spread in the execution time increases, which is associated with an increase in the influence of stochastic communication effects in the considered Grid system. In this case, the relative position of the distribution densities is determined by the rules for ranking competing schedules. For example, in case "A" (Fig. 12) for p = 2, the forward and reverse cascade circuits lead on average to almost identical estimates of the operating time. The difference between them is much smaller than the corresponding range of variability, however, the variation in operating time for a direct cascade circuit is 1.5 times greater than for a reverse cascade circuit. Therefore, the final choice of the parallelization scheme is determined based on the strategy: less risk - less productivity or more risk - more productivity.

On the other hand, in the process of performing the task, there are possible cases (situation “B” at p = 24), when the distribution densities of the operating time for different schemes overlap, however, the difference between the average values is comparable with their range of variability. Then the ranking of competing schedules must be carried out taking into account a given level of significance of the error. In particular, in situation “B”, in 5% of cases, the reverse cascade circuit will give a better result than the uniform one, despite the fact that on average the uniform circuit works 1.5 times faster. 12 also shows that the performance curves have a pronounced minimum: for cascade circuits at p = 6, for uniform circuits at p = 12, which determines the optimal mode of task execution. The maximum acceleration on average is about 2.5, and the gain in comparison with competing cascade schedules is 2 times, which is typical for Grid systems due to the significant contribution of the communication component as one of the determining factors (along with the amount of available memory, disk space and etc.),

Thus, the implementation of the intelligent Grid system provides a flexible, coordinated, separate use of various computer resources of remote virtual organizations as a community of users with dynamically changing composition for sharing distributed Grid resources within the framework of common scientific and technical goals and objectives.

Claims (1)

Intelligent Grid system for high-performance data processing, containing a formalized logic of the functioning of a set of geographically distributed high-performance computing systems, combined into a single system that provides ease of administration and use of resources, the core of which is a server that selects user programs from a relational database, runs these programs on remote high-performance computing resources and saves work results A program in the form of files in the database for interacting with the server and processing its commands on each remote computing resource, monitors and reports to the server the status of the resource and the programs running on it, provides storage of files with calculation programs, initial data and results, supports job queues based on user criteria, monitors the status of available computing resources, transfers user programs to computing systems, receives and stores files with received p results, characterized in that the intellectual problem-oriented Grid system is implemented on the basis of system integration of computing and information components, the formalized control logic of which is associated with solving resource-intensive problems in the study of complex phenomena and patterns of dynamic systems, as well as in the functioning of virtual organizations and polygons at performing complex calculations and modeling, for which an intellectual support function block was additionally introduced into the computer complex ionization of the Grid system, interconnected by means of a program control unit with human-computer interaction units and a block of applied Grid services, and the intellectual support unit contains an expert system that ensures the operation of the Grid system in a given computing environment and decision-making on the management of computing processes, adaptation unit that implements adaptive learning procedures due to the ability to control a computing process with dynamically changing information, choice preferred computing technology for data processing, setting logical models to perceive new information and extracting “hidden” knowledge, a composite application generator that implements the functions of developing alternative solutions, a human-computer interaction unit containing an intelligent interface that supports user interaction with the computing environment under conditions of heterogeneous computing resources , the uncertainties of the characteristics of the problem and the incompleteness of the source information.

Images