WO2025063857A1 - Method and system for moving data in a cloud environment - Google Patents

Abstract

A method for moving data in a cloud environment includes receiving a request to move data from data sources to a target source, generating a sequence of data movement operations with the aid of an API service, and sending said sequence to a task scheduler. The sequence of operations is added to a task queue of a worker node in accordance with a given priority parameter, a set of interfaces for working with a data source is determined, and the possibility of performing the task using the resources available is checked. An optimal route for movement of the data is then determined and the data are read from an initial data source and transferred to a target data source in accordance with the determined route. The technical result is an increase in the reliability of data movement.

Description

METHOD AND SYSTEM FOR MOVING DATA IN A CLOUD ENVIRONMENT

AREA OF TECHNOLOGY

[0001] The invention relates generally to the field of data storage and processing, and in particular to a method and system for moving data in a cloud environment,

5 ensuring timely, uninterrupted delivery and movement of large volumes (from several terabytes) of structured and/or unstructured and/or heterogeneous (multimodal) data, including for the purposes of training machine learning models.

LEVEL OF TECHNOLOGY 0 [0002] A solution for managing data storage is known from the prior art, disclosed in patent US 8156086 B2, published 10.04.2012. In the known solution for managing data storage, the following is carried out:

- performing a first storage operation on a first set of data at a first location to generate a plurality of data blocks, wherein said performing a first storage operation for at least one of the plurality of data blocks further includes the steps of: generating a second set of data associated with the first set of data; creating a first set of metadata associated with the first set of data; and storing the second set of data and the first set of metadata on a storage device;

20 - storing a copy of the first set of metadata in an index remote from the storage device;

- generating a second set of metadata associated with a second set of data stored on the storage device;

- comparing the first set of metadata in the index with the second set of metadata to check for differences between the first and second sets of data without access to the first and second sets of data;

- performing a second operation of storing a second set of data, the second storage operation includes the steps of: generating a third set of data associated with the second set of data; generating a third set of metadata containing information for retrieving the third set of data; and storing the third set of data in a second location; and - compare the first set of metadata in the index with the third set of metadata to check if there are differences between the first and third sets of data.

[0003] Also known are a method and system for multiplexing pipeline data for the purposes of creating backup copies, described in patent US 7315923 B2, published 01.01.2008. In the known solution, a first data stream is received, containing first data, wherein the first data is received by a first data agent specific to a particular application; a second data stream is received, containing second data, wherein the second data is received by a second data agent for a particular application; the first and second data streams are combined into a single stream from one or more data blocks of an archive file, including recording the first data from the first data stream and the second data from the second data stream into the first data block of the one or more data blocks; transmitting one or more data blocks via a transport channel to a backup medium; and storing one or more data blocks on the backup medium.

[0004] Also known are a method and an information management system described in application US 2020/0319694 A1, published 02/14/2023. The known system comprises: a network storage system comprising computer hardware configured to: configure power management parameters for a media agent, wherein the media agent is a component in the information management system that interacts with secondary storage devices to perform operations with data stored using the secondary storage devices; identify, using a power management module, a task to be performed in the network storage system, comprising an operation type in which the task is associated with the media agent; based, at least in part, on the operation type of the task, determine a power state for the media agent associated with the task; and upon determining that the power state of the media agent is at or below a specified level: add the task to a task list associated with the media agent, wherein the task list comprises tasks of the operation type; determining whether the task list exceeds a specified threshold; and based on the power management module determining that the specified threshold is exceeded, directing the media agent to execute one or more tasks in the task list. [0005] Also known are systems and methods for creating single instances of data blocks in a data storage system, described in patent US 8578120 B2, published 05.11.2013. This document mentions a computing system comprising: one or more storage devices stored on a physical medium; one or more logical containers including several deduplicated data blocks corresponding to data objects; and one or more data structures indicating whether the data blocks are referenced; one or more databases storing information indicating whether the data blocks are referenced; and a secondary storage computing device programmed to: receive instructions for deleting a first set of data blocks from a first logical container; for each of the data blocks in the first set, determine from the databases a reference to the data block; and if the data block is not referenced, update the data structure to indicate that the data block is not referenced; determining, based on data structures, that a threshold number of contiguous data blocks in the first logical container that are not referenced has been reached; and providing access for storing a portion of one or more physical media corresponding to contiguous data blocks in the first logical container, wherein the data structures and databases are not part of the native file systems of the storage devices.

[0006] Also known are data storage systems and methods described in patent US 7343453 B2, published 11.03.2008. This document discloses a hierarchical data storage system that provides methods for assessing the state of stored data relative to consumer needs using weighted parameters that can be defined by the user.

[0007] The development of technologies and methods for processing big data on powerful computing complexes and in a cloud environment additionally requires the implementation of new mechanisms for managing large data sets due to the increase in transmission time due to the limited network resource and throughput of data storage, and ensuring the readiness of data at the required point in time when creating continuous data processing pipelines. DISCLOSURE OF INVENTION

[0008] The technical problem or task posed in this technical solution is the creation of a simple and reliable method and system for moving large volumes of data in a cloud environment.

[0009] The technical result is an increase in reliability when moving data from different data sources in accordance with a specified time by which the said movement should be performed, and the conditions for moving the data.

[0010] The specified technical result is achieved by implementing a method for moving data in a cloud environment, comprising the steps of:

- a request is received via API service 20 for moving the first data set from the first data source (ID) 30 and the second data set from the second ID 31 to the target ID 40, containing at least information about the conditions for moving the data; a request is generated via API service 20, containing a sequence of operations for moving the data, as well as a parameter of the priority of moving, determining the order of execution of the moving operation, and the generated request is sent to the address of Task Scheduler 50;

- add a new task to move data to the queue using Task Scheduler 50 in accordance with the mentioned priority parameter;

- when a task is received from the execution queue, the task is transferred to Work Node 60;

- define, by means of Work Node 60, a list of interfaces for working with ID 30, ID 31 and ID 40;

- perform, via Work Node 60, an assessment of the possibility of carrying out the data movement task based on the available resources;

- determine by means of the Work Node 60 the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in the said IDs;

- determine by means of Work Node 60 the optimal route for moving data from ID 30 and ID 31 to ID 40; - check the operability of the IDs used to move data using the Working Node 60;

- by means of the Work Node 60, the first and second sets of data are read from the ID 30 and ID 31 and the said data are transferred to the address of the target ID 40 in accordance with the conditions of data movement.

[0011] In one particular example of implementing the method, ID 30, ID 31 and ID 40 are in the same cloud (i.e., logical pool).

[0012] In another particular example of implementing the method, at least one ID 30, ID 31 or ID 40 is located in another cloud relative to the said IDs. [0013] In another particular example of implementing the method, data transmission to the address of the target ID 40 is carried out taking into account the error correction code, data increment or parallel data transmission parameters.

[0014] In another particular example of implementing the method, an additional step is performed in which the registration of all data sources participating in the data movement is checked.

[0015] In another particular example of implementing the method, the request for data movement contains information about at least one additional target ID to which the first and second sets of data should be moved, and the Task Scheduler 50 generates several tasks for data movement with the same priority parameter depending on the number of additional IDs.

[0016] In another particular example of implementing the method, the following steps are additionally performed by means of the Task Scheduler 50: extracting from the data movement request a time value before the expiration of which the data movement should be completed; extracting from memory data characterizing the execution time of similar tasks for movement along a given route; determining, on the basis of the extracted data, the predicted execution time of the movement task; rearranging the data movement task in the queue for its execution at the specified time value, taking into account the predicted execution time of other tasks contained in the queue and the values of their priorities.

[0017] In another particular example of implementing the method for determining the optimal route by means of the Work Node 60, the following steps are performed: determining the route from ID 30 and ID 31 to ID 40, as well as from the IDs in which copies of the data of ID 30 or ID 31 that should be moved are stored to ID 40; select the optimal route for moving data, taking into account the IDs in which copies of the data are stored.

[0018] In another particular example of implementing the method, an additional step of checking the integrity of the transferred data is performed.

[0019] In another particular example of implementing the method, the Work Node 60 additionally performs the step of synchronizing the completion of the process of moving data from ID 30 and 31, including taking into account the increment.

[0020] In another particular example of implementing the method, the Work Node 60 additionally performs version control of the moved data.

[0021] In another particular example of implementing the method, metrics associated with the data movement operation and resource usage are additionally recorded.

[0022] In another particular example of implementing the method, the data transport operation is performed in accordance with a specified schedule. [0023] In another particular example of implementing the method, the first or second data sets contain structured and/or unstructured data and/or heterogeneous (multimodal) data.

[0024] In another preferred embodiment of the claimed solution, a data movement system in a cloud environment is presented, comprising at least three IDs, an API service, a Task Scheduler, at least one Worker Node, and the system is configured to implement the above-mentioned method.

BRIEF DESCRIPTION OF DRAWINGS

[0025] The features and advantages of the present technical solution will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

- Fig. 1 shows the general scheme of registration of data sources;

- Fig. 2 shows a general diagram of the data movement system;

- Fig. 3 and 4 show the general scheme of data processing;

- Fig. 5 shows an example of the general appearance of a computing device. IMPLEMENTATION OF THE INVENTION

[0026] Below, the concepts and terms necessary for understanding this technical solution will be described.

[0027] In this technical solution, the term “system” means, among other things, a computer system (in particular, an information system, computing complexes, computing clusters), a computer (electronic computer), a CNC (computer numerical control), a PLC (programmable logic controller), computerized control systems, specialized devices implementing calculations using FPGA and ASIC technologies, and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[0028] A command processing unit is an electronic unit, computing device, or integrated circuit (microprocessor) that executes machine instructions (programs).

[0029] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices. The data storage device may include, but is not limited to, hard disk drives (HDD), flash memory, ROM (read-only memory), solid state drives (SSD), and optical drives.

[0030] A program is a sequence of instructions intended for execution by a computer control unit or command processing device.

[0031] A database (DB) is a collection of data organized according to a conceptual structure that describes the characteristics of the data and the relationships between them, and such a collection of data supports one or more application areas (ISO/IEC 2382:2015, 2121423 “database”).

[0032] A signal is a material embodiment of a message for use in transmitting, processing and storing information.

[0033] A logical element is an element that implements certain logical dependencies between input and output signals. Logical elements are usually used to construct logical circuits of computers, discrete circuits of automatic control and management. For all types Logical elements, regardless of their physical nature, are characterized by discrete values of input and output signals.

[0034] Automated system (AS) is an organizational and technical system that ensures the development of decisions based on the automation of information processes.

[0035] In accordance with the diagram shown in Fig. 1, in the first stage, the data source 30 is registered in the data movement system in the cloud environment. The operation of registering the data source 30 is performed in advance before the start of the data movement both with respect to the source of the data being moved and with respect to the storage object, i.e. the target data source in which the data will be stored after the movement.

[0036] To register the data source 30, the user, via the user device 10, fills out a request form to the API service 20 using the user graphical interface 11, or in the command line mode (user UI). The user device 10 may be a portable or stationary computer, telephone, smartphone, tablet or other computing device equipped with wired and/or wireless communication means.

[0037] The request form contains the storage identifier (name or identifier), the data source type ("cold storage", "hot storage"), the file system type (HDFS, S3, NFS, etc.), authorization information (token, login-password pair). The completed form can be saved in the user UI module 11 for the purposes of repeated entry, as well as for the purposes of automated data processing.

[0038] Additionally, the user specifies the parameters for access to the data source 30 - the type of access (permanent, session or scheduled), encryption keys for organizing a secure connection, data transfer rate, data transfer protocol (transfer with or without confirmation of delivery, multi-channel transfer), data transfer mode (synchronous or asynchronous transfer).

[0039] Accordingly, the completed request form is sent by the device 10 to the API service 20, which, upon receiving the request, generates a request to the data source 30 to confirm its availability and sends the generated request to the address of the data source. For receiving and processing data, the API service 20 can be implemented on the basis of at least one server equipped with software and logical elements for performing the functions assigned to it. The API service provides interfaces for registering data sources and initializing the data transport operation, as well as interfaces to relevant metrics of operations performed by the System, maintains an event log, and transmits commands for performing tasks from the User to the Scheduler 50 and the Worker Node 60, as well as reports and notifications on the status of planned and performed operations.

[0040] Upon receipt of a request for availability confirmation, the data source 30 starts a verification procedure, including a source availability check (network availability/network connectivity check), compatibility check (data transfer format and protocol check), user authorization on the data source 30, checking for permission to access the data stored in the data source, specifying a set of access rights (data read only, data write only, data read and write, no permissions). The API service 20 waits for a response from the data source within a waiting period pre-set by the developer in the settings of the API service 20. For example, the waiting period may be from a fraction of a second to tens of seconds.

[0041] Data source 30 returns a response to API service 20 with the results of the check, including confirmation of network availability, confirmation of format and protocol compatibility, confirmation of user authorization, confirmation of permission to access data. API service 20 analyzes the content of the response taking into account the waiting time.

[0042] If the API service 20, after a specified waiting period, has not received a response from the data source confirming the availability of the data source over the network, the API service considers the data source unavailable and sends a message to the user device 10 about the unavailability of the data source for displaying it to the user via the UI 11.

[0043] If the API service 20 receives a response confirming incompatibility within a specified waiting period or does not receive a response from the data source 30 confirming compatibility of formats and/or data transfer protocols after the waiting period has expired, the API service 20 considers the data source 30 unavailable and sends a compatibility error message to the user device 10 when establishing a connection to display it to the user via the UI 11. [0044] If the API service 20 receives a response from the data source 30 within the specified waiting period containing information about the user's authorization refusal, the API service 20 sends a message about the authorization refusal (access refusal) to the user's device 10 for displaying it to the user via the UI 11.

[0045] If the API service 20, after a specified waiting period, has not received a response from the data source 30 confirming the user's authorization on the data source, the API service 20 considers the data source 30 unavailable and sends a message to the user device 10 about the unavailability of the data source 30 indicating the status "data source is not responding" for display to the user via U1 11.

[0046] If the API service 20 receives a response from the data source 30 within a specified waiting period containing information about a denial of access to the data stored on the source, the API service 20 sends a message about a denial of access to the data to the user device 10 for displaying it to the user via the UI 11.

[0047] If the API service 20 has not received a response confirming the set of access rights to the data stored on the source after a specified waiting period, the API service 20 sends a message to the user device 10 about the denial of access to the data for displaying it to the user via the UI 11. [0048] If all the requested confirmations are successfully received from the data source 30, the API service 20 registers the data source 30, after which the API service 20 sends a confirmation of the registration of the data source to the user device 10 for displaying it to the user via the UI 11.

[0049] After the required number of data sources 30 are registered, the process of moving/transporting data in the cloud environment using the registered sources can be started, and a one-time data transport between the registered sources or a scheduled data transport between the registered sources can be performed.

[0050] A one-time transport is a one-time operation in which the user does not plan to repeat the operation without changing the parameters and does not automate the process of repeat execution. A one-time transport is usually the main operation in the development and debugging data processing algorithms, as well as when performing data mining tasks. The data movement process will be described below with reference to Figs. 2, 3 and 4.

[0051] At the first stage (100), the user, by means of the device 10, forms a request for merging/moving the first data set from the first data source (DS) 30 and the second data set from the second DS (31) to the target DS 40, wherein DS 30, DS 31 or DS 40 may be located both in the same cloud (i.e., logical pool) with the other mentioned DS, and in different clouds. In an alternative embodiment of the presented solution, the user may specify several target DS 40 for parallel movement of the first and second data sets to several DS 40.

[0052] To generate a request, the user fills out a request form to the API service 20 using the user UI 11. In said request form, the user may specify: identifiers ID 30, ID 31 and ID 40 (e.g., ID name, IP address, etc.), type of transportation (one-time or scheduled), authorization information (tokens, login-password pairs), addresses and identifiers of objects containing the moved data, and a list of the moved data, addresses and identifiers of objects to which the data will be moved, parameters of the priority of the operation. The filled out form may be saved in the user module U1 11 for the purposes of scheduled transportation, as well as for the purposes of automated data processing. Information on the condition of the data movement is also specified (with the deletion of data from the source after the movement or the saving of copies of the data with an indication of the version of the data set, or without this).

[0053] The request form and the request may also contain fields for entering and transmitting such parameters as the need to use an error correction code, the transmission of an increment (addition) of data, the applied algorithm for checking the integrity of the data, conditions in the event of the execution of the task by the Scheduler when these conditions are met, information about the sources of metadata and instructions for the movement of metadata, parameters for parallel data transfer (movement of data with or without consolidation from several sources or movement to several target sources - data recipients).

[0054] For example, the user may request to combine at least one photograph stored in ID 30 and metadata, such as characterizing the properties of the file, namely - file name, creation date, file size, hash sum, etc., stored in ID 31 in ID 40. Accordingly, the mentioned data merging request will additionally contain:

- User ID;

- user authorization data;

- instructions for merging data: one-time or on a schedule;

- the parameter of the move priority, which determines the order of execution of the move operation;

- at least one condition selected from: the need to apply an error correction code, for example, to load the first or second data set from an alternative ID in which a copy of the data is stored; the transmission of an increment (addition) of data, for example, indicating the author of the first data set;

The ID of the applied data integrity verification algorithm; a condition for performing the data transfer, for example, which may indicate that the data transfer to ID 40 will be performed only if there is free space for storing the data or indicating a time value before the expiration of which the data transfer should be completed; information about the metadata sources and instructions for transferring the metadata, for example, metadata characterizing the properties of an image, such as the last names of those depicted in the photo or tags indicating the location where the photo was taken; parameters for parallel data transfer indicating the transfer of data with or without consolidation from several sources or the transfer to several target sources - data recipients. [0055] After receiving the mentioned request (101), the API service 20 analyzes the instructions for combining the data and identifies it, for example, as a request for a one-time data transfer between the registered IDs 30, ID 31 and ID 40, after which it checks the registration of all data sources participating in the data transfer. To check the registration, the API service 20 accesses the memory with which it is equipped, which contains information about the registered IDs. If the API service 20 has determined that at least one ID is not registered, then a notification about the presence of an unregistered ID is sent to the user device 10. [0056] If all IDs are registered, then the API service 20 generates a request containing the necessary sequence of data movement operations, in particular a command to move the first data set from ID 30 and the second data set from ID 31 to ID 40, as well as a parameter of the movement priority determining the order of execution of the movement operation, and sends the generated request to the Task Scheduler 50, which can be implemented on the basis of at least one computing device equipped with logical elements for performing the functions assigned to it. In addition, other information contained in the data movement request can be sent to the Scheduler 50. The Task Scheduler 50 processes the queue of transport tasks, polls the system for the presence of suitable events that will entail the initialization of the sought operations, maintains an event log.

[0057] Upon receipt of the mentioned request for movement, the Scheduler of the system 50, in accordance with the priority parameter, adds a new task for movement (transportation) of data to the queue and sends a confirmation of the placement in the queue to the address of the API service 20. In this case, the Scheduler 50 may also include in the confirmation information about the predicted start date of the task execution, the place of the task in the queue, the priority of the task (if the Scheduler 50 has several queues of different priorities), and other information.

[0058] In case the data movement request contains several target IDs 40, then the Task Scheduler 50 generates several data movement tasks with the same priority parameter depending on the number of IDs 40 and adds all the tasks to the queue, wherein each data transport process in ID 40 is defined as a separate independent process at the execution stage, and all processes during parallel data transport are created with the same priority.

[0059] Upon receiving confirmation from Scheduler 50 about placing the task in the queue, API service 20 sends confirmation about placing the task in the queue to the address of user device 10 for display to the user via UI 11.

[0060] The system scheduler 50 processes the task queue in accordance with predetermined prioritization algorithms and task-specific priority parameters, including but not limited to prioritization methods: without additional prioritization (FIFO method, first in, first out), with prioritization in the form of allocation of priority queues (each queue has its own priority, tasks from the queue with a higher priority are always executed first), prioritization of proportional resource allocation (in this case, a higher priority queue receives more resources to complete a task, for example, in the case of three queues, four tasks are executed from the highest priority queue, then two tasks from the medium priority queue are executed, then one task from the lowest priority queue, then the cycle repeats), and other methods and algorithms.

[0061] When a task needs to be executed based on a condition, Scheduler 50 suspends processing of the specific task until the condition is met, after which the task is executed with high or highest priority.

[0062] Additionally, the scheduler 50 may be configured to rearrange the moving task in the queue. For example, the scheduler 50 may be equipped with a memory that contains data describing the execution time of similar moving tasks, which may be entered by the developer of the scheduler or stored by the scheduler as historical data when executing similar data moving tasks, taking into account the data transfer route.

[0063] Accordingly, the Scheduler 50 extracts from the data movement request a time value before which the data movement should be completed, extracts from the memory data characterizing the execution time of similar tasks for movement along the specified route and, on their basis, determines the predicted execution time of the movement task, after which, taking into account the predicted execution time of other tasks contained in the queue and the values of their priorities, it rearranges the movement task in the queue for its execution at the specified time value. The mentioned task can be rearranged both forward in the queue for its execution at the specified time value, and backward so that other tasks are executed at the time value specified for them.

[0064] When a task is received from the execution queue, the Scheduler 50 of the system transfers the task for execution (102) to the Worker Node 60, which, when the task is received for execution, performs the following set of actions - stages. [0065] At step 103, the Worker Node 60 determines interfaces for working with the ID 30, ID 31 and ID 40, and then connects to said ID. The determination of said interfaces and the establishment of a connection can be carried out by known methods automatically or on the basis of previously made settings of the network and computing environment, taking into account the data transfer protocols implemented at various levels of the OSI network model (https://ru.wikipedia.orq/wiki/CeTeBan OSI model), including the Application layer (for example, when taking into account and applying conditions and triggers), the Presentation layer (for example, when applying data encryption when transmitting between data sources), the Session layer (for example, to establish communication sessions when transmitting according to a schedule). In this case, the IDs can be connected to the transmission network at the Transport, Network and Data Link layers.

[0066] After the Worker Node 60 has connected to the said IDs, the Worker Node 60 makes a preliminary assessment of the possibility of performing the data movement task based on the available resources. For example, the Worker Node 60 can determine the total amount of data stored in the IDs 30 and 31 intended for movement, and determine the availability of free space for storing data in the ID 40. If the Worker Node determines that the available resources are insufficient to perform the said task, then the Worker Node 60 sends a corresponding notification to the user device 20 via the API service 20. If the said resources are sufficient, then the Worker Node 60 proceeds to the next step (105).

[0067] In the next step (105), the Worker Node 60 determines the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in said IDs. For example, the Worker Node 60 can determine that the files in the target ID 40 are stored in jpeg format or occupy the same amount of data, and based on this information, as a method of interaction with the ID 40, determine that the data transfer should be performed only in a certain format, i.e. jpeg, and the data volume should not exceed the data volume of the files stored in the ID 40, unless the developer has set permissions to store data of any format and volume in the ID 40. The specified file format is given as an example, while the method described in this application can process datasets of structured and unstructured files containing audio and voice information in vaw, MP3 and other formats, video information in avi, MP4 and other formats, as well as file representations in plain text, csv, tabular representation formats, in encoded form, for example, using Base64 representation, binary representation, representation in JSON and xml formats and in other formats and representations.

[0068] Accordingly, if permissions for storing data of any format and volume in ID 40 are not set and at least one file from the files being moved in ID 30 or ID 31 is saved in a format other than jpeg (for example, in png format), or its volume is several times greater than the volume of files saved in ID 40, then in accordance with the previously determined method of interaction with ID 40, such a file will not be transferred, and a corresponding notification will be sent to the user of device 10.

[0069] After the method of interaction with the target ID 40 is determined, the Worker Node 60 proceeds to the step 106 of constructing an optimal route for moving data from the ID 30 and the ID 31 to the ID 40 in the data transmission network. The following may act as a data transmission network: a local area network; an Intranet network divided into segments and domains; a virtual computing environment in a cloud computing infrastructure; a virtual computing environment that transmits data via dedicated channels through the Internet or through other networks; or combinations of the mentioned networks.

[0070] The optimal route can be determined by widely known methods, wherein to determine the optimal route, the Worker Node 60 determines the route from the ID 30 and the ID 31 to the ID 40, as well as from the IDs in which copies of the data of the ID 30 or the ID 31 that should be moved are stored to the ID 40, after which the optimal route for moving the data is selected, for example, based on an analysis by the Worker Node 60 of the delay time in transmitting data along the route from the ID 30 and the ID 31 to the ID 40 and from the IDs in which copies of the data are stored to the ID 40. The ID information with copies of the data stored in the ID 30 and the ID 31 can be stored in advance in the memory of the Worker Node 60 or provided by the user in a request for moving the data.

[0071] After the optimal data transmission route is determined, the Worker Node 60 sends a corresponding signal to check the operability of the IDs involved in the data movement and confirm their operability and readiness to transmit and receive data, respectively, and then, depending on the data integrity verification technology, assigns the first and second sets of data intended for movements, data integrity parameters. The integrity parameter may be a checksum or a hash sum determined by the Worker node 60 using known methods for the first and second data sets, or the data size may be designated as the integrity parameter.

[0072] Then, the Working Node 60 proceeds to the data movement step 107 in accordance with the data movement conditions, in particular, the Working Node 60 reads data from the ID 30 and the ID 31 or from the IDs in which copies of the data are stored according to a previously determined route, and transmits (transports) the data to the address of the target ID 40, and also taking into account the error correction code, the data increment (addition) and the parameters of parallel data transmission. For example, in accordance with the data movement conditions, the first and second sets of data stored in the ID 40 can be supplemented with the specified metadata, or if a plurality of photographs are moved, then in accordance with the data movement conditions, the metadata can be stored both for each photograph and for a specified number of photographs.

[0073] During the transportation, the Worker Node 60 collects and analyzes information about the integrity of the transmitted data 108 by comparing the integrity parameters of the first and second sets of data and the integrity parameters of the same data transferred to the target ID 40, determined in the manner described above after their transfer. In an alternative embodiment of the presented solution, the data integrity check can be checked by encoding the first and second sets of data received from the ID, transferring them to the target ID 41 and then decoding. If the data received after decoding corresponds to the data from the first or second set of data, for example, their size corresponds, then the Worker Node 60 makes a decision on the successful completion of the data transfer.

[0074] Accordingly, the Worker Node 60 makes a similar decision on the successful completion of the data transfer if the said data integrity parameters match. The confirmation of the successful transportation of the corresponding set and/or fragment of data is sent by the Worker Node 60 to the Scheduler 50, and in the absence of confirmation of successful transportation, the Scheduler 50 sends a command to the Worker Node 60 for retransmission until success is confirmed. Also, a command for retransmission of the data of the set and/or fragment of data is sent by the Scheduler 50 in the event of problems and failures during the transfer process. [0075] Also, notifications and reports on the operations performed and their status are sent via the API service 20 to the user device 10, if necessary enriched with notifications with the parameters of the task being performed. [0076] Additionally, the Worker Node 60 is configured to synchronize the completion of the process of moving data from the IDs 30 and 31, taking into account the increment. The synchronization process consists, in particular, in the fact that by the time the transfer of both photographs and metadata is completed, all pairs are collected, and each photograph is associated with its own metadata, each set of metadata is associated with its own photograph. Accordingly, the Worker Node 60 checks whether the first and second sets of data and the increment have been transferred to the ID 40 before moving on to the next task or the next stage of the task.

[0077] Also additionally, before moving the first or second set of data, the Worker Node 60 checks whether the ID 40 contains a similar set of data. If the ID 40 does not contain a similar set of data, then the Worker Node 60 assigns the first version to them after moving the first or second set of data. If the ID 40 already contains a similar set of data, then the first or second set of data being moved is assigned the next version of the data depending on the version of the similar set of data.

[0078] After all operations of the assigned tasks are completed, the Scheduler 50, which monitors the movement of data by the Worker Node 60, records metrics 110 associated with the operation itself and the use of resources, in particular, the time of movement of the files of the first and second data sets, the amount of memory occupied by them, the number of files transferred, confirmation of the execution of the operation at a specified time, the value of the delay in the transfer of data, etc. Using the API Service 20, the user of the device 10, as well as other modules and systems, can be provided with access to relevant metrics for analysis and preparation of reports.

[0079] If this is provided for by the transportation plan, Task Scheduler 50, upon receiving confirmation, sends to the address of API service 20 a confirmation of the successful transportation of a set and/or fragment of data, on the basis of which API service 20 issues a command to the address of ID to delete the corresponding data in order to prevent the storage of redundant copies of data.

[0080] Scheduled data transport is an operation that is repeated without changing parameters, with the possibility of automating the process of repeated execution. Scheduled data transport is, as a rule, the main operation in the industrial operation of information systems and computing complexes for data processing, including systems operating in real time. If the user, when forming a data movement request, specified the data movement according to a schedule as an instruction for combining data, then the data processing system operates as follows.

[0081] At the first stage (100), the user, in the manner described above, by means of the device 10 forms a request for combining the first set of data from the first data source (ID) 30 and the second set of data from the second ID (31) in the target ID 40. In this case, the rules for the priority of compliance with the conditions specified in the request include, but are not limited to, checking the readiness of data for transportation, permission for transportation external to the system (business logic), completion of transportation tasks before a specific moment specified by the user, the beginning of the execution of a task according to a schedule, and other parameters. Specific priorities for the listed rules and the conditions for their application may change depending on the conditions of application of the system that is the subject of the invention.

[0082] Next, in the manner described previously 101, a request for data movement is sent to the API service 20. Upon receiving the request, the API service 20 analyzes the received request, identifies it as a request for scheduled data transport, checks the registration of all data sources participating in the data movement, generates a request containing the necessary sequence of operations for data movement, and sends the generated request to the Task Scheduler 50.

[0083] When a request for movement according to the schedule is first received, the Scheduler 50 of the system adds a new transport task to the beginning of the queue, receives confirmation of the readiness of the data for transport from the data source and sends a confirmation of the placement in the queue to the address of the API service 20. In this case, the Scheduler 50 may also include in the confirmation information about the expected start date of the task, the place of the task in the queue, the priority of the task (if the Scheduler 50 has several queues of different priorities), and other information.

[0084] The Scheduler 50 then calculates the time to complete the data transport operation specified by the schedule as the ratio of the data volume to the data transfer rate measured based on previous data transports on a specific route. If the predicted completion time of the transport is after a user-defined time, increases the priority of the task and moves the task forward in the execution queue.

[0085] The scheduler 50 collects, analyzes, and stores the necessary metrics describing the data transport process, including information about the data transfer rate along specific routes based on the results of measurements during the execution of previous data transport tasks.

[0086] When the queue for executing the task has arrived, the Scheduler 50 checks the confirmation of the readiness of the data for transportation from the data source, compares the current time and the time of the start of the task according to the schedule, as well as the estimated time of completion of the task and the completion time specified by the user. If the time of the scheduled start of the task has not arrived, and the transportation task will be completed at the required time, the Scheduler 50 puts the task in second place in the queue, repeats the calculation of the estimated time of completion of the transportation task, compares it with the completion time specified by the User 50, and in the case of a positive response about the completion of the tasks on time, accepts the next priority task from the queue for which the execution time is not specified.

[0087] If the predicted completion time of the transportation task, moved to the second place in the queue, comes after the transportation completion time specified by the User, the scheduled transportation task returns to the first place in the queue, and the Scheduler 50 begins to execute it.

[0088] If there are two or more data transport tasks in the queue, the tasks are first rearranged in the queue in the order of the start time of execution and taking into account the estimated completion time. If the start time of execution for each of these tasks has not yet occurred, and both tasks are estimated to be completed before the time specified by the User, the Scheduler 50 accepts the task with priority #3 or lower in analogy with the algorithm described earlier in the manner.

[0089] If the predicted completion times of two or more transportation tasks require a change in the order of execution in in accordance with the prioritization conditions described earlier, Scheduler 50 makes the appropriate change to the task queue

[0090] Simultaneously with the first receipt of a request for movement according to the schedule, the System Scheduler 50 adds a new transportation task to the transportation schedule in future periods, reserving a place in the queue. [0091] Since the volume of data to be transported may change when operations are repeatedly executed, the Scheduler 50 calculates the forecast time of the start and end of the operation in advance of the execution time according to the schedule, and also checks the availability of data sources and their readiness for transportation. The time period for the advance forecast is set during the installation of the system and is adjusted by a fixed or variable value in accordance with the successful or unsuccessful completion of the previous transportation task along the corresponding route. In order to execute transportation tasks on time, a threshold interval for a time reserve for compensation of the forecast error can also be set in the system.

[0092] If the data sources are not ready for transportation (unavailability, lack of authorization on the source, lack of access rights, etc.), the Scheduler 50 sends a corresponding notification to the API service 20, which in turn forwards a corresponding warning (Alert) about the risk of failure to perform transportation to the User UI 11 to take the necessary actions to restore the availability of the sources.

[0093] When sources are ready, information about the predicted start and end times of transportation and the readiness status of the sources is stored in the event logs of Scheduler 50.

[0094] After all the conditions for priorities and conditions for performing the transportation are met, the System Scheduler transfers the task for execution to the Work Node 60. The movement of data by the work node is performed in the manner described earlier in accordance with steps 102 - 109.

[0095] During the transportation process, the Scheduler 50 collects and analyzes information about the integrity of the transmitted data, receives confirmation of the successful transportation of the corresponding set and/or fragment of data, and in the absence of confirmation of successful transportation, monitors and performs retransmission until success is confirmed. [0096] API service 20 sends notifications and reports on the operations performed and their status to the User, if necessary enriching the notifications with the parameters of the task being performed. In the event of problems and failures during the transmission process, Scheduler 60 retransmits the corresponding set and/or fragment of data.

[0097] After the operation of the assigned tasks is completed, the System records metrics related to the operation itself and the resources used. The User and other modules and systems are provided access to relevant metrics for analysis and reporting.

[0098] Scheduler 60 polls the system for the presence of suitable events that will cause the initialization of the desired operations.

[0099] The user has the ability to stop a deterministic process (planned or already running) via the user interface. In this case, via the user UI 11, a corresponding request to stop operations is sent to the API service 20, including the identifiers of the operations, the conditions of execution (the period of time in which these operations must be performed) and other necessary parameters. Upon receiving a request to stop, the API service 20 requests confirmation of the stop operation, and after receiving the User's confirmation, sends a corresponding command to the Scheduler and the Worker Node.

[0100] When an operation is interrupted, necessary actions must be taken to preserve the data, including pausing or canceling the operations to remove duplicate data.

[0101] In the case of parallel data transfer from a data source to a target source, the system treats each transport process as a separate independent process during execution. All processes in parallel data transport must be created with the same priority.

[0102] Duplicate data is deleted upon completion of parallel transfer processes of a portion of the data or the entire set of transferred data.

[0103] When predicting the execution time of parallel data transports, the duration of execution of the process with the maximum execution time is taken into account for constructing forecasts.

[0104] If Scheduler 50 and Worker Node 60 detect a large number of transmission errors or do not receive integrity confirmation transmitted data, the Scheduler interrupts the transmission process and sends a notification to the User via API service 20 to take the necessary actions, while simultaneously generating an error report.

[0105] The presented solution provides guaranteed timely controlled secure movement of large and super-large volumes of structured and unstructured data and data streams in cloud environments and in local environments built using cloud technologies, including virtualization, scaling, access restriction, data collaboration, streaming and online data processing, continuous and batch data processing, data consolidation and transformation, data tagging, metadata collection, synchronous and asynchronous processing of data streams, backup, data delivery at the right time or when ready, removal of redundant copies and copies of data and other technologies and methods

[0106] In general (see Fig. 5), the computing device (200) contains one or more processors (201) connected by a common information exchange bus, including computing modules for accelerating computations - GPU (graphics processing unit), TPU (tensor processing unit), IPU (integrated processing unit), as well as modules based on FPGA and ASIC microprocessors, memory means such as RAM (202) and ROM (203), input/output interfaces (204), input/output devices (means) (205), and a device (means) for network interaction (206).

[0107] The processor (201) (or several processors, a multi-core processor, etc.) can be implemented on the basis of CISC, RISC, ARM, etc. architecture, and selected from a range of devices that are widely used at present, for example, from such manufacturers as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, Huawei Kunpeng™ and Nvidia™, IBM™, Baikal™, IVA™, etc. The processor or one of the processors used in the device (200) must also include a computing accelerator, implemented, for example, in the form of a graphics processor, such as an NVIDIA GPU with a software model compatible with CUDA, Graphcore, TPU (Tensor Processing Unit), IPU (Integrated Processing Unit), as well as modules based on FPGA and ASIC microprocessors, the type of which is also suitable for the full or partial implementation of the method, and can also be used for training and application of machine learning models in various information systems.

[0108] RAM (202) is a random access memory and is intended for storing machine-readable instructions executable by the processor (201) for performing the necessary operations for logical data processing. RAM (202), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, software modules, etc.). In this case, the available memory capacity of the graphics card, graphics processor or other computing accelerators can act as RAM (202). [0109] ROM (203) is one or more permanent data storage devices, for example, a hard disk (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0110] To organize the operation of the components of the device (200) and to organize the operation of external connected devices, various types of I/O interfaces (204) are used. The selection of the corresponding interfaces depends on the specific design of the computing device, which may be, without limitation: PCI, AGP, PS/2, IrDa, FireWire, COM, SATA, IDE, USB (2.0, 3.0, 3.1, micro, mini, type C), HDMI, DVI, VGA, RJ45, single-mode and multi-mode optical interfaces SC and ST, WiFi, NVLink, InfiniBand, STM, ATM, etc.

[0111] To ensure user interaction with the computing device (200), various I/O information means (205) are used, for example, a keyboard, a display (monitor), a touch display, a touchpad, a joystick, a mouse, a light pen, a stylus, a touch panel, a trackball, speakers, a microphone, augmented reality means, optical sensors, a tablet, light indicators, a projector, a camera, biometric identification means (a retina scanner, a fingerprint scanner, a voice recognition module), etc.

[0112] The network interaction means (206) provides data transmission via an internal or external computer network, such as an Intranet, the Internet, a LAN, etc. One or more means (206) may include, but are not limited to: an Ethernet card, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Wi-Fi module, single-mode and multi-mode optical interface modules, NVLink, InfiniBand, STM, ATM and DR- [0113] Additionally, satellite navigation means may also be used as part of the device (200), for example, GPS, GLONASS, BeiDou, Galileo. Precise time signals received from the mentioned satellite navigation means may be used to synchronize the operation of geographically separated devices. [0114] The specific selection of elements of the device (200) for implementing various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0115] Modifications and improvements of the above-described embodiments of the present technical solution will be clear to those skilled in the art. The foregoing description is provided only as an example and does not carry any limitations. Therefore, the scope of the present technical solution is limited only by the scope of the appended claims.

Claims

CLAUSE OF INVENTION. 1. A method for moving data in a cloud environment, comprising the steps of: - a request is received via API service 20 for moving the first data set from the first data source (ID) 30 and the second data set from the second ID 31 to the target ID 40, containing at least information about the conditions for moving the data; a request is generated via API service 20, containing a sequence of operations for moving the data, as well as a parameter of the priority of moving, determining the order of execution of the moving operation, and the generated request is sent to the address of Task Scheduler 50; - add a new task to move data to the queue using Task Scheduler 50 in accordance with the mentioned priority parameter; - when a task is received from the execution queue, the task is transferred to Work Node 60; - define, by means of Work Node 60, a list of interfaces for working with ID 30, ID 31 and ID 40; - perform, via Work Node 60, an assessment of the possibility of carrying out the data movement task based on the available resources; - determine by means of the Work Node 60 the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in the said IDs; - determine by means of Work Node 60 the optimal route for moving data from ID 30 and ID 31 to ID 40; - check the operability of the IDs used to move data using the Working Node 60; - by means of the Work Node 60, the first and second sets of data are read from the ID 30 and ID 31 and the said data are transferred to the address of the target ID 40 in accordance with the conditions of data movement. 2. The method according to item 1, characterized in that ID 30, ID 31 and ID 40 are located in the same cloud (i.e., logical pool). 3. The method according to claim 1, characterized in that at least one ID 30, ID 31 or ID 40 is located in a different cloud relative to the said IDs. 4. The method according to item 1, characterized in that the data transmission to the address of the target ID 40 is carried out taking into account the error correction code, data increment or parallel data transmission parameters. 5. The method according to paragraph 1, characterized in that it additionally contains a stage in which the registration of all data sources participating in the movement of data is checked. 6. The method according to claim 1, characterized in that the request for moving data contains information about at least one additional target ID to which the first and second sets of data should be moved, and the Task Scheduler 50 generates several tasks for moving data with the same priority parameter depending on the number of additional IDs. 7. The method according to item 1, characterized in that the Task Scheduler 50 additionally performs the following steps: - extract from the data transfer request the time value before which the data transfer must be completed; - extract data from memory that characterize the time it takes to complete similar tasks to move along a given route; - based on the extracted data, the predicted time for completing the movement task is determined; - performs a rearrangement of the data movement task in the queue for its execution at the specified time value, taking into account the predicted execution time of other tasks contained in the queue and the values of their priorities. 8. The method according to item 1, characterized in that in order to determine the optimal route, the following steps are performed by means of the Working Node 60: - determine the route from ID 30 and ID 31 to ID 40, as well as from the IDs in which copies of the data of ID 30 or ID 31 that should be moved are stored, to ID 40; - select the optimal route for moving data, taking into account the IDs in which copies of the data are stored. 9. The method according to item 1, characterized in that an additional step of checking the integrity of the transferred data is performed. 10. The method according to item 1, characterized in that the stage of synchronizing the completion of the process of moving data from IDs 30 and 31 is additionally performed by means of Work Node 60, including taking into account the increment. 11. The method according to claim 1, characterized in that additionally, the Work Node 60 performs version control of the moved data. 12. The method according to item 1, characterized in that metrics associated with the operation of moving data and the use of resources are additionally recorded. 13. The method according to item 1, characterized in that the execution of the data transportation operation is carried out in accordance with a specified schedule. 14. The method according to claim 1, characterized in that the first or second data sets contain structured and/or unstructured data and/or heterogeneous (multimodal) data. 15. A data movement system in a cloud environment, containing at least three IDs, an API service, a Task Scheduler, at least one The Working Unit, wherein the system is designed with the ability to implement the method according to any of paragraphs 1-14.