WO2019132689A1 - Automated software code generation for a corporate data warehouse - Google Patents

Abstract

This technical solution relates generally to the field of computer technology, and more particularly to systems and methods for the automated generation of software code for a corporate data warehouse. A method for the automated generation of software code for a corporate data warehouse includes: obtaining metadata describing a setting for data transformation mechanisms for loading data to a detailed layer level and calculating the data marts of a warehouse; generating at least one template for updating the data of the detailed layer and a data mart of the data warehouse; generating software code for loading data to the detailed layer of the data warehouse and calculating data marts on the basis of the metadata obtained and the data update template generated; installing the software code generated in the previous step in the data warehouse environment for performing loading; reusing the detailed layer and data mart update metadata. The technical result is an increase in the stability of detailed layer and data mart algorithms and a decrease in the number of incidents in the data warehouse.

Description

 AUTOMATIC GENERATION OF PROGRAM CODE FOR CORPORATE

 DATA STORAGE

TECHNICAL FIELD

 [1] This technical solution, in general, relates to the field of computer technology, and in particular to systems and methods for the automatic generation of software code for an enterprise data warehouse.

BACKGROUND

 [2] Analytical data warehouses are now becoming a necessary attribute for corporations working with large amounts of data, and are used to identify and create new financial instruments, maintain existing ones, monitor and identify accounting problems, create new and correct existing financial strategies.

 [3] The data source for such systems is almost always heterogeneous systems, serving both front-office systems, representing the corporation's face to customers, and back-office applications that perform financial accounting. In addition, very often the vaults also store middle-office data for analyzing their work. Thus, the source of data for the corporate repository are hundreds of systems that are constantly changing in accordance with the requirements of business development.

 [4] In such conditions, the creation of a clear one-type system for collecting data in a single model is a necessary requirement for obtaining relevant and reliable data coming from source systems.

 [5] From the state of the art, patent Ns US6604110B1 "Automated software of the metadata-based repository" patent, patent holder: International Business Machines Corp, publication date: 10/31/2000, is known. This technical solution discloses a method for providing executable program code for use in an enterprise data management model (EDM) application that transfers data from one or more data sources to an EDM application database for use in response to user commands.

[6] This invention only works online in response to user commands, which requires some effort and affects the speed of work. Moreover, this invention provides only executable code, which significantly reduces the possibilities of a technical solution and does not provide scripts or metadata, on the basis of which in runtime this metadata will determine the execution script.

SUMMARY OF INVENTION

 [7] This technical solution is aimed at eliminating the shortcomings inherent in existing solutions, known from the prior art.

 [8] A technical problem (or technical problem) in this technical solution is the implementation of automatic generation of program code for the corporate data warehouse.

 [9] The technical result, which appears when solving the above technical problem, is an increase in the stability of the algorithms of the detailed layer and data marts, as well as a decrease in the number of incidents in the data warehouse.

 [10] An additional technical result is an increase in the efficiency of the data warehouse and a reduction in the time taken to perform typical operations for analysts through the use of metadata and data update templates.

 [11] Additionally, the speed of data migration to another platform is enhanced by reusing metadata and data templates.

 [12] This technical result is achieved due to the implementation of the method of automatic generation of program code for the corporate data warehouse, in which metadata are obtained that describe the setting of the data transformation mechanisms for loading them to the detailed layer level and calculating the data mart storefronts; form at least one template update data detailed layer and data marts data storage; carry out the generation of software code for loading data into the detailed storage layer and calculating data marts based on the received metadata and the generated data update template; install the program code generated in the previous step to the data storage environment for downloading; reuse of metadata update detailed layer and data marts.

[13] In some embodiments, metadata are stored in relational database tables. [14] In some embodiments, metadata is a description of the structure of tables, the relationships between them, partitioning rules, a description of data marts.

 [15] In some embodiments, the implementation of the metadata is constant and calculated.

 [16] In some embodiments, metadata are placed in the form of a column-based structure of objects and their primary keys in a database.

 [17] In some embodiments, the data update pattern is described in the 82T file.

[18] In some embodiments, additionally, when generating program code, a work log is obtained that includes all input macro variables, as well as a trace file.

 [19] In some embodiments, the metadata for updating the detailed layer and data marts are reused by making delta changes to the S2T file.

 [20] In some embodiments, the reuse of the metadata of the update of the detailed layer and the data marts is performed by switching the basic update patterns.

BRIEF DESCRIPTION OF THE DRAWINGS

 [21] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

 [22] Figure 1 shows an exemplary implementation of a method for automatically generating program code for an enterprise data storage;

 [23] FIG. Figure 2 shows the architecture of the storage area of the corporate storage database;

 [24] FIG. 3 schematically shows the logical structure and ETL-processes of building a single semantic layer;

[25] FIG. 4 shows a scheme for calculating surrogate keys;

 [26] FIG. Figure 5 shows the complete path for automatic code generation.

DETAILED DESCRIPTION OF THE INVENTION

s [27] This technical solution can be implemented on a computer, in the form of an automated system (AS) or computer-readable media containing instructions for performing the above method.

 [28] The technical solution can be implemented as a distributed computer system.

 [29] In this solution, a system means a computer system, a computer (electronic computer), a CNC (numerical control), a PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined computational sequence operations (actions, instructions).

 [30] A command processing device is an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[31] The command processing device reads and executes machine instructions (programs) from one or more data storage devices. In the role of a storage device can act, but not limited to, hard drives (HDD), flash memory, ROM (read-only memory), solid-state drives (SSD), optical drives.

[32] Program - a sequence of instructions intended for execution by a computer control device or command processing device.

 [33] The following will describe the terms and concepts necessary to implement this technical solution.

[34] ETL (from the English. Extract, Transform, Load - “extract, transform, load”) is one of the main processes in managing data warehouses, which includes: extracting data from external sources; transforming and refining them to fit the needs of the business model; and loading them into the data warehouse.

[35] S2T is an Excel format file containing information about all data transformations of a subject area that is subject to ETL development.

[36] Surrogate keys are a set of metadata that defines the rules for creating primary keys of a data warehouse, as well as rules for converting from natural keys of source data to primary keys of a warehouse. [37] A natural key is a set of attributes of an entity described by a record that uniquely identifies it (for example, the passport number for a person).

 [38] The detailed layer is a set of tables of a specific physical area of the data warehouse, which is the final layer for ETL processes and which contain all the business data of the data domain storage.

 [39] Metadata - service information of the solution, describing the settings of the mechanisms and the results of the work of the data integration subsystems, the data warehouse and the analytical application (for example, information about downloaded files).

 [40] The ETL framework is server-based software that manages and loads data from external systems into the data warehouse.

 [41] The semantic layer is a set of physical and virtual data marts that provide the data of the FSLDM model in a transformed form in accordance with the purpose of a particular storefront.

 [42] Data Showcase (Data Mart; other translation options: specialized data storage, data kiosk, data market) —a data storage slice, which is an array of thematic, narrowly targeted information, targeted, for example, to users of one working group or department.

 [43] OLAP is a data processing technology consisting in the preparation of summary (aggregated) information based on large data arrays, structured according to a multidimensional principle. Implementations of OLAP technology are components of software solutions of the Business Intelligence class.

 [44] Patch (English patch / paetj / - patch) - information intended for automated introduction of certain changes in computer files and databases.

 [45] A method for automatically generating program code for an enterprise data warehouse shown in FIG. 1, is as follows.

 [46] Step 101: receive metadata describing the configuration of data transformation mechanisms for loading them to the detailed layer level and calculating data mart storefronts;

[47] In the corporate data warehouse 200 shown in FIG. 2, a clear system of stages of loading and converting data from external automated systems was recorded, accompanied by unified logging of all stages, dynamic and static parallelization and the possibility of recovery in case of failure from any download location.

 [48] Metadata is an important part of the data warehouse architecture 200. Metadata is data that describes the rules by which the store operates. For example, from the point of view of a warehouse database, metadata is a description of table structures, relationships between them, partitioning rules, description of data marts, etc. From the point of view of ETL, metadata are descriptions of the rules for extracting and transforming data, the frequency of execution of ETL processes, etc.

 [49] In some embodiments, the implementation of the metadata of corporate vault 200 includes:

 • information about the data stored in the repository, their business description and storage structure;

 • a description of the structure of data sources, both external and internal, of their availability;

 • information on the structure of the ETL processes, their frequency of execution, the applicable rules for data cleaning and conversion;

• a description of the presentation of data that helps the user to work with the B1 application;

 • information about security settings, authentication rules and assigned access rights;

 • statistics on the utilization of resources, data accesses, etc., which helps the administrator to optimize the operation of the warehouse database.

 [50] In some embodiments, metadata management is performed by separate tools for each of the storage components. For example, for an Oracle database whose metadata is stored in system tables and configuration files, this tool is the Oracle Enterprise Manager.

 [51] In some embodiments, the implementation of the metadata can be stored in special PostgreSQL tables.

[52] Attributes can be used as metadata, which are parameters that allow managing the ETL process. In some embodiments, the implementation of the metadata, and accordingly the attributes can be constant and calculated. The values of constant attributes do not depend on the results. previous data downloads to the repository. Calculated attributes are variable values depending on the results of previous ETL downloads. Attribute values can be entered into ETL structures through a web interface. Such attributes can be the loading mode (for example, archive, incremental), the version of the loading specification, the sequence number of the incremental loading, the lower limit of the upload for each loaded entity, etc.

 [53] As metadata, statistics can be used that are values stored during an ETL process that can be used to calculate calculated attributes in subsequent processes. Statistics values can be stored in tables by means of relational database stored procedures. As a relational DBMS can be used such as MySQL, PostgreSQL or Oracle, without being limited to.

 [54] In some embodiments of the invention, the metadata is placed in the form of column structure of objects and their primary keys in the database. This data can be presented either in the form of a tabular presentation and / or in the form of Excel spreadsheets, and / or in any relational database.

 [55] Metadata is obtained from primary external data systems 230 and loaded into temporary storage area 210 (FIG. 2).

 [56] The temporary storage area 210 is an intermediate layer between external data sources and a persistent storage area in which a detailed layer and data marts are located. In this area, the data extracted from external data sources (DBMS, csv, dbf, xml files, web services, etc.) are saved, data is cleared, transformed, enriched, and prepared for loading into the permanent storage area 220. Often, the next cycle of processing and loading data into the repository cannot be started until all the necessary data from various external data sources are retrieved, and for a number of reasons (geographic distribution, different system operation cycles, etc.), the data in the sources can be available at different points in time. The temporary storage area 210 serves to collect all the necessary data before starting the transformation.

[57] Step 102: forming at least one pattern of updating the data of the detailed layer and data marts of the data storage; [58] The detailed layer is the main corporate data warehouse. Transformed and cleaned detailed data from external systems 230 and data sources, and basic classifiers are stored in this area. This area contains the following types of entities: reference books and classifiers; entities containing actual values; entities describing relationships.

 [59] Reference books and classifiers can determine the following information:

 • participants of the main business processes - customers, suppliers, branches, services, products, etc .;

 • basic reference books - date and time, currency, countries, etc .;

• other directories - reflecting the needs of the business in the necessary data analytics, defining, in terms of which directories, the actual data should be analyzed.

 [60] Entities containing actual values are transactional data from external systems 230 and data sources. For example, information about customer payments made, invoices, transactions, etc.

 [61] Entities containing relationships define relationships between all other entities. For example, the connection may be the following: client-service-bank.

 [62] Data marts are objects of storing analytical information aimed at supporting specific business functions, specific company divisions. At the database level, storefronts are usually implemented in a star or snowflake pattern and contain data from the detailed layer area. Also, data marts can be implemented as a multidimensional OLAP cube. Data marts are the basis for multidimensional analysis (OLAP) data. In the corporate data warehouse 200, the detailed layer and data marts are located in the permanent data storage area 220 (FIG. 2).

[63] Implementation of metadata transformations and framing them into stored procedures is implemented using data update templates that work like java run-time substitutions, similarly to the Apache Jakarta Project, where this technique was used as the basis of the Model-View Controller template for web applications. Conversion rules can be stored in a column view. This data can be presented as a table. representations and / or in the form of Excel spreadsheets, and / or in any relational database.

 [64] In some embodiments, the data update pattern is a SQL procedure template.

 [65] This technical solution uses the strategy of updating the detailed layer data and data marts, which is a way to write the initial data set in the detailed layer table format to the detailed layer table, which solves the problem of building a history.

слой Т) полностью стандартен, и может быть описан в 52Т-файле посредством указания названия стратегии обновления и ее параметров в виде шаблона обновления данных. [66] In contrast to the ways of formatting the source data in the format of the detailed layer (layer B -> layer N), the data update pattern (layer N

layer T) is completely standard, and can be described in a 52T file by specifying the name of the update strategy and its parameters as a data update template.

 [67] In this technical solution, for example, the following data update templates “data snapshot - data snapshot”, “data snapshot - data history”, “data history - data history” can be used. These strategies are implemented depending on the type of source data and requirements for the final data.

 [68] The data refresh strategy “data snapshot - data snapshot” works as follows. Non-historical data from the data source is transferred to non-historical tables in the detailed data storage layer. This data update template can be used for surrogate key containers (contracts, clients, etc.), as well as for transactional data (financial transactions, transactions, etc.). If the source fields used to calculate the primary key of the detailed layer can be overwritten on the source, then they support emulation of the replication mode. Replication is a process that means copying data from one source to another (or to many others) and vice versa. Moreover, this data update template can work with data accumulation and without data accumulation.

[69] The data refresh strategy “data snapshot - data history” works as follows. Data for which no change history is maintained at the source is stored in a detailed layer of a permanent data warehouse with history creation. At the same time, the history for later periods is removed from the detailed layer than the one that came in the current update package. If the fields source used to calculate the primary key of the detailed layer can be overwritten on the source, then support replication mode emulation.

 [70] The strategy of updating data “data history - data history” works as follows. Performs full replication of the data history, which is maintained at the source with reference to the date / time of an attribute value change (or linked to date periods - dates / start and end dates), to the detailed data storage layer. At the same time, to exclude problems with updating the "date of relevance" field at the source, support replication mode emulation. When loading source datasets with a history for the date, gaps in the history of the detailed layer are not allowed. No intersection in the history of the detailed layer is allowed; an earlier period of two intersecting ones is subject to truncation. A history for a certain period can also be loaded with a “snapshot-snapshot” template, if there is no need to eliminate the intersection of historical periods. For this, it is necessary to provide emulation of the replication mode; the key should include both the fields on the basis of which the primary key is calculated in the detailed layer and the fields containing the beginning and end of the relevance period.

 [71] The formation of a surrogate key can be carried out by keeping and mapping the initial sets of natural keys to a sequence of integer values or, as shown in FIG. 4, can be determined based on the value of the unique key of an entity using the following formula:

 [72] For entities with a unique key consisting of one attribute.

 SURROGATE KEY = SHA1 (NNN || SOURCE KEY1)

[73] For entities with a composite unique key:

 SURROGATE KEY = SHA1 (NNN || SHA1 (SOURCE KEY1) [[||

 SHA1 (SOURCE_KEY2)] || ... || SHA1 (SOURCE_KEYN)])

 [74] where

 [75] NNN - 3-character code of the source system;

 [76] SHA1 - the function of obtaining the hash value according to the SHA1 algorithm;

[77] SOURCE KEY1 .. SOURCE KEYN - columns included in the unique key of the table from the source system, if the unique key is represented by more than one field, then the concatenated value of all the fields in the key with the separator “|” is used; [78] And - operation of concatenation.

 [79] The surrogate keys obtained according to this scheme can be in the form of a 160-bit hexadecimal code. In the future, such keys are conveniently used to bind non-unified entities.

 [80] In some embodiments, an Apache view is used.

Hive for calculating surrogate key values. Hive is an engine that turns SQL queries into chains of map-reduce tasks. The engine includes components such as Parser (parses incoming SQL queries), Optimizer (optimizes the query for greater efficiency), Planner (schedules tasks for execution), Executor (3a starts tasks on the MapReduce framework.

 [81] Hive also requires a metadata repository. SQL assumes work with such objects as a database, a table, columns, lines, cells, etc. Since the data itself that Hive uses is simply stored as files in hdfs — it is necessary to store somewhere the correspondence between the Hive objects and the actual files.

 [82] Step 103: generate program code for loading data into the detailed storage layer and calculating data marts based on the received metadata and the generated data update template;

 [83] FIG. 5 shows the process of automatic generation of software code, which consists of the following steps.

 [84] 1) Retrieving the SQL code based on the data update pattern, which is at least one stored procedure, from an S2T file;

 [85] An S2T file might look like this:

[86] 2) encapsulation of this program code into an executable module (procedure), with standard input and output parameters, a logging and error handling system, and ways to restore the boot process in the event of a technological failure;

 [87] 3) inserting a call to this procedure into a metadata script that is read by the framework in run-time mode and sequentially and / or in parallel executes instructions from the template.

 [88] Within the procedure templates (FIG. 5), various “case” or “if else” branches of the implementation are implemented, depending on the launch flags. In turn, the launch flags are set by the general project flags, the flags of the conversion stage, the specific conversion flags, etc.

 [89] The processing of each template results in directly generated working program code. Additionally, as a result of the template operation, a work log is received, which includes all input macro variables, as well as a trace file (trace file), which can contain all values of variables, used data structures and arrays (collections).

 [90] Each transformation can contain either the name or symbol of the templates used.

 [91] The input data for the procedure template can be: scalar variables, vector elements (arrays, lists, hash tables), as well as multidimensional elements (vector vectors). The elements can be either the simplest elements or complex objects (objects based on the developed Java classes).

 [92] Step 104: automatically install the program code generated in the previous step onto the data warehouse environment to perform the downloads;

[93] In this step, a patch is created that contains the generated program code in the format and structure of the standards-compliant release-configuration management, which is installed on the development environment in run-time mode. The patch can also include a patch installation script with instructions for the admin console. The patch, ready to be installed on the corporate data warehouse development environment, is compatible with the system of automatic distribution of patches to the stands. [94] The objects, databases that are included in the patch, are installed on the development environment in the same manner. If errors occur, they are in the event log and visible to the user. The set of event logs contains a detailed analysis of possible errors and warnings. Objects of the same type are grouped in directories by data types and are located inside the suffixes of the database names and their subsequent location.

 [95] Table 1 below shows a typical patch structure. Particular attention should be paid to the SetupManual_000000.txt, in which a sequence of steps is automatically created for installing the patch (distribution) on all environments.

 Table 1

[96] In the “Compile” = ”true” mode, all patch objects are created on the development environment. That is, in the mode of code generation, all objects that will be included in the patch are created on the development environment.

[97] Step 105: reuse the detailed layer update metadata and data marts.

 [98] When making changes in the formulation of the task, they reuse the metadata for updating the detailed layer and data marts by introducing a delta of changes in the above S2T form

[99] When changing the type of relational database, reuse of update metadata of the detailed layer and data marts is carried out to form the filling scenario by switching the basic update templates.

 [100] Then re-use the generated program code and metadata. After the implementation of the first version of the patch (distribution) for a task, changes to this task almost always follow, caused by both new business requirements and changes caused by corrections of the first version. The result of these changes may be the creation of new objects, deletion of old objects, change of procedures, change of scripts.

 [101] To implement these changes can use two approaches.

 [102] The first approach is to make changes to the existing S2T file (structure) and re-create the entire patch, that is, implement a new patch (cumulative patch). After this, changes can be highlighted by the version control system (git) used to maintain the code.

[103] The second option is to perform automatic marking in the S2T tabs of the modified transformations and then only they will be included in the patch. The patch interface should be marked as incremental in the interface. [104] There are changes caused by the needs of the work process. These changes are made to the existing metadata described in S2T and during generation, the previously used metadata is reused.

 [105] Aspects of the present invention may also be implemented using a data processing device that is a computing machine or system (or tools such as a central / graphics processor or microprocessor) that reads and executes a program stored in memory to perform the functions described above. the embodiment (s) and the method shown in FIG. 1, the steps of which are performed by a computing machine or device by, for example, reading and executing a program recorded on a memory device in order to perform the functions of the embodiment described above (s). To this end, the program is recorded on a computer, for example, via a network or from a medium for recording various types, serving as a storage device (for example, a computer-readable medium).

[106] The data processing device may have additional features or functionality. For example, a data processing device may also include additional data storage devices (removable and non-removable), such as, for example, magnetic disks, optical disks, or tape. Storage devices can include volatile and nonvolatile, removable and non-removable media implemented in any method or using any technology for storing information, such as computer-readable instructions, data structures, program modules or other data. Storage device, removable storage and non-removable storage are examples of computer storage media. Computer storage media includes, but is not in a limiting sense, random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or memory made using a different technology, ROM on a CD disk (CD-ROM), digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tapes, storages on magnetic disks or other magnetic storage devices, or any other medium that can be used on to store the desired information and which the data processing device can access. The processing unit may also include input device (s) such as keyboard, mouse, pen, voice input device, touch input device, and so on. An output device (s) such as a display, speakers, a printer, and the like may also be included in the system.

 [107] The data processing device contains communication connections that allow the device to communicate with other computing devices, such as over a network. Networks include local area networks and global networks along with other large scalable networks, including but not limited to corporate networks and extranets. A communication connection is an example of a communication environment. Typically, the communication medium may be implemented using computer-readable instructions, data structures, program modules or other data in a modulated information signal, such as a carrier wave, or in another transport mechanism, and includes any information delivery medium. The term "modulated information signal" means a signal that has one or more of its characteristics changed or set to encode information in this signal. For example, but without limitation, communication media includes wired environments such as a wired network or direct-wired connection, and wireless environments such as acoustic, radio frequency, infrared, and other wireless environments. The term “computer readable medium”, as used herein, includes both data carriers and communication media. The sequences of the processes described in this document may be performed using hardware, software, or a combination of both. When processes are performed using software tools, the program in which the sequence of processes is recorded can be installed and can be executed in the memory of a computer embedded in the specialized hardware, or the program can be installed and run on a general purpose computer that can perform various processes. .

[108] For example, a program may be pre-recorded onto a recording medium, such as a hard disk, or a ROM (Read Only Memory). Alternatively, the program may be temporarily or permanently stored (recorded) on a removable recording medium, such as a floppy disk, CD-ROM (CD for playback only), MO (magneto-optical) disk, DVD (digital versatile disk), magnetic disk or semiconductor memory. Removable recording media can be distributed in the form of so-called, sold through a retail network software.

[109] The program can be installed from a removable recording medium, described above, on a computer, or it can be transferred by cable from a download site to a computer, or it can be transferred to a computer via network data transmission channels, such as a LAN (local area network) or The Internet. A computer can receive a program transmitted in this way and can install it on a recording medium, such as an internal hard disk.

 [110] The processes described in this document can be performed sequentially in time, as described, or can be executed in parallel or separately, depending on the processing characteristics of the device performing the processes, or according to need. The system described in this document is a logical set of multiple devices and is not limited to the structure in which these devices are installed in a single package.

Claims

FORMULA  1. A method for automatically generating software code for an enterprise data warehouse, comprising the following steps:  • receive metadata describing the configuration of data transformation mechanisms for loading them to the detailed layer level and calculating data mart storefronts;  • form at least one pattern of updating the data of the detailed layer and data marts of the data warehouse;  • generate software code for loading data into the detailed storage layer and calculating data marts based on the received metadata and the generated data update template;  • install the program code generated in the previous step to the data storage environment for downloading; • reuse the metadata of updating the detailed layer and data marts.  2. The method according to claim 1, characterized in that the metadata is stored in the tables of the relational database.  3. The method according to claim 1, characterized in that the metadata is a description of the structures of the tables, the relationships between them, partitioning rules, the description of data marts.  4. The method according to claim 1, characterized in that the metadata is constant and calculated.  5. The method according to claim 1, characterized in that the metadata are placed in the form of a column structure of objects and their primary keys in the database.  6. The method according to claim 1, characterized in that the data update pattern is described in the 82T file. 7. The method according to claim 1, characterized in that, additionally, when generating the program code, a work log is obtained, which includes all input macro variables, as well as a trace file. 8. The method according to claim 1, characterized in that they re-use the metadata for updating the detailed layer and data marts by making delta changes to the S2T file.  9. The method according to claim 1, characterized in that the reuse of update metadata of the detailed layer and data marts is performed by switching the basic update templates.  10. The system of automatic generation of software code for the corporate data warehouse, containing:  • at least one data processing device; • at least one data storage device; • at least one program where one or more programs are stored on one or more data storage devices and executed on one or more data processing devices, wherein the one or more programs include the following steps:  i. receive metadata describing the configuration of data transformation mechanisms for loading them to the detailed layer level and calculating data mart storefronts;  and. form at least one template update data detailed layer and data marts data storage;  Sh. Carry out the generation of program code for loading data into the detailed storage layer and calculating data marts based on the received metadata and the generated data update template;  iv. install the program code generated in the previous step to the data storage environment for downloading;  v. reuse of metadata update detailed layer and data marts.