KR20190087807A - Datalake framework - Google Patents

Abstract

The present invention relates to a data rake framework capable of constructing an enterprise data rake so that an enterprise or an organization can provide large-capacity data generated through a business system to a user or system that consumes data by capturing, processing, and analyzing .
The present invention relates to a data processing apparatus and method for a data processing system, the data processing system including a data obtaining unit connected to a work magnetic machine for generating business data within the enterprise to obtain the business data, A data transfer unit 30 for supporting communication to transfer the acquired business data and a transfer rate control unit 40 for controlling the transfer rate of business data transferred through the data transfer unit 30 A lambda architecture unit 50 for modeling business data transferred through the data transfer unit 30, converting the data into transformation data, performing batch processing or real time processing, and generating and managing metadata for the transformation data, A data security unit 60 for encrypting the converted data to protect the converted data from being leaked to the outside, A portion (70) and, the user device 200 wants to use the stored conversion data is configured to include the data provider 80 for providing extracts the transform data.

Description

Data Lake Framework {DATALAKE FRAMEWORK}

The present invention relates to a data rake framework. More particularly, the present invention relates to a data rake framework for capturing, processing, and analyzing large-volume data generated by a business enterprise or an organization through a business system, To the data rake framework.

Due to recent advances in IT technology, the use of the Internet and so on in the enterprise increases and consumes a large amount of data.

Therefore, companies are aware of the necessity of enterprise data construction and analysis system to store and manage large amount of data, and are building data warehouse or data silos.

The data warehouse is a practical application methodology that efficiently integrates and manages each database management system that is distributed and operated within a large organization and provides the basis for an efficient decision-making system. • Transformation and alignment tools, database marketing system, meta data, end user access and utilization tools.

Such a data warehouse may include a data warehouse generating data warehouse from a database storing business information on a certain business field, as disclosed in Japanese Patent Laid-open No. 10-2018-0000413 (published on Jan. 03, 2013) A method for creating a warehouse, comprising: acquiring a model database according to the business field; generating a multidimensional modeling structure by performing Extraction, Transformation & Transfer (ETL) on the model database; To a model database to generate a reference database, and generating the data warehouse from the reference database based on the multidimensional modeling structure.

Data warehouses, however, are at risk of failures due to the sheer size and complexity of data volumes, and have the potential to invest significant time and money.

Therefore, there are a growing number of companies that use data rake to compensate for the disadvantages of data warehouses.

Data Rake defines a common database structure first and then stores all kinds of data instead of populating it with data that fits that structure, making it available in the required format when needed.

Data Rake enables all types of data to be collected and stored at any cost, at low cost. It can be used to prevent data security and unauthorized access, catalog, search and find relevant data from the central repository, perform new types of data analysis There are advantages.

However, data rake has a problem of speed reduction because it processes a large amount of data.

Accordingly, there is a need for a framework that can prevent the speed of data from being degraded while realizing the advantages of data rake for storing and managing vast amounts of data in the enterprise.

Disclosure of Invention Technical Problem [8] The present invention has been proposed in order to solve the above-described problems, and it is an object of the present invention to provide a data rake The purpose of the framework is to provide.

A Data Rake framework that can capture and manage metadata in terms of data traceability, data lineage, and security based on data sensitivity across data flows so that it can be delivered to users or systems that consume data while data is flowing And the like.

It is another object of the present invention to provide a data rake framework capable of selecting whether or not to encrypt data when storing data.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other matters not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a data rake framework including a data acquiring unit (20) for acquiring the business data, the data acquiring unit (20) being connected to a business intelligent machine (100) A data transfer unit 30 for communicating with the acquiring unit 20 to transfer the acquired business data to the work magnetic susceptor 100, A transmission speed control unit 40 for controlling the transmission speed of the business data; a communication unit 40 for modeling the business data transmitted through the data transmission unit 30, converting the data into converted data, A lambda architecture unit 50 for generating and managing meta data, and a data encryption unit 50 for encrypting the encrypted data to protect the converted data from being leaked to the outside A data storage unit (70) for extracting and providing the transformed data to a user device (200) that desires to use the stored transformed data, 80).

The lambda architecture unit 50 includes a phase data analysis unit 51 for performing data cleaning, processing, modeling, and conversion of the business data into transformed data, A real-time processing unit 53 for real-time processing the converted data converted by the phase data analyzing unit 51, and a batch processing unit 53 for improving the quality of the converted data, A data processing support unit 54 for supporting machine learning and data science processing based on the business data to a real-time processing unit 53 and a metadata unit 52 for generating and managing metadata for the converted data, 55).

In the present invention, the data security unit 60 includes an encryption selecting unit 61 for selecting whether to protect the converted data, a password setting unit 62 for setting a password to protect the converted data, And a cryptographic management unit (63) for managing a master encryption key for decrypting the encrypted conversion data.

The user device 200 includes a user communication unit 210 for communicating with the data security unit 60 and the data providing unit 80 and a user information unit for inputting user information and generating user information And a data request unit (230) for generating a request signal including the generated user information and a user IP address for identifying the user equipment (200) and requesting the converted data, wherein the data request unit 60 includes a user authentication unit 64 for determining whether the user information included in the request signal received from the user device 200 is previously registered user information and authenticating the user information, (65) for granting a right to use conversion data through user information, an IP address verification unit (65) for checking whether a user IP address included in the request signal is included in a preset IP address range, And can be configured to include an indentation (66).

As described above, according to the present invention, it is possible not only to improve the speed of the GPU and the SSD by capturing, processing, analyzing and storing the large-capacity data generated by the business system in accordance with the data flow standard, Can be prevented.

In addition, metadata can be captured and managed from the perspective of data traceability, data lineage, and security based on data sensitivity across data flows during data entry, enabling users or systems consuming data to efficiently extract desired data. There is, of course, the effect of improving security.

In addition, when data is stored, it is possible to select whether or not the data is encrypted, thereby preventing the data from being leaked to the outside.

1 is a block diagram of a data rake framework according to an embodiment of the present invention,
FIG. 2 is a diagram illustrating a physical resource of a data rake framework according to an exemplary embodiment of the present invention,
FIG. 3 is a flow chart illustrating the cloud busting and cloud spanning of the data rake framework according to an embodiment of the present invention,
4 is a block diagram illustrating a lambda architecture part of a data rake framework according to an embodiment of the present invention,
FIG. 5 is a block diagram illustrating a data security unit of a data rake framework according to an exemplary embodiment of the present invention,
6 is a block diagram illustrating a user equipment of a data rake framework according to an embodiment of the present invention,
7 is a flowchart for storing data in a data rake constructed by a data rake framework according to an embodiment of the present invention;
8 is a flowchart of providing data in a data rake constructed with a data rake framework according to an embodiment of the present invention;

Hereinafter, a data rake framework according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a data rake framework according to an embodiment of the present invention. FIG. 2 is a physical resource of a data rake framework according to an embodiment of the present invention. Cloudbusting of the framework and cloud spanning.

5 is a data security unit of a data rake framework according to an embodiment of the present invention. FIG. 6 is a block diagram of a data rake framework according to an embodiment of the present invention. And is a user device of the data rake framework according to the embodiment.

FIG. 7 is a flowchart for storing data in a data rake constructed by the data rake framework according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method for storing data in a data rake framework constructed by a data rake framework according to an embodiment of the present invention. Which is a flowchart for providing data in FIG.

In the drawings, the same reference numerals are given to the same elements even when they are shown in different drawings. In the drawings, the same reference numerals as used in the accompanying drawings are used to designate the same or similar elements. And detailed description of the configuration will be omitted. Also, directional terms such as "top", "bottom", "front", "back", "front", "forward", "rear", etc. are used in connection with the orientation of the disclosed drawing (s). Since the elements of the embodiments of the present invention can be positioned in various orientations, the directional terminology is used for illustrative purposes, not limitation.

As shown in FIG. 1, the data rake framework according to an embodiment of the present invention includes a plurality of task mirrors 100 for generating task data in an enterprise, A data acquiring unit 20 for acquiring the business data acquired by the data acquiring unit 20, a data transfer unit 30 for supporting communication to transfer the business data acquired by the data acquiring unit 20, A lambda architecture unit 50 for modeling business data transferred through the data transfer unit 30, converting the data into converted data, and batch processing or real-time processing the data, A data security unit 60 for encrypting and protecting the converted data, a distributed storage unit 70 for storing the encrypted converted data, And a data providing unit 80 for providing the conversion data to the user device 200 that desires to transmit the converted data.

By constructing the data rake framework as described above, it is possible to construct an enterprise data rake so that a corporation or an organization can capture, process, and analyze large-scale data generated through a business system and provide it to a user or a system consuming data do.

The business entity 100 can generate business data including business information by using a business system in an enterprise or an organization.

And is transmitted to the data rake to store and manage a large volume of business data generated from a plurality of business machines 100 that generate the business data.

As shown in FIGS. 2 and 3, the data rake includes physical resources 10 such as a computing resource 11, a storage resource 12, a networking resource 13, and the like for supporting a logical function.

The computing resource 11 may be divided into a CPU, a GPU, and an external multi-cloud. At this time, the CPU is composed of several cores, and the external multi-cloud is provided by cloud bursting.

The storage resource 12 may include a RAM / ROM, a hard disk drive (HDD), a solid state hybrid drive (SSHD), a solid state drive (SSD), a cloud storage, and a software-defined storage (SDS). Here, the cloud storage and SDS are provided by cloud spanning.

The networking resource 13 may logically be configured with a switch, a router, a bridge, etc., and may be configured with software-defined networking (SDN) and virtual networking technology.

As described above, physical resources can be configured to support logical functions.

The data acquiring unit 20 is connected to the task console 100 and receives and acquires the task data. That is, the data acquisition unit 20 is connected to the task server 100 via the Internet, cloud computing, etc., and receives the task data from the task server 100 to acquire the task data.

At this time, the data obtaining unit 20 may convert the business data into a message that can be additionally processed.

In other words, in order to perform additional processing after acquiring the business data in the data acquiring unit 20, the acquired business data is parsed, the parsed business data is converted, and a message is generated and temporarily stored in the buffer memory .

The data obtaining unit 20 should be flexible enough to accommodate various schema specifications in order to convert the obtained business data into messages. That is, since data can be classified into structured data, quasi-structured data, and unstructured data, and all types of data can not be defined by a schema, the data acquiring unit 20 acquires It should be flexible enough to accommodate various schema specifications.

In addition, the data acquiring unit 20 must have a quick connection mechanism to smoothly push the converted business data message to the data rake for further processing.

And converts the business data acquired through the data acquiring unit 20 into a message and transmits the message to the data transfer unit 30. [

The data transfer unit 30 supports communication for transferring the obtained business data to the lambda architecture unit 50 while separating the business logic unit 100 connected to the data acquisition unit 20.

In other words, the data transfer unit 30 separates the connection between the data acquisition unit 20 and the work magnetic unit 100 to prevent unnecessary data from being input, and the data acquisition unit 20 The transmission of the business data converted into the received message is guaranteed.

The data transfer unit 30 must have persistence of the message in order to guarantee transfer of the business data converted into the message, and the persistence of the message can be supported by the storage medium.

In addition, the data delivery unit 30 supports communication of a queue for 1: 1 communication and a topic messaging structure for 1: n posting and subscription.

The data transfer unit 30 can prevent the business data acquired by the data acquisition unit 20 from being modified while being transferred.

The transfer rate control unit 40 controls the transfer rate of the task data transferred to the lambda architecture unit 50 through the data transfer unit 30.

The transmission speed control unit 40 must ensure high scalability that can be expanded under various load conditions, and it can process various data at a high speed by securing the high expandability.

The transfer rate control unit 40 may provide fail-safe or fail-over fault tolerance, and may support a plurality of processes by supporting multi-thread and multi-event execution.

The fail-safety is intended to prevent a catastrophic result in the event of a failure or malfunction of the data rake, and the fail-over of the system can be avoided if the operation of the data rake is stopped, To allow the device to automatically work alternately.

Also, the delivery rate control unit 40 may convert the structure of the business data into a target data format, and may provide a seamless form of the transferred data for further processing of the business data. That is, the transfer rate control unit 40 can convert the task data into a target data format required when processing data in the lambda struc- ture unit 50, and by providing a seamless form of the task data, It is possible to prevent inappropriate information from being input to the data.

As shown in FIG. 4, the lambda architecture unit 50 receives the business data transmitted through the data transfer unit 30, converts the business data into transformation data, In real-time processing, which is a method of processing data that is constantly flowing without being limited in range.

Here, the lambda architecture unit 50 can select and process one of the batch processing and real-time processing, but it is necessary to solve the problem of merging batch data and real-time data in addition to batch processing or real- .

The lambda architecture unit 50 includes a phase data analysis unit 51 for modeling the business data, a batch processing unit 52 for collectively processing the modeled business data, A data processing support unit 54 for supporting machine learning and data science processing to the batch processing unit 52 and the real time processing unit 53 for processing the modeled business data, And a metadata unit 55 for generating metadata in the business data.

The phase data analyzer 51 converts the business data received from the data transmitter 30 into the transform data, which is modeled data, through phase data analysis.

That is, the phase data analyzer 51 models the business data by applying data modeling algorithm to the business data, and converts the business data into conversion data.

The batch processing unit 52 is capable of ensuring optimal activity of the system resources and is capable of performing batch processing on the business data received through the data transfer unit 30, It can be applied to long-running tasks to output.

In addition, the batch processing unit 52 can provide a mechanism for reproducing or returning to restore batch processing, and supports machine learning and data science processing based on the job data to generate high-quality converted data Should be.

In addition, the batch processor 52 may provide a view of deduplication, error data detection, and data lineage to improve the data quality of the transformed data.

The real-time processing unit 53 is separate from the batch processing unit 52 and is for real-time processing of the business data.

The real-time processing unit 53 can generate a data model related to real-time processing, all long-term execution processes must be delegated to a batch processing mode, and fast access and storage support are required so that the business data is not processed and accumulated do.

At this time, the real-time processing unit 53 must generate an output model in a manner that can be merged with the batch data set processed in the batch processing unit 52. [ In other words, the real-time processing unit 53 generates an output model so as to be merged with the batch data set processed in the batch processing unit 52, thereby providing enhanced data when providing data to the user equipment 200 in the future .

By configuring the real-time processing unit 53, it is possible to perform a quick operation on the data stream even though a large amount of business data is transferred from the work client machine 100.

The data processing support unit 54 supports the machine learning and data science processing based on the business data by the batch processing unit 52 and the real time processing unit 53.

The quality of the converted data processed in the batch processing unit 52 and the real-time processing unit 53 can be greatly improved due to the data processing support unit 54.

In addition, the data processing support unit 54 stores the converted data, which has been improved in quality by the batch processing or real time processing, in the distributed storage unit 70, (80).

The metadata unit 55 manages the metadata to generate, store, manage, and retrieve metadata about the transformed data.

The metadata is structured data on data, and is data assigned to a content according to a certain rule in order to efficiently find and use the information found in a large amount of information. That is, when the data request is received from the user equipment 200, the metadata unit 55 generates and manages metadata for each transformed data in order to extract requested data from a plurality of transformed data stored in the data rake, .

The metadata unit 55 can provide CI (Content Integration), CD (Content Delivery) and CD (Content Deployment) of the S / W. That is, the metadata unit 55 can provide content integration, content delivery, and content placement functions of the S / W.

In addition, the metadata unit 55 provides cloud busting and cloud spanning to provide the computing resources and storage resources.

The cloud bursting is an application distribution model used in a hybrid cloud (hybrid cloud) environment. When the computing capacity of the workflow server 100 is exceeded, the cloud bursting is automatically transferred to the public cloud due to excess demand, .

The cloud spanning is a delivery model in which application program components requiring a large amount of computing resources are simultaneously distributed in a plurality of cloud environments, and a plurality of computers can be connected and cooperated with each other.

In the lambda architecture unit 50, the business data is processed and transferred to the data security unit 60 to protect the generated conversion data.

The data security unit 60 encrypts the converted data to protect the data as shown in FIG. That is, the data security unit 60 protects the converted data by encrypting it, thereby preventing a security incident that occurs due to an increase in the number of users who make a decision using the large-capacity data analysis.

The data security unit 60 includes an encryption selection unit 61 for selecting whether or not to protect the conversion data, and a decryption unit 62 for decrypting the conversion data, And a password management unit 63 for managing a master encryption key for decrypting the converted data.

The encryption selection unit 61 may determine whether to protect the transformed data according to the degree of importance of the transformed data before storing the transformed data, and may select whether to proceed with the encryption.

When the encryption selection unit 61 does not protect the conversion data, the conversion data is stored in the dispersion storage unit 70. [

On the other hand, if the encryption selecting unit 61 selects to protect the converted data, the encrypted data is encrypted through the encryption setting unit 62 before the converted data is stored in the distributed storage unit 70.

The encryption setting unit 62 sets a password for the conversion data when the encryption selection unit 61 is selected to encrypt the conversion data. The password for encrypting the conversion data may be directly input or automatically input by the encryption setting unit 62. [

The encryption of the conversion data set through the encryption setting unit 62 can be managed by the encryption management unit 63. [

The encryption management unit 63 can manage a master encryption key for decrypting the encrypted data. The master encryption key can decrypt a plurality of encrypted conversion data.

The cryptographic management unit 63 can decrypt the cryptographic data by managing the master encryption key.

Meanwhile, the data security unit 60 may use a standard TLS (Transport Layer Security) protocol while the business data is transmitted from the task console 100 to the data acquisition unit 20. That is, the data security unit 60 can protect business data transmitted from the business entity 100 to the data acquisition unit 20 using a standard TLS (Transport Layer Security) protocol, It is possible to prevent the liquid from leaking to the outside.

It is possible to prevent the data from being leaked to the unauthenticated user equipment 200 due to the data security unit 60. [ That is, since the converted data is encrypted before being permanently stored in the distributed storage unit 70 by the data security unit 60, and the data is decrypted before being retrieved, the user device 200 accessing the data provides transparency And the user device 200 does not need to change the code in order to encrypt or decrypt the data, thereby providing convenience to the users.

Also, the data security unit 60 authenticates the user device 200 requesting the conversion data. That is, the data security unit 60 can check whether the user device 200 requesting the conversion data has the right to receive the conversion data.

The data security unit 60 includes a user authentication unit 64 for authenticating the user information received from the user device 200 and a user authentication unit 64 for using the converted data And an IP address verifying unit 66 for verifying the IP address of the user device 200. The IP address verification unit 66 determines whether the IP address of the user device 200 is authorized.

The data security unit 60 includes a user authentication unit 64, an authorization unit 65 and an IP address verification unit 66 to authenticate the work server 100 or the user equipment 200. [ May be used.

The user authentication unit 64 is for authenticating the user device 200, and preferably stores user information in advance.

The user authentication unit 64 receives the user information generated in the user equipment 200 and determines whether the user information is included in the user information stored in advance. That is, the user authentication unit 64 determines whether the user information provided from the user device 200 is stored in the user information stored in advance, and authenticates the user device 200 when stored in the user information stored in advance , And does not authenticate the user device 200 if it is not stored in the user information stored in advance.

The user authentication unit 64 should not provide data to the unauthenticated user equipment 200. [

At this time, the user authentication unit 64 may use multi-step authentication.

The authority granting unit 65 grants a use right through the user information received from the user equipment 200 authenticated by the user authenticating unit 64. In other words, the right of use of the conversion data, which can be provided for each user information previously stored, should be set in the right granting unit 65. In accordance with the user information of the authenticated user equipment 200, .

Here, the authority granting unit 65 may include an account related task by RBAC (Role Based Access Control) granting authority to a position and a task related to data and computing resource related to attribute based access control (ABAC) And the authorization for.

The access right to the data rake can be controlled by the right granting unit 65 so that even if the authenticated user equipment can not receive all the converted data, important information can be prevented from being leaked.

The IP address confirmation unit 66 previously sets a range of IP addresses that can provide the conversion data and sets the IP address of the user device 200 to the IP address of the user device 200, Range.

When the user IP address of the user device 200 is included in a preset IP address range, the user device 200 authenticates the converted user data so as to provide conversion data, And is not authenticated so as not to be able to provide the conversion data to the user device 200 when the user device 200 is not included in the IP address range.

The security of the data rake can be greatly improved by checking the user IP address of the user device 200 through the IP address verifying part 66. [

On the other hand, the data security unit 60 can perform audit by a log of data management activities and diagnosis by metadata of data-related activities.

The distributed storage unit 70 stores the encrypted conversion data through the data security unit 60. [

The distributed storage unit 70 may support serial operations and random operations, and may be flexible and extensible for storage of multiple data structures.

In addition, the distributed storage unit 70 can define the overall solution responsiveness of the lambda architecture unit 50 with respect to the task data and the data stream received through the data acquisition unit 20. That is, the distributed storage unit 70 determines the responsiveness in accordance with the slowest system in the connection chain, and when the distributed storage unit 70 is not fast, the real-time processing unit 53 of the lambda architecture unit 50 performs So that the lambda architecture unit 50 interferes with real-time processing.

The data providing unit 80 is for providing the stored conversion data to the user device 200 desiring to use the converted data.

The data providing unit 80 may transmit the stored converted data to the user equipment 200 through a data service or an exporting method. At this time, the converted data may be transmitted to the user equipment 200 through a message, File, data dump, or the like.

The data providing unit 80 also provides a merged view of the batch processed data and the real-time processed data, and must be scalable and responsive to the application consuming the converted data.

6, the user equipment 200 desiring to receive the conversion data through the data providing unit 80 may be a user or a system that consumes data, as shown in FIG. 6, A user information part 220 for generating user information, and a user IP address for identifying the user information and the user device 200, And a data request unit 230 for requesting data.

The user communication unit 210 can communicate using the Internet or the like, but is not limited thereto.

The user information unit 220 generates user information by inputting information such as a user's ID, title, and the like. And can be distinguished from other users by generating user information through the user information unit 220.

The data request unit 230 generates a request signal including the user information generated in the user information unit 220 and the user IP address. That is, the data requesting unit 230 generates a request signal so as to be able to identify the other user including the user information and the user IP address to request the conversion data.

The request signal generated in the data request unit 230 is transmitted to the data security unit 60 and the data providing unit 80 through the user communication unit 210. [

As shown in FIG. 7, the data rake constructed through the data rake framework configured as described above acquires the business data generated from the business console 100 through the data acquiring unit 20 S10).

And transfers the obtained business data to the lambda architecture unit 50 through the data transfer unit 30. [

The lambda architecture unit 50 models the received business data, converts the converted business data into converted data, and then selects one of batch processing and real time processing (S11, S12, S13).

At this time, the lambda architecture unit 50 improves the quality after batch processing or real-time processing of the converted data.

In addition, the lambda architecture unit 50 generates and stores metadata of the transformed data (S14), and transmits the transformed data to the data security unit 60. [

The data security unit 60, which has received the conversion data, selects whether the converted data is secured or not (S15).

If the data security unit 60 determines that the conversion data is required to be secured, the encryption setting unit 62 sets a password for the conversion data (S16).

The encryption management unit 63 transmits a password to the encryption management unit 63 so as to decrypt the encrypted conversion data in the encryption setting unit 62, The key must be managed.

The converted data encrypted through the encryption setting unit 62 is transferred to the distributed storage unit 70 and stored (S17).

On the other hand, if the data security unit 60 does not need the security of the converted data, the converted data is transmitted to the distributed storage unit 70 and stored (S17).

Meanwhile, in order to receive data from the data rake constructed through the data rake framework according to an embodiment of the present invention, as shown in FIG. 8, first, the user device 200 requests to provide conversion data A request signal including user information and a user IP address is generated.

The user equipment 200 transmits a request signal to the data security unit 60 and the data providing unit 80 to request the provision of the converted data at step S20.

Upon receipt of the request signal from the user equipment 200, the data security unit 60 determines whether the user information of the user equipment 200 is registered in advance (S21).

That is, the data security unit 60 authenticates the user device 200 when the user information of the user device 200 is registered in advance through the user authentication unit 64, If the information is not registered in advance, the user device 200 is not authenticated so that the converted data is not transmitted.

The authenticated user information is transmitted to the right granting unit 65 through the user authenticating unit 64 to grant the right to use the converted data for the user information at step S22.

Here, the user IP address included in the request signal of the user device 200 to which the use right is granted may be checked through the IP address verifying unit 66 to see if it is included in the preset IP address range.

When the user IP address of the user equipment 200 is included in the IP address range, the user equipment 200 is connected to the data providing unit 80 so that the user equipment 200 can receive the converted data (S23).

If the user IP address of the user device 200 is not included in the IP address range, the user device 200 and the data provider 80 are not connected.

The data rake framework for building the data rake used above can speed up the GPU and SSD by capturing, processing, analyzing and storing large amounts of data generated by the business system from the enterprise or organization on a data flow basis So that it is possible to prevent an overload.

In addition, metadata can be captured and managed from the perspective of data traceability, data lineage, and security based on data sensitivity across the lifecycle while data is flowing, enabling users using the data or systems consuming the data to efficiently and efficiently Can be extracted as well as security can be improved.

In addition, when data is stored, it is possible to select whether or not the data is encrypted, thereby protecting the data and preventing the data from being leaked to the outside.

The embodiments of the present invention described above and shown in the drawings should not be construed as limiting the technical idea of the present invention. The scope of protection of the present invention is limited only by the matters described in the claims, and those skilled in the art will be able to modify the technical idea of the present invention in various forms. Accordingly, such improvements and modifications will fall within the scope of the present invention if they are apparent to those skilled in the art.

10: physical resource 11: computing resource
12: Storage resources 13: Networking resources
20: data acquisition unit 30:
40: transfer rate control section 50: lambda architecture section
51: phase data analysis unit 52:
53: real-time processing unit 54: data processing support unit
55: Metadata section 60: Data security section
61: encryption selection unit 62: password setting unit
63: password management unit 64: user authentication unit
65: permission authority 66: IP address verification unit
70: Distributed storage unit 80: Data providing unit
100: Business magnetic device 200: User device
210: user communication unit 220: user information unit
230: Data request unit

Claims (4)

A data acquiring unit (20) for acquiring the business data, the data acquiring unit (20) being connected to a business entity (100)
A data transfer unit 30 for separating the work magnetic unit 100 connected to the data acquisition unit 20 and supporting communication for transferring the acquired work data,
A transmission rate control unit 40 for controlling the transmission rate of business data transmitted through the data transmission unit 30,
A lambda architecture unit 50 for modeling business data transferred through the data transfer unit 30, converting the data into transformation data, performing batch processing or real-time processing, and generating and managing metadata for the transformation data,
A data security unit 60 for encrypting the converted data to protect the converted data from being leaked to the outside,
A distributed storage unit 70 for storing the encrypted conversion data,
And a data providing unit (80) for extracting and providing the converted data to a user device (200) desiring to use the stored converted data.
The method according to claim 1,
The lambda architecture unit 50 includes a phase data analysis unit 51 for performing data cleaning, processing, modeling, and conversion of the business data into converted data; A real-time processing unit 53 for real-time processing the converted data converted by the phase data analyzing unit 51; a batch processing unit 52 for improving the quality of the converted data; A data processing support unit 54 for supporting machine learning and data science processing based on the business data to a real time processing unit 53 and a metadata unit 55 for generating and managing metadata for the converted data, The data rake framework comprising:
The method according to claim 1 or 2,
The data security unit 60 includes an encryption selecting unit 61 for selecting whether to protect the converted data, an encryption setting unit 62 for setting a password for protecting the converted data, And a password management unit (63) for managing a master encryption key for decrypting the converted data.
The method of claim 3,
The user device 200 includes a user communication unit 210 for communicating with the data security unit 60 and the data providing unit 80, a user information unit 220 for generating user information by inputting user information, And a data request unit 230 for generating a request signal including the generated user information and a user IP address for identifying the user equipment 200 to request converted data,
The data security unit 60 includes a user authentication unit 64 for determining whether the user information included in the request signal received from the user equipment 200 is previously registered user information and authenticating the user information, (65) for granting use rights of the converted data through the authenticated user information in the request signal, an IP address verification unit (65) for checking whether the user IP address included in the request signal is included in the preset IP address range, (66). ≪ / RTI >

Images