WO2022107994A1 - Big data augmented analysis profiling system - Google Patents

Abstract

The purpose of the present invention is to propose a big data augmented analysis profiling system which can increase data processing efficiency, and the present invention provides a big data augmented analysis profiling system comprising: a data collection module; a data preprocessing module; a data quality management module; an augmented analysis profiling module; and an augmented analysis collaboration module.

Description

Big Data Augmented Analysis Profiling System

The present invention relates to a big data augmented analysis profiling system.

Big data analysis is a technology that extracts values from data and analyzes the results from a set of large (tens of terabytes) structured or unstructured data that exceeds the capabilities of existing database management tools.

The development of big data technology, which is characterized by the generation, collection, analysis, and expression of various types of large-scale data, more accurately predicts the diversified modern society, makes it work efficiently, and provides customized information for each individualized modern society member; It makes management and analysis possible, and it also enables technology that was previously impossible to be realized.

As such, big data presents the possibility of providing valuable information to society and mankind in all areas such as politics, society, economy, culture, and science and technology, and its importance is being highlighted.

In general, big data analysis goes through data collection/storage, data preprocessing, data purification (profiling), data analysis, and data visualization.

Recently, data sources for big data collection include unstructured data such as voice, document, and SNS data, semi-structured data such as log data, machine data and operational data, and structured data such as DB/DW.

Because big data analysis requires a series of complex processes from data collection to visualization, it is necessary to shorten the data processing time and increase the efficiency of data processing while sharing the work between other users.

In order to solve the problems of the prior art, the present invention intends to propose a big data augmented analysis profiling system capable of increasing data processing efficiency.

In order to achieve the above object, according to an embodiment of the present invention, as a big data augmented analysis profiling system, receiving data including at least one of unstructured data, semi-structured data, and structured data from a data source a data collection module for classifying a class of the filtered data by extracting characteristics of the data, and storing the classified data in one or more databases; a data pre-processing module that divides data stored in the one or more databases and improves the data processing capacity per node by applying a multi-threading technique for simultaneous pre-processing of the divided data; A data quality management module that improves data quality by adding regular expressions (Regular Expressions or Regex) as metadata to detect missing values, outliers, or sensitive information in data according to the characteristics of the data domain; an augmented analysis profiling module that configures a workflow according to a user's purpose and provides a machine learning-based runtime environment; and the data collection module, the data preprocessing module, the data quality management module, and the augmented analysis profiling module provide an interface that can identify the author, comment and data throughput for the process performed in each module, A big data augmented analysis profiling system is provided that includes an augmented analysis collaboration module that records a log for the process, machine learning the recorded log, and provides guide information to the process performed by other workers.

The data collection module may pre-classify a class according to a feature extracted from each of a plurality of data using a preset algorithm.

The characteristics of the data may include a type, a domain, and a maximum/minimum value of each of one or more fields.

The type may be at least one of number, text, and binary data, and the domain may be at least one of category, date, time, amount, coordinate, percentage, fraction, and exponent.

The data preprocessing module may include a memory cache that provides a metadata sharing management function for a large amount of data preprocessing, such as a preprocessing target list, a preprocessing definition template (Recipe), and data division processing.

The data pre-processing module may include a Split module performing a multi-thread management function of dividing the pre-processing target data, and creating and retrieving a thread for pre-processing the divided data.

The data pre-processing module may include an integration module that integrates and stores pre-processed data from the split module.

You can manage semantic meta information about data dictionary, keyword, and regular expression.

The regular expression may be configured based on a rule, manage data quality such as detection and classification according to data characteristics, and may provide a user-defined function such as registering and modifying a regular expression algorithm for a data dictionary.

The augmented analysis collaboration module may output a workspace interface capable of creating, deleting, and modifying each work process.

Advantageous Effects of Invention According to the present invention, there is an advantage in that it is possible to increase the identification accuracy of a data domain, and increase the accuracy of the data quality evaluation and the accuracy of the analysis result.

1 is a diagram showing the configuration of a big data augmented analysis platform according to a preferred embodiment of the present invention.

2 is a view for explaining a data collection process of the data collection module according to an embodiment of the present invention.

3 to 4 are diagrams illustrating data of different classes.

5 is a diagram showing the architecture of the data pre-processing module according to the present embodiment.

6 is a diagram for explaining a data quality management module according to an embodiment of the present invention.

7 is a diagram illustrating the configuration of an augmented analysis profiling module according to an embodiment of the present invention.

8 is a diagram exemplarily illustrating a workspace interface provided by the augmented analysis collaboration module according to the present embodiment.

9 is a diagram exemplarily illustrating a collaboration interface provided by the augmented analysis collaboration module according to the present embodiment.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

1 is a diagram showing the configuration of a big data augmented analysis platform according to a preferred embodiment of the present invention.

As shown in FIG. 1 , the big data augmented analysis platform according to this embodiment includes a data collection module 100 , a data preprocessing module 102 , a data quality management module 104 , an augmented analysis profiling module 106 and augmented analytics collaboration module 108 .

The data collection module 100 intelligently collects and classifies data for big data analysis.

2 is a view for explaining a data collection process of the data collection module according to an embodiment of the present invention.

1, the big data intelligent collection module 100 according to the present embodiment is connected to the data source 200 through a network, and stores the collected data in one or more databases 202-n.

Data collected from the data source 200 may include structured data, semistructured data, and unstructured data.

Structured data is data and files stored in fixed fields (columns) like tables in a relational database system, and spreadsheet-type data in which data properties are distinguished by specified rows and columns.

In the case of structured data, since it has a schema structure, the data search process is standardized in the order of table search, column structure search, and row search.

Semi-structured data has metadata corresponding to the schema of structured data inside the data. As data generally stored in a file format, it includes log data, machine data, and operation data.

In the case of semi-structured data, it is necessary to understand what type of data it is because it has metadata about the data structure inside the data. Apply parsing rules that can parse data by identifying regularities within data.

The unstructured data includes text data capable of language analysis such as voice, document, and SNS data, and multimedia data such as voice, image, and video.

The data collection module 100 according to the present embodiment receives data including structured, semi-structured, and unstructured data from a data source 200 connected through a network, and classifies the input data.

The data collection module 100 according to the present embodiment classifies data classes by extracting features of data.

According to this embodiment, the class classification process may be performed through a pre-machine-learning algorithm.

According to this embodiment, k-NN, RNN, and BERT algorithms are machine-learned in advance with data for which the class is known.

After the pre-machine learning is completed, when new data is input, the data collection module 100 extracts features of the data, and calculates the probability of belonging to each of a plurality of classes by using the extracted features as input values to the machine-learning algorithm. .

A data classification database may be built through pre-machine learning, and the data classification database stores a plurality of predefined classes and information about features corresponding to each class.

Characteristics of data may include a type, a domain, and a maximum/minimum value of each of one or more fields (columns).

Here, the type may be numeric, text, and binary data, and the domain may include category, date, time, amount, coordinate, percentage, fraction, and exponent.

Data has a plurality of fields, and the data collection module 100 inputs the types, domains, and maximum/minimum values of the plurality of fields into a pre-machine-learning algorithm to compare classes having characteristics similar to those of the corresponding data, and Calculate the probability value for each class of

3 to 4 are diagrams illustrating data of different classes.

Referring to FIG. 3 , the data collection module 100 inputs types, domains, and maximum/minimum values of a plurality of fields into a pre-machine learning algorithm to compare classes having characteristics similar to those of the corresponding data, and a plurality of classes Calculate the probability value for each.

In FIG. 3 , the first field is time, and the second to fifth fields are numbers having a maximum/minimum value, so the data collection module 100 may determine the corresponding data as a weather-related class.

In addition, as shown in FIG. 4 , when a predetermined integer repeatedly appears in the first field, the second field is data having minimum and maximum values, and the third field is text and the same text appears repeatedly, these features are extracted Thus, it is classified into a class related to a room.

5 is a diagram showing the architecture of the data pre-processing module according to the present embodiment.

Referring to FIG. 5 , the data preprocessing module 102 according to the present embodiment divides data in order to improve the file-based (Comma-Separated Values, CSV) data preprocessing capacity constraint, and divides the data for simultaneous preprocessing of the divided data. By applying a multi-threading technique, the data processing capacity per node is improved.

The memory cache of the data preprocessing module 102 provides a meta information sharing management function for preprocessing large amounts of data, such as a preprocessing target list, a preprocessing definition template (Recipe), and data division processing.

The Split module of the data pre-processing module 102 performs a multi-thread management function of dividing the pre-processing target data and creating and retrieving a thread for pre-processing the divided data.

The integration module of the data pre-processing module 102 integrates and stores pre-processed data from the Split module.

Distribute File Storage (DFS) of the data preprocessing module 102 supports data preprocessing storage using remote distributed file storage.

The data preprocessing module 102 according to this embodiment enables data management in a memory-based “key-value” structure to improve data processing speed, and enables fast data access through an in-memory store (Redis (Remote Dictionary Server)) to share meta-information necessary for data division and simultaneous data processing.

Here, Redis is a non-relational database that guarantees fast read and write speeds for storing and retrieving all data in memory, and supports five data types: String, Set, Sorted Set, Hash, List.

In addition, the data pre-processing module 102 uses HDFS-based remote distributed file storage to enable data pre-processing through external storage in addition to local storage.

In addition, the data pre-processing module 102 applies volatile to enable main memory sharing of a data stream from HDFS by applying a Java threading technique to secure data integrity during pre-processing and integration of divided data.

6 is a diagram for explaining a data quality management module according to an embodiment of the present invention.

Referring to FIG. 6 , the data quality management module 104 defines a domain to be identified and registers the domain as metadata. In addition, the data quality management module 104 improves data quality by adding a regular expression (Regular Expression or Regex) as metadata to detect missing values, outliers, or sensitive information of data according to the characteristics of the domain of the managed data. .

The data quality management module 104 manages semantic meta information for a data dictionary, a keyword, and a regular expression.

Data dictionary can be added and changed, and data categories can be viewed, registered, and edited to expand data domains that require additional data quality measurement.

The regular expression according to the present embodiment can be configured based on rules, manages data quality such as detection and classification according to data characteristics, and provides a user-defined function such as registration and modification of a regular expression algorithm for a data dictionary.

Data missing values and outliers can be detected and sensitive information can be identified by the regular expression defined in the data quality management module 104 .

In addition, domain automatic classification is possible based on the metadata for the regular expression defined by the data quality management module 104 .

In the data quality management module 104, Imputation (mean, regression) for data quality improvement is modularized and applied, and data distribution and statistical measurement functions for data quality improvement are applied.

According to this embodiment, an Apache Lucene-based search engine for improving domain automatic identification performance can be applied, and a large amount of data domains can be determined through a search index.

In relation to the Apache Lucene-based search engine, it is possible to analyze the Apache Lucene index file structure and search results through Look, and it is possible to create, inquire, and delete indexes for data dictionary using Apache Lucene Open API. It also shares semantics for data integration with the data pre-processing module 102 .

7 is a diagram illustrating the configuration of an augmented analysis profiling module according to an embodiment of the present invention.

The augmented analysis profiling module 106 according to this embodiment configures a workflow so that the augmented analysis profile step is executed according to the user's purpose, and a runtime for independent execution and organic combination of the profiling module based on machine learning It is developed as a framework that can provide a (run-time) environment.

Referring to FIG. 7 , the service pool provides a pool for executing machine learning-based augmented analysis profiling.

The augmented analysis profiling module 106 defines the augmented analysis profiling execution flow, and generates Config necessary for all executions such as data loading and storage. In addition, it performs the function of configuring and distributing pipelines.

The pipeline performs management functions such as execution sequence of augmented analytics profiling, batch execution of predefined workflows such as iteration cycle, stop and recall, etc.

According to this embodiment, the augmented analysis profiling module developed as a single module is wrapped with Apache Beam to enable streaming processing in the pipeline.

Apache Beam can be deployed and run on various runtimes such as Google Dataflow, Spark, etc. In addition, various Artifacts are provided to provide common functions such as Data Read/Write and interface between modules.

Two types of execution environments can be provided for pipeline runtime: Direct Runner and Apache Spark Runner.

Direct Runner does not need to configure a separate external analysis cluster, so it can be used in the development/test stage, and in the actual production execution environment, it is provided in a stand-alone form so that Apache Spark can be utilized.

The pipeline executed through Apache Spark provides a function so that the augmented analysis profiling module 106 can read data from externally configured HDFS and RDBMS or write analysis results.

The augmented analysis collaboration module 108 according to this embodiment enables collaboration between a plurality of workers in performing a data collection/refinement/profiling analysis process.

As shown in FIG. 8 , the augmented analysis collaboration module 108 according to the present embodiment provides a workspace interface capable of managing each work process.

In the workspace interface according to the present embodiment, each worker may create, modify, and delete a work process.

In addition, the augmented analysis collaboration module 108 provides a history management and sharing service for the analyzed profiling.

As shown in FIG. 9 , the augmented analytics collaboration module 108 can identify authors, comments, and data throughput for processes performed in a series of modules such as data collection, preprocessing, META profiling, and visualization profiling. It provides an interface and records the log of the process performed by each module.

The recorded log can be provided as guide information for the process performed by other workers later through machine learning.

The process in each module for big data augmented analysis profiling analysis according to the present embodiment may be performed in a device including a processor and a memory.

The processor may include a central processing unit (CPU) or other virtual machine capable of executing a computer program.

The memory may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. The memory may also include volatile memory, such as various random access memories.

The above-described embodiments of the present invention have been disclosed for purposes of illustration, and various modifications, changes, and additions will be possible within the spirit and scope of the present invention by those skilled in the art having ordinary knowledge of the present invention, and such modifications, changes and additions should be regarded as belonging to the following claims.

Claims (10)

As a big data augmented analysis profiling system, receiving data including at least one of unstructured data, semi-structured data, and structured data from a data source, extracting features of the data, classifying the class of the filtered data, and storing the classified data in one or more databases data collection module to store; a data pre-processing module that divides data stored in the one or more databases and improves the data processing capacity per node by applying a multi-threading technique for simultaneous pre-processing of the divided data; a data quality management module that improves data quality by adding regular expressions (Regex) as metadata to detect missing values, outliers, or sensitive information in data according to the characteristics of the data domain; an augmented analysis profiling module that configures a workflow according to a user's purpose and provides a machine learning-based runtime environment; and Provides an interface that can identify authors, comments, and data throughput for the processes performed in the data collection module, the data preprocessing module, the data quality management module, and the augmented analysis profiling module, and the process performed in each module A big data augmented analysis profiling system including an augmented analysis collaboration module that records logs for and provides guide information to the processes performed by other workers by machine learning the recorded logs. According to claim 1, The data collection module, A big data augmented analysis profiling system that pre-classifies classes according to features extracted from each of a plurality of data using a preset algorithm. 3. The method of claim 2, The data feature is a big data augmented analysis profiling system including a type, domain, and maximum/minimum values of each of one or more fields. 4. The method of claim 3, the type is at least one of numeric, text, and binary data; The domain is at least one of a category, date, time, amount, coordinates, percentage, fraction, and index of a big data augmented analysis profiling system. According to claim 1, The data preprocessing module is a big data augmented analysis profiling system including a memory cache that provides a metadata sharing management function for a large amount of data preprocessing such as a preprocessing target list, a preprocessing definition template (Recipe), and data division processing. 6. The method of claim 5, The data pre-processing module is a big data augmented analysis profiling system including a Split module that divides pre-processing target data, and performs a multi-thread management function of creating and retrieving threads for pre-processing of the divided data. 7. The method of claim 6, The data pre-processing module is a big data augmented analysis profiling system including an integration module for integrating and storing the pre-processed data from the Split module. According to claim 1, Big data augmented analysis profiling system that manages semantic meta information for data dictionary, keyword, and regular expression. 9. The method of claim 8, Big data augmented analysis profiling that the regular expression can be configured based on rules, manages data quality such as detection and classification according to data characteristics, and provides user-defined functions such as registration and modification of regular expression algorithms for data dictionary system. 9. The method of claim 8, The augmented analysis collaboration module is a big data augmented analysis profiling device that outputs a workspace interface that can create, delete and modify each work process.

Images