CN114676198A - Benchmark evaluation system for multimode database and construction method thereof - Google Patents

Abstract

The invention discloses a multi-mode database-oriented benchmark evaluation system and a construction method thereof. The system includes a distributed database module, a resource monitoring module, a message transmission module, a workload execution module and an evaluation record management module. The method includes the following steps: (1) Install and deploy related database clusters; (2) Install and deploy Prometheus and Grafana resource monitoring software; (3) Implement asynchronous message transmission queues based on Zookeeper and Kafka; (4) Implement load execution programs based on Java language, It is used to connect asynchronous message transmission queues and distributed database clusters; (5) Based on Java language and front-end framework vue, it implements evaluation record management, and provides visual pages for evaluation task creation and evaluation results viewing.

Description

A benchmarking system for multimodal database and its construction method

technical field

The invention relates to a benchmark evaluation system oriented to a multimodal database and a construction method thereof, and the technical field of software performance evaluation and measurement analysis.

Background technique

There is an increasing demand for analyzing and processing multimodal data, which can be divided into structured data, semi-structured data and unstructured data. Structured data includes relational data, key-value pairs, and graph data. When it is necessary to manage multimodal data, which multimodal database to choose is an unavoidable key issue. Existing research work on multi-modal databases mainly focuses on comparing the differences in execution time of multi-modal databases, without paying attention to the resource status of servers. If the server runs out of memory, the database will also face downtime. In a distributed environment, system failures are considered normal events rather than unexpected events, so the evaluation of database resource consumption is crucial.

SUMMARY OF THE INVENTION

Purpose of the invention: Aiming at the problems and deficiencies in the prior art, the present invention provides a multi-modal database-oriented benchmark evaluation system and its construction method, and provides a quantifiable comparison solution for multi-modal data processing.

Technical solution: A benchmark evaluation system oriented to a multi-modal database, comprising: a distributed database module, a resource monitoring module, a message transmission module, a workload execution module and an evaluation record management module. The message transmission module provides source data for the workload execution module; the workload execution module stores the data read from the message transmission module to the distributed database module; the resource monitoring module collects resource data during workload execution ; The evaluation record management module is responsible for managing the execution results of the workload execution module.

The distributed database module includes two multimodal databases: ArangoDB and OrientDB. The distributed database module is the basis of the evaluation system and the main body of the evaluation system.

The resource monitoring module is composed of distributed monitoring systems Prometheus and Grafana, and is used to evaluate the resources occupied by the database. The resource monitoring module can monitor the resource usage such as CPU, memory, bandwidth, and throughput of the server. This system mainly monitors CPU and memory usage.

The message transmission module implements asynchronous message transmission queues by Zookeeper and Kafka. In this asynchronous message transmission queue, Zookeeper acts as a registry, so that the workload execution module can asynchronously obtain data from Kafka. Kafka is used as a stream data processing platform. Before the evaluation starts, the message transmission queue producer stores the data in the data set into Kafka through the Java program to provide data for consumers. The message transmission queue implemented by Zookeeper and Kafka not only realizes the decoupling of producers and consumers, but also can adapt to the load processing speed to achieve asynchronous communication.

The workload execution module designs a large number of loads by simulating real big data application scenarios. It can be roughly divided into three types of workloads: single-table addition, deletion, modification and query workloads, multi-threaded workloads, and multi-table joint query workloads. Single-table CRUD workloads only perform CRUD operations on one table at a time, thereby simulating data migration and data instantiation operations; multi-threaded workloads create multiple threads to perform tasks, thus simulating big data application scenarios The database handles high concurrent operations of multiple requests at the same time; the multi-table joint query workload simulates operations with complex business logic in big data scenarios through joint query between tables.

The evaluation record management module lowers the threshold of database evaluation by constructing an easy-to-interact Web page, and enhances the interactivity of database evaluation. The evaluation record management module is divided into two parts: the task submission part and the evaluation record viewing part. In the task submission section, users can select relevant parameters through the Web page and submit evaluation tasks. In the evaluation record viewing section, users can view the status and parameters of all evaluation tasks.

A method for constructing a benchmark evaluation system oriented to a multimodal database, comprising the following steps:

(1) In order to improve the reliability and availability of the database, and also to simulate real large database application scenarios. First, you need to deploy the master-slave cluster for the database installation involved in the evaluation.

(2) In order to accurately count the resource occupancy of the server, the distributed resource monitoring software Prometheus and Grafana need to be installed. Prometheus is essentially a time series database, which realizes the purpose of resource monitoring by regularly sampling the resource situation of the server. Grafana is a graphical interface tool that can present the data collected by Prometheus to the interface. The resource monitoring software provides various server resource monitoring functions including CPU, memory, bandwidth, and throughput. This system mainly monitors two types of resources: CPU and memory.

(3) In order to realize the decoupling of data input and output and realize asynchronous communication, it is necessary to install and deploy an asynchronous message transmission queue composed of Zookeeper and Kafka. In this message queue, Zookeeper is used as a registry to achieve load balancing of messages, and Kafka is used as a stream data processing platform. The use of message queues can solve the problem of the mutual influence of data input and output, avoid a link error, and the entire process must be restarted, realizing the decoupling of input and output. Use message queues to solve data input and output problems. If you don't use message queues and let the program read files directly, the whole system is coupled together, and if there is a problem in one link, the evaluation will have to start all over again. After using the message queue, the input and output links are separated, and even if there is a problem, there is no need to redo it all. In addition, this also helps to locate the wrong location. Secondly, the message queue can cache data. The producer stores the data in Kafka in advance, and the consumer fetches it when needed, realizing the function of asynchronous communication.

(4) The workload execution module is implemented in Java language, and converts the workload into a runnable program by coding. The workload execution module needs to be deployed in the JDK1.8 environment. This module contains a large number of workloads, which can be mainly divided into three categories: single-table addition, deletion, modification and query workloads, multi-threaded workloads, and multi-table joint query workloads.

(5) The evaluation record management module is implemented in Java language and Vue front-end framework. This module provides evaluation task submission function and evaluation record viewing function. It needs to be deployed in the environment of JDK1.8 and nodeJS.

In the step (3), an asynchronous message transmission queue is implemented with Zookeeper and Kafka. The producer of the message transmission queue writes the data in the multi-mode data set into Kafka through a Java program before the evaluation starts. After the evaluation starts, the work The load execution module reads data from Kafka as a consumer of the asynchronous message transmission queue. The specific steps are:

Input: multimodal dataset;

Output: Kafka stream data;

First, the input stage determines whether the next line of the multi-mode data set file exists. If it exists, the next line is read and sent to Kafka. If the next line does not exist, it is read to the end of the file, and the input stage ends. Secondly, in the output stage, the consumer connects to Kafka and consumes the data sent to Kafka through the corresponding topic in the input stage, until the consumer actively closes the connection and the output ends.

In the step (4), the workload execution module implemented by the Java language has the following specific algorithm steps:

Input: load number, database number to be evaluated;

Output: execution time, CPU and memory occupied;

After the message queue obtains the data, the workload execution module starts to run the program according to the load number selected by the user and the database number to be evaluated. First, the system creates a record marked for evaluation, which contains only the load number and database label when it is created. The system executes the workloads in the order in which the records were submitted, and marks the currently executing records as under evaluation. During the running of the program, the running time of each record will be saved, and the resource monitoring module composed of Prometheus and Grafana will be called to obtain the CPU and memory usage. After the execution of the workload program is completed, the workload execution module saves the execution time and resource occupancy in the database, and marks the current record as evaluated.

In the step (5), the evaluation record management module provides users with an easy-to-interact Web interface, which is divided into two parts:

(a) The evaluation task submission part, which presents parameters such as load number and database number in the form of options for users to choose. At the same time, there is detailed description information for each task. By selecting the corresponding parameters and clicking the execute button, the workload execution module will create a task to be evaluated and save it in the database;

(b) Evaluation record viewing section, which displays all evaluation records. Users can filter the evaluation records they want to view according to the evaluation status, load type, and database number.

In order to simulate real big data application scenarios, the system has designed a large number of load types, which can be divided into the following three categories: single-table addition, deletion, and modification workloads, multi-threaded workloads, and multi-table joint query workloads. Single-table addition, deletion, modification and query workloads are used to simulate scenarios of data migration and data persistence; multi-threaded workloads are used to simulate high-concurrency application scenarios; multi-table joint query workloads are used to simulate scenarios with complex business logic. At the same time, in order to more objectively evaluate the performance of multi-mode database processing multi-mode data, the specific steps of the evaluation are: before the evaluation starts, build a distributed master-slave cluster for all databases to improve the reliability and availability of the system. Install and deploy distributed resource monitoring systems Prometheus and Grafana to monitor server performance. Implementing Kafka producers in Java to store data sets in Kafka not only realizes the decoupling of producers and consumers, but also enables asynchronous communication by adapting to the load processing speed. When the evaluation begins, the user submits the evaluation task through the Web evaluation record management module provided by the system, and the workload execution module creates a new evaluation task after receiving the task, and updates the evaluation result to the database after the evaluation. After the evaluation, users can view the evaluation results through the Web page provided by the system.

The beneficial effects of the present invention are as follows: according to the characteristics of multi-modal data, a benchmark evaluation system oriented to multi-modal database and its construction method are proposed. According to the load type provided by the system, users can simulate real big data application scenarios, and from Accurate, fair and fast evaluation of multimodal databases in terms of execution time and resource consumption. Thus, a quantifiable and comparable solution is provided for the management of multimodal data.

Description of drawings

FIG. 1 is a schematic structural diagram of an embodiment of the present invention.

Detailed ways

Below in conjunction with specific embodiments, the present invention will be further illustrated, and it should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. The modifications all fall within the scope defined by the appended claims of this application.

As shown in Figure 1, a benchmark evaluation system for multi-modal databases includes: a distributed database module, a resource monitoring module, a message transmission module, a workload execution module, and an evaluation record management module.

The distributed database module involves a total of two multimodal databases: ArangoDB and OrientDB. Each database is installed and deployed on Alibaba Cloud servers in the form of master-slave distributed clusters. The distributed database module is the basis of the evaluation system and the main body of the evaluation system.

The resource monitoring module is composed of distributed resource monitoring software Prometheus and Grafana. It is used to monitor the resource usage during the execution load of the evaluation system. In order to comprehensively evaluate the performance of the database, the system not only evaluates the time required for each database to execute the load, but also evaluates the CPU and memory occupied by each database during the execution of the load.

The message transmission module consists of Zookeeper and Kafka. The message transmission queue not only realizes the decoupling of the system, but also can adapt to the load processing speed to achieve asynchronous communication. The producer generates and stores the data in the dataset into Kafka before the evaluation starts. After the evaluation starts, consumers consume the data in the queue according to the topic.

The workload execution module is implemented by the Java language and depends on the JDK1.8 version. This module simulates real big data application scenarios and designs a large number of loads. It can be divided into three categories: single-table addition, deletion and modification workloads, multi-threaded workloads, and multi-table joint query workloads. The single-table CRUD workload is used to simulate data migration and persistence scenarios. Multi-threaded workloads are used to simulate multi-threaded, high-concurrency big data application scenarios. The multi-table federated query workload is used to simulate real complex business logic application scenarios.

The evaluation record management module is implemented in Java language and Vue front-end framework. Depends on the basic environment of JDK1.8 and nodeJS. This module provides evaluation task submission function and evaluation record viewing function. On the evaluation task submission page, the user can select the workload and the database to be evaluated and submit the evaluation task to the background by clicking the Execute button. On the evaluation record viewing page, all evaluation records will be displayed, and users can filter the evaluation records they want to view according to the evaluation status, load type, and database number.

Before the evaluation starts, a series of preparations need to be done for each module. In the distributed database module, it is necessary to test the connectivity of each database to ensure the availability of the database. Simultaneously import the data into the database. In the resource monitoring module, you need to check whether Prometheus and Grafana are working properly, and whether the resource monitoring page can accurately monitor the server's resources. In the message transmission module, it is necessary to check whether Zookeeper and Kafka can run normally, and test whether the producer can store data in the message queue and whether the consumer can obtain data from the message queue according to different topics. In the workload execution module, it is necessary to test whether each load can run normally and whether the execution time and resource occupancy can be obtained after the load runs. In the evaluation record management module, it is necessary to test whether the evaluation tasks can be correctly submitted and whether the task status and information in the evaluation list are correct.

The invention also discloses a method for constructing a benchmark evaluation system of a multi-modal database, comprising the following steps:

Step 1: Download and install the database to be evaluated on the cloud server, and deploy it as a master-slave cluster. Since different multi-mode database installation and deployment and cluster building methods are different, taking ArangoDB as an example, the process of building a multi-mode database cluster is as follows: Install the corresponding version of ArangoDB according to the instructions on the ArangoDB official website.

(1) Add the repository key;

(2) Install ArangoDB using the apt-get command.

(3) Build a database cluster:

First start ArangoDB on the server that is the master node. By default, ArangoDB starts as a cluster.

Then start the remaining two slave nodes. After all the ArangoDB instances on the three virtual machines are started, the terminal will prompt that the multi-mode database can be accessed through the browser, and the ArangoDB cluster is successfully built.

Step 2: Install and deploy resource monitoring software Prometheus and Grafana. Prometheus is equivalent to a time series database for storing server resource usage. Grafana is used to display the data in Prometheus on the page in the form of graphs. The specific installation process is:

(1) Prometheus and Grafana are installed on one of the slave servers;

(2) Install the Prometheus exporter on the monitored server;

(3) Configure the server IP with the exporter installed into the Prometheus configuration file;

(4) Add Prometheus as source data in Grafana;

Finally, start Prometheus and Grafana. You can see the resource usage such as CPU, memory, bandwidth, and throughput by editing the formula on the Grafana page. This system mainly monitors the usage of CPU and memory. The formula for CPU utilization is:

((sum(increase(node_cpu_seconds_total{mode="idle"}[1m])) by(instance)) / (sum(increase(node_cpu_seconds_total[1m])) by(instance))))*100//Use the current node The occupied CPU is divided by the sum of the current node CPU to get the CPU utilization

The formula for memory utilization is:

((node_memory_MemTotal_bytes -node_memory_Buffers_bytes -node_memory_Cached_bytes -node_memory_MemFree_bytes -node_memory_Slab_bytes) /node_memory_MemTotal_bytes)*100// Divide the memory occupied by the current node by the sum of the memory of the current node to get the memory utilization

Step 3: Install and deploy Zookeeper and Kafka. In the message queue composed of Zookeeper and Kafka, Zookeeper is equivalent to a registry, and Kafka is equivalent to a stream data processing platform. The specific installation process is as follows:

(1) Download and decompress Zookeeper and Kafka in the three cloud servers;

(2) Add the IP and port information of the three servers to the Zookeeper configuration file:

server.1=172.16.150.154:2888:3888

server.2=172.16.150.155:2888:3888

server.3=172.16.150.156:2888:3888

(3) Modify the Zookeeper log directory and port information in the Kafka configuration file server.properties:

log.dirs=/opt/kafka/data/kafka-logs

zookeeper.connect=localhost:2181

(4) After starting Zookeeper and Kafka, enter the jps command in the console to verify whether the asynchronous message queue is successfully installed;

An asynchronous message transmission queue is implemented with Zookeeper and Kafka. The producer of the message transmission queue writes the data in the multi-modal data set (the multi-modal data set in this embodiment is the UniBench data set) to Kafka through a Java program before the evaluation starts. , after the evaluation starts, the workload execution module reads data from Kafka as a consumer of the asynchronous message transmission queue. The specific steps are:

Input: UniBench dataset;

Output: Kafka stream data;

First, the input stage determines whether the next line of the UniBench dataset file exists. If it exists, the next line is read and sent to Kafka. If the next line does not exist, it is read to the end of the file, and the input stage ends. Secondly, in the output stage, the consumer connects to Kafka and consumes the data sent to Kafka through the corresponding topic in the input stage, until the consumer actively closes the connection and the output ends.

Step 4: Install and deploy the workload execution module. The implementation of the workload execution module is to convert the workload into a runnable program by writing Java code. The specific installation and deployment process is as follows:

(1) Install the Java language dependency environment JDK1.8;

(2) Package and upload the Java program to the server;

(3) Start the Java package;

The workload execution module implemented in Java language, the specific algorithm steps are as follows:

Input: load number, database number to be evaluated;

Output: execution time, CPU and memory occupied;

After the message queue obtains the data, the workload execution module starts to run the program according to the load number selected by the user and the database number to be evaluated. First, the system creates a record marked for evaluation, which contains only the load number and database label when it is created. The system executes the workloads in the order in which the records were submitted, and marks the currently executing records as under evaluation. During the running of the program, the running time of each record will be saved, and the resource monitoring module composed of Prometheus and Grafana will be called to obtain the CPU and memory usage. After the execution of the workload program is completed, the workload execution module saves the execution time and resource occupancy in the database, and marks the current record as evaluated.

Step 5: Install and deploy the evaluation record management module. The evaluation record management module is implemented in Java language and Vue front-end framework. The collective installation and deployment process is as follows:

(1) Install the Java language dependency environment JDK1.8;

(2) Install the latest version of nodeJS;

(3) Package the workload execution module and the evaluation record management module respectively and upload them to the server;

(4) Start the front-end and back-end programs.

The evaluation record management module provides users with an easy-to-interact web interface, which is divided into two parts:

(a) The evaluation task submission part, which presents parameters such as load number and database number in the form of options for users to choose. At the same time, there is detailed description information for each task. By selecting the corresponding parameters and clicking the execute button, the workload execution module will create a task to be evaluated and save it in the database;

(b) Evaluation record viewing section, which displays all evaluation records. Users can filter the evaluation records they want to view according to the evaluation status, load type, and database number.

Claims (10)

1. A benchmark evaluation system oriented to a multimodal database, characterized in that it comprises: a distributed database module, a resource monitoring module, a message transmission module, a workload execution module and an evaluation record management module; The message transmission module provides source data for the workload execution module; the workload execution module stores the data read from the message transmission module to the distributed database module; the resource monitoring module collects resource data during workload execution ; The evaluation record management module is responsible for managing the execution results of the workload execution module. 2 . The multimodal database-oriented benchmark evaluation system according to claim 1 , wherein the distributed database module comprises two multimodal databases: ArangoDB and OrientDB. 3 . 3. The benchmark evaluation system oriented to multimodal database according to claim 1, is characterized in that, described resource monitoring module is made up of distributed monitoring system Prometheus and Grafana, is used to evaluate the resources occupied by database, including CPU and Grafana. Monitoring of memory usage. 4. the benchmark evaluation system for multi-mode database according to claim 1, is characterized in that, described message transmission module realizes asynchronous message transmission queue by Zookeeper and Kafka; In this asynchronous message transmission queue, Zookeeper is a registration center, making The workload execution module can asynchronously obtain data from Kafka; the Kafka is a stream data processing platform; before the evaluation starts, the message transmission queue producer stores the data in the data set in Kafka through a Java program to provide data for consumers. 5. The benchmark evaluation system oriented to a multimodal database according to claim 1, wherein the workload execution module simulates the load, including three types of loads: a single-table addition, deletion, modification, and query workload, a multi-threaded workload, and a multi-threaded workload. Table joint query workload; single-table CRUD workloads only perform CRUD operations on one table at a time, thereby simulating data migration and data instantiation operations; multi-threaded workloads perform tasks by creating multiple threads, thereby Simulate high-concurrency operations in which the database processes multiple requests at the same time in the big data application scenario; the multi-table joint query workload simulates operations with complex business logic in the big data scenario through the joint query between tables. 6. The multimodal database-oriented benchmark evaluation system according to claim 1, wherein the evaluation record management module reduces the database evaluation threshold by constructing an easy-to-interact Web page, and enhances the database evaluation interactivity; The evaluation record management module is divided into two parts: the task submission part and the evaluation record viewing part; In the task submission section, users can select relevant parameters and submit evaluation tasks through the web page; In the evaluation record viewing section, users can view the status and parameters of all evaluation tasks. 7. A method for constructing a benchmark evaluation system oriented to a multimodal database, comprising the steps of: (1) Install and deploy master-slave clusters for databases involved in evaluation; (2) Install the distributed resource monitoring software Prometheus and Grafana, which are used to count the resource usage of the server; Prometheus is a time-series database, which realizes the purpose of resource monitoring by regularly sampling the resource status of the server, including monitoring two types of resources: CPU and memory. ; Grafana is a graphical interface tool that can present the data collected by Prometheus to the interface; (3) Install and deploy an asynchronous message transmission queue composed of Zookeeper and Kafka; in this message queue, Zookeeper is used as a registry to achieve message load balancing, and Kafka is used as a stream data processing platform; (4) Convert the load into a runnable program by coding, and obtain the workload execution module, which includes three types of workloads: single-table addition, deletion, modification and query workload, multi-threaded workload, and multi-table joint query workload ; (5) Provide evaluation task submission function and evaluation record viewing function. 8 . The method for constructing a multimodal database-oriented benchmarking system according to claim 7 , wherein in the step (3), Zookeeper and Kafka are used to implement an asynchronous message transmission queue, and the producer of the message transmission queue Before the evaluation starts, the data in the multi-mode data set is written into Kafka through the Java program. After the evaluation starts, the workload execution module reads the data from Kafka as a consumer of the asynchronous message transmission queue. The specific steps are as follows: Input: multimodal dataset; Output: Kafka stream data; First, the input stage determines whether the next line of the multi-mode data set file exists. If it exists, the next line is read and sent to Kafka. If the next line does not exist, it is read to the end of the file, and the input stage ends; secondly, in the output stage , the consumer connects to Kafka and consumes the data sent to Kafka through the corresponding topic input stage, until the consumer actively closes the connection and the output ends. 9 . The method for constructing a multimodal database-oriented benchmark evaluation system according to claim 7 , wherein, in the step (4), the workload execution module implemented by the Java language, the specific algorithm steps are as follows: 10 . Input: load number, database number to be evaluated; Output: execution time, CPU and memory occupied; After the message queue obtains the data, the workload execution module starts to run the program according to the load number selected by the user and the database number to be evaluated; first, the system will create a record marked for evaluation, which only contains the load number and database. Label; the system executes the workload in the order in which the records are submitted, and marks the currently executed records as being evaluated; the running time of each record is saved during the program running, and the resource monitoring module composed of Prometheus and Grafana is called to obtain the CPU and Memory usage; after the program executing the workload finishes running, the workload execution module saves the execution time and resource usage to the database, and marks the current record as evaluated. 10. The method for constructing a multimodal database-oriented benchmark evaluation system according to claim 7, wherein in the step (5), the evaluation record management module provides users with an easy-to-interact Web interface, which is divided into: Two parts: (a) The evaluation task submission part, which presents the load number and database number parameters in the form of options for users to choose; at the same time, there is description information for each task; after the user selects the corresponding parameters, the workload execution module will Create a task to be evaluated and save it in the database; (b) Evaluation record viewing section, which displays all evaluation records; users can filter the evaluation records they want to view according to evaluation status, load type, and database number.

Images