CN119396803A

CN119396803A - Optimization strategy determination method, device, computer equipment, readable storage medium and program product

Info

Publication number: CN119396803A
Application number: CN202411647357.9A
Authority: CN
Inventors: 陈羽飞; 姬永飞
Original assignee: China Telecom Cloud Technology Co Ltd
Current assignee: China Telecom Cloud Technology Co Ltd
Priority date: 2024-11-18
Filing date: 2024-11-18
Publication date: 2025-02-07

Abstract

The present application relates to an optimization strategy determination method, apparatus, computer device, computer readable storage medium and computer program product. The method comprises the steps of reading a performance index set of a target database, wherein the target database is a distributed database, calculating the data inclination condition of the target database according to the performance index set, determining abnormal factors influencing the data inclination under the condition that the target database is inclined, and generating a target optimization strategy according to the abnormal factors. The method can effectively solve the problem of data inclination.

Description

Optimization strategy determination method, apparatus, computer device, readable storage medium, and program product

Technical Field

The present application relates to the field of database technologies, and in particular, to an optimization strategy determination method, an apparatus, a computer device, a computer readable storage medium, and a computer program product.

Background

The distributed database realizes high availability and expandability by storing data on a plurality of nodes, and is widely applied to big data scenes. However, due to uneven distribution of data among nodes, data skew problems are easily caused, namely, overall performance is affected by excessive load of partial nodes. Data tilting often results in excessive consumption of resources for certain nodes, thereby slowing down query efficiency and reducing system stability.

At present, the data inclination problem is usually solved by adopting methods such as distribution key optimization, data redistribution and the like in the prior art. For example, by adjusting the distribution key, data is more evenly distributed across the nodes to reduce the load on certain nodes. But these methods often rely on predefined distribution keys or static partitioning schemes, which are difficult to cope with dynamic changes in data access patterns. In practical application, with the continuous change of data volume and access request, it is difficult for a single distribution key strategy to keep load balance continuously, so that the data tilting problem may appear again, and the data tilting problem cannot be solved effectively.

Disclosure of Invention

In view of the foregoing, it is desirable to provide an optimization strategy determination method, apparatus, computer device, computer-readable storage medium, and computer program product that can effectively solve the data skew problem.

In a first aspect, the present application provides an optimization strategy determination method, the method comprising:

reading a performance index set of a target database, wherein the target database is a distributed database;

calculating the data inclination condition of the target database according to the performance index set;

determining an anomaly factor affecting data skew in the case where the target database is data skewed;

and generating a target optimization strategy according to the abnormal factors.

In one embodiment, the calculating the data inclination of the target database according to the performance index set includes:

The random tree is obtained by randomly selecting any one performance index from the performance index set as a root node and recursively dividing other performance indexes based on the root node;

calculating path lengths from the other performance indexes to the root node;

calculating an abnormal score corresponding to the performance index set according to the path length;

and under the condition that the abnormality score is larger than an abnormality condition threshold value, judging that the performance index has data inclination.

In one embodiment, the determining the anomaly factor that affects the tilt of the data includes:

calculating a correlation coefficient corresponding to each performance index in the performance index set;

and determining the abnormality factor according to the correlation coefficient and a preset abnormality threshold.

In one embodiment, the generating a target optimization strategy according to the anomaly factors includes:

acquiring a history optimization record;

Predicting according to the abnormal factors and the historical optimization to obtain an initial optimization strategy;

Predicting the operation effect of the initial optimization strategy, and optimizing the initial optimization strategy based on the operation effect until the initial optimization strategy reaches an optimization target, so as to obtain the target optimization strategy.

The method according to the above, further comprising:

Monitoring the performance index set;

If the variation degree of the performance index set reaches an optimization standard, training an optimization model according to the performance index set, wherein the optimization model is used for optimizing the initial optimization strategy according to the running effect;

the training process of the optimization model comprises the following steps:

Acquiring a sample index;

inputting the sample index into an initial model for training to obtain a prediction optimization strategy;

and generating a feedback result based on the optimization strategy, and optimizing the initial model according to the feedback result to obtain parameters until the initial model iteration is ended, so as to obtain the optimized model.

In one embodiment, after generating the target optimization strategy, the method further includes:

Performing a test verification on the target optimization strategy in a simulated test environment, and applying the target optimization strategy to the target database after the test verification is passed.

In a second aspect, the present application provides an optimization strategy determining apparatus, the apparatus comprising:

the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for reading a performance index set of a target database;

the inclination calculation module is used for calculating the data inclination condition of the target database according to the performance index set;

A factor determining module for determining an abnormal factor affecting the data skew in the case that the target database is data skewed;

And the strategy generation module is used for generating a target optimization strategy according to the abnormal factors.

In a third aspect, the present application provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method described above.

In a fifth aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.

The optimization strategy determining method, the device, the computer equipment, the computer readable storage medium and the computer program product further determine abnormal factors causing data inclination after determining that the target database is inclined according to the performance index set, and finally generate a corresponding target optimization strategy according to the abnormal factors in a targeted manner. Because, the target optimization strategy is generated based on anomaly factors. Thus, the target optimization strategy can be effectively faced with a dynamic application environment.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are needed in the description of the embodiments of the present application or the related technologies will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other related drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.

FIG. 1 is a diagram of an application environment for an optimization strategy determination method in one embodiment;

FIG. 2 is a flow diagram of a method of optimizing policy determination in one embodiment;

FIG. 3 is a block diagram of an optimization strategy determination device in one embodiment;

Fig. 4 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The optimization strategy determination method provided by the embodiment of the application can be applied to an application system shown in fig. 1. The application system shown in fig. 1 includes a distributed database, a performance monitoring module, and a performance optimization module. The distributed database is responsible for storing and managing mass data, ensuring high-efficiency access and consistency of the data, and the client interacts with the distributed database to initiate a query request and receive a query result.

The performance monitoring module deploys a lightweight performance monitoring agent on each distributed database node. The agent program can collect system-level performance indexes such as CPU utilization rate, memory use condition, disk I/O statistics, network flow and the like in real time as a performance index set. Meanwhile, the monitoring agent program also collects the SQL query statement being executed, records the query text, the execution plan, the query time, the return line number and other indexes as a performance index set, and transmits the collected performance index set to a centralized time sequence database, such as InfluxDB (an open source time sequence database) or Prometaus (a monitoring and alarming tool) through a message queue. The performance monitoring module also collects metadata information of the database, such as table structure definition, index information, partition configuration and the like, and provides basis for subsequent performance analysis.

The performance optimization module is mainly used for reading the collected performance index set from the time sequence database, analyzing according to the performance index set to obtain a target optimization strategy, specifically, the performance optimization module firstly reads the performance index set of the target database, then calculates the data inclination condition of the target database according to the performance index set, determines abnormal factors influencing the data inclination under the condition that the target database is inclined, and finally generates the target optimization strategy according to the abnormal factors.

In an exemplary embodiment, as shown in fig. 2, an optimization strategy determining method is provided, and the performance optimization module in fig. 1 is taken as an example to illustrate the method, which includes the following steps 202 to 206. Wherein:

Step 202, reading a performance index set of a target database, wherein the target database is a distributed database.

Optionally, the performance optimization module may read the collected performance index set from the time series database according to a preset time interval. The time sequence database is used for storing the performance index set acquired by the performance monitoring module from the target database.

Further, the preset time interval can be dynamically adjusted according to actual service requirements. For example, during periods of higher database load, shorter time intervals (e.g., once per minute) may be selected to timely capture potential performance bottlenecks, while during periods of lower load, the acquisition intervals may be appropriately extended (e.g., once per hour) to reduce system load.

Because database performance metrics are typically time-sequential, wave-like, and periodic, storing a set of performance metrics using a time-series database can provide the following advantages to ensure efficient, real-time, and flexible performance data processing, thereby supporting performance optimization based on historical data and real-time data.

Step 204, calculating the data inclination condition of the target database according to the performance index set.

Since the performance index set reflects key data for the operational state of various aspects of the database, such as query response time, load, I/O throughput, CPU utilization, etc., these data can help identify data skew problems.

Illustratively, if a node or partition is significantly more loaded than other nodes or partitions, this typically means that the distribution of data across the portion of the database is not uniform, and some nodes need to handle more requests or larger amounts of data, creating performance bottlenecks. Therefore, by analyzing performance indexes such as CPU utilization, disk I/O and the like, it can be judged which nodes or partitions bear excessive requests or data processing loads. Such unbalanced load distribution is often directly related to data maldistribution (i.e., data skew).

For example, if some queries have significantly longer response times on a certain database node than other nodes, it is likely that there is a skew in the data distribution of that node. In particular, in a distributed database, the "hot spot" area of data may cause certain queries to be particularly concentrated on a certain node or partition, causing increased latency. Thus, by analyzing the query response times and delays of the different nodes, it is possible to identify which nodes present a performance bottleneck. If the response time of some queries is high in duration, it may be because the queries access the sloped portion of the data.

Alternatively, the performance index set may be analyzed by a data distribution analysis method to determine whether a data skew condition exists in the target database. The data distribution method is to find out whether a large amount of data is stored in some partitions or nodes or not by checking the distribution of the data among the nodes, the partitions or the tables, so that the load is unbalanced.

Alternatively, the data tilt condition may be calculated by a pre-trained machine learning model that can quickly identify the data tilt condition of the database. The machine learning model may be an isolated forest, a K-means cluster, principal Component Analysis (PCA), etc.

In step 206, in the case that the target database is tilted, an anomaly factor that affects the tilt of the data is determined.

The phenomenon of data skew is merely a surface manifestation of system performance problems, and factors that cause data skew may be very diverse, such as data maldistribution, query pattern bias, some node loading anomalies, etc. Direct optimization may not fundamentally solve the problem if the specific cause of the data skew is not known, and may even lead to other problems. Thus, it is further determined that the anomaly factors contribute to locating the root cause of the problem, thereby taking more targeted optimization measures.

In this embodiment, an anomaly factor that causes data skew is further determined in the case where it is determined that the target database is in the data skew.

Alternatively, the anomaly factors may be identified by anomaly correlation analysis. For example, correlation analysis is performed on different performance indexes to observe whether correlation exists between the indexes. If the CPU utilization, disk I/O, response time, etc. metrics have synchronous peaks on some nodes, this may be a systematic problem due to data skew. Anomaly correlation analysis can help more fully understand the effects of data skew.

Alternatively, the anomaly may be identified by correlation analysis. Correlation analysis is useful in detecting and analyzing anomalies, particularly in multi-index systems, where correlations between different performance indices can be revealed, helping to locate anomalies that may lead to data skew.

And step 208, generating a target optimization strategy according to the abnormal factors.

After determining the anomaly factors causing the data skew, a corresponding target optimization strategy is generated according to the anomaly factors. The target optimization strategy is an optimization strategy generated according to abnormal factors so as to solve data inclination caused by the abnormal factors.

Alternatively, the target optimization strategy may be generated in combination with the anomaly factors and the historical optimization record. And (3) selecting an optimization scheme for effectively relieving similar anomalies through analyzing the historical optimization records. For example, if a previous query optimization strategy was successful in handling hot spot data tilting, the system may prioritize application of a similar query optimization scheme.

Optionally, specific parameters of the optimization strategy are adjusted according to feedback of the optimization effect in the history. For example, if the effect of a resource expansion strategy is not ideal, and the history may show that the amount of resources increased at that time is insufficient or maldistributed, then the amount of expanded resources may be increased or a more appropriate distribution may be selected when generating the target optimization strategy.

In the above embodiment, after determining that the target database is in the data skew condition according to the performance index set, further determining an anomaly factor that causes the data skew, and finally generating a corresponding target optimization strategy according to the anomaly factor in a targeted manner. Because, the target optimization strategy is generated based on anomaly factors. Thus, the target optimization strategy can be effectively faced with a dynamic application environment.

In one embodiment, the calculating the data inclination condition of the target database according to the performance index set includes constructing a random tree corresponding to the performance index set, wherein the random tree is obtained by randomly selecting any one performance index from the performance index set as a root node and recursively dividing other performance indexes based on the root node, calculating path lengths of the other performance indexes to the root node, calculating an abnormal score corresponding to the performance index set according to the path lengths, and judging that the target database is inclined in the data under the condition that the abnormal score is larger than an abnormal condition threshold value.

In this embodiment, first, according to the random number corresponding to the performance index set firmware, it is determined whether the target database has a data skew problem by means of the random number.

Optionally, constructing the random number corresponding to the performance index set includes randomly selecting any one performance index from the performance index set as a root node, and recursively dividing the performance index based on the root node to obtain the random number.

Illustratively, "CPU usage" is randomly selected as a characteristic. A split point, such as 70, is randomly selected between a minimum (e.g., 50) and a maximum (e.g., 100) of CPU utilization. At this time, the first node generated is the root node, dividing the data into two parts, a subset with CPU utilization less than or equal to 70 and a subset greater than 70. For each subset, features (e.g., disk I/O) and segmentation points are again randomly selected, and recursive segmentation continues until each data point is isolated to a leaf node.

Optionally, for each performance index in the set of performance indices, its path length from the root node to the leaf node is calculated. Where each traversal of a node, the path length is increased by 1, continues recursion until each performance metric is isolated to a leaf node. If during traversal, the data point is isolated to a leaf node, the path computation stops, returning to the current path length.

Alternatively, to make data sets of different sizes comparable, the path length is typically normalized, and the normalized path length is used to calculate the anomaly score, the shorter the path, the more easily the data points are isolated, and the higher the anomaly score.

One point to be noted is that a set of performance indicators corresponds to an anomaly score. Because a set of performance indicators generally represents the overall state of change at a point in time or a location.

Optionally, the anomaly score is compared with an anomaly condition threshold value, wherein the anomaly condition threshold value is preset as a criterion for determining whether the target database has data skew. Wherein the abnormal condition threshold may be derived from historical empirical data.

Illustratively, when the anomaly score is greater than the anomaly condition threshold, it is determined that the target database is at a data tilt. Otherwise, the target database is free of data skew.

In this embodiment, the anomaly scores corresponding to the performance index set may be quickly and accurately calculated by building a random tree.

Further, in one embodiment, determining the anomaly factors affecting the data skew includes calculating correlation coefficients corresponding to the performance indicators in the performance indicator set, and determining the anomaly factors according to the correlation coefficients and a preset anomaly threshold.

Alternatively, the target variable is first selected, and then the correlation coefficient between each performance index and the target variable is calculated. Wherein the correlation coefficient is used for measuring the correlation between the performance index and the target variable. Illustratively, the higher the correlation coefficient, the stronger the relationship between the two variables. After the correlation coefficient is obtained, the correlation coefficient is compared with a preset abnormality threshold, and an abnormality factor is determined. For example, if the correlation coefficient is greater than the preset anomaly threshold, the performance index is determined to be an anomaly factor.

For example, the target variable may be set as the query performance, and then the correlation coefficient between each performance index and the target variable is calculated.

Alternatively, pearson correlation coefficients may be used to quantify the relationship between each performance indicator and the target variable.

Exemplary sets of performance metrics are as follows:

query ID	Query response time	Index usage	CPU utilization	Number of disk operations
					1	200	Miss	70	300
2	50	Hit	40	100
					3	250	Miss	80	400
4	60	Hit	45	110
					5	300	Miss	85	500

In this embodiment, the target variable is the query response time. The pearson correlation coefficients are used to quantify the relationship between the various performance indicators and the corresponding times of the query.

The correlation coefficient calculated by the pseudo design is as follows:

Index (I)	Correlation coefficient with query response time
		Index usage	0.85
CPU utilization	0.65
		Number of disk operations	-0.75

Since the index use case and the CPU use rate are strongly related to the query response time, it is explained that the index use case and the CPU use rate have a significant influence on the query performance. Further, assuming that the abnormal condition threshold is 0.7, the index use case is set as an abnormal factor.

Further, in one embodiment, the generating the target optimization strategy according to the anomaly factors includes obtaining a historical optimization record, predicting according to the anomaly factors and the historical optimization to obtain an initial optimization strategy, predicting an operation effect of the initial optimization strategy, and optimizing the initial optimization strategy based on the operation effect until the initial optimization strategy reaches an optimization target to obtain the target optimization strategy.

Firstly, the system can acquire a history optimization record, and search out a history case similar to the current abnormal factors and a corresponding optimization strategy. These records contain experience in successfully handling similar problems and can provide an important reference for the optimization strategy of the current anomaly factors. Based on the characteristics of the current anomaly factors and the historical optimization records, the system predicts an initial optimization strategy. For example, if the history shows that the "data tilt" problem is mostly solved by "adjust distribution key", then "adjust distribution key" is used as the initial optimization strategy.

Alternatively, the initial optimization strategy may be predicted in combination with the historical optimization record of the previous time period.

Firstly, predicting the operation effect of the initial optimization strategy, and estimating the operation effect of the initial optimization strategy through simulation or based on historical data. If the operational effect is close to the expected optimization goal, the initial optimization strategy is used as the target optimization strategy to be further verified in the test environment, and if the operational effect is not ideal, the strategy is readjusted or other optimization combinations and parameters are explored to find a more effective scheme. In subsequent iterations, the feedback effect of each strategy is continuously evaluated, the behavior of different strategies under different conditions is recorded and accumulated, and which strategies are more effective under the current situation are learned from the performance. In this way, in the subsequent optimization process, the more effective policy combinations will be favored to be selected and attempts to inefficiency or ineffectiveness policies will be reduced. Finally, through multiple rounds of feedback and improvement, gradually converging to a candidate strategy with the best effect, and ensuring that the maximum improvement of the performance is realized on the basis of reaching the optimization target.

The method further comprises the steps of monitoring the performance index set, training an optimization model according to the performance index set if the change degree of the performance index set reaches an optimization standard, optimizing an initial optimization strategy according to an operation effect by the optimization model, obtaining a sample index, inputting the sample index into the initial model for training to obtain a predictive optimization strategy, generating a feedback result based on the optimization strategy, and optimizing the initial model to obtain parameters according to the feedback result until iteration of the initial model is finished to obtain the optimization model.

Optionally, a performance index set is monitored first, and each performance index, such as CPU usage, memory usage, disk I/O, etc., is continuously tracked, so as to learn the running state in real time. If the performance is stable and the index change is within the normal range, the current running state is continuously monitored and maintained. However, if the variation degree of the monitored performance index set reaches the preset optimization standard, the situation that the performance may be bottleneck or unbalanced in load is indicated, and at this time, an optimization flow needs to be triggered to ensure that the performance is recovered to the optimal state. Therefore, an optimization flow is triggered based on the current performance index set, and a training process of the optimization model is started.

In the training process of the optimization model, sample indexes are firstly obtained, data samples related to the current performance condition are collected, and the sample indexes are used as model input to provide data support for the model. Then, the sample indexes are input into an initial model for training, the initial model generates a predictive optimization strategy, namely a preliminary optimization scheme, based on the input data, and then a corresponding feedback result is generated based on the optimization strategy.

In order to enable the optimization model to be continuously learned and improved, parameters of the initial model are optimized and adjusted according to feedback results. The reason for this is that the actual running situation is complex, the initial model cannot completely adapt to all scenes in the initial stage of training, and the initial model needs to be continuously adjusted through feedback to find more suitable parameter and strategy combinations. By repeated feedback and adjustment, the prediction capability of the initial model is gradually improved, and the effect of the optimization strategy is also continuously improved. The whole process is iterated through multiple rounds of feedback, namely, the feedback is recorded after an optimization strategy is applied in each round, and model parameters are adjusted based on the feedback, so that the performance prediction capability of the initial model is continuously optimized.

The iterative process is continued until the training process of the initial model reaches a preset convergence condition or the iterative effect is stable, and the training of the model is finished at the moment to obtain a final optimized model.

In one embodiment, after the generating the target optimization strategy, the method further comprises performing test verification on the target optimization strategy in the simulated test environment, and applying the target optimization strategy to the target database after the test verification is passed.

Optionally, the target optimization strategy is tested in a simulated test environment to verify its performance and stability in the simulated test environment to ensure that it does not negatively impact the actual target database.

Optionally, the test verification process includes evaluating the performance index improvement effect of the target optimization strategy, such as query response time, resource utilization, and the like. If the test and verification pass, the target optimization strategy can be stably operated in the actual environment and reach the expected effect, and then the target optimization strategy is applied to the target database so as to realize the performance optimization and the load balancing of the system. By verifying in the test environment and then applying the test environment to the target database, the potential risk of the optimization strategy to the production environment can be effectively reduced.

It should be noted that the simulated test environment should be consistent with the target database.

Alternatively, the process of building the simulation database may include creating similar data sets in the test environment based on the actual data size and distribution of the production environment. Features of the data, including data volume, distribution, index, table structure, etc., may be simulated by extracting a portion of the data from the production environment, or using data generation tools. If the target database is in the data inclination problem, the situation can be reappeared in the test environment, and whether the optimization strategy can effectively relieve the uneven load problem caused by the data inclination or not can be tested through unbalanced data distribution.

In the above embodiment, after the test passes, the target optimization strategy is applied to the target database, so as to reduce the risk to the actual system.

In other embodiments, the application system further includes a visualization management module, where the visualization management module provides a visual management console that displays a real-time performance data dashboard for each node. The administrator can view the root cause analysis report of the performance problem and understand the key factors that lead to performance degradation. The system can provide optimization suggestions for performance problems, and an administrator can check the prediction effect of the optimization schemes and perform manual intervention and adjustment. An administrator can input knowledge about business characteristics, SLA requirements and the like through a control console, and the system is helped to provide an optimization scheme which is more close to actual requirements. The module records the past optimization behavior and effect and provides visual optimization history tracking for the administrator.

It should be noted that the optimization strategy determination method provided by the application can be applied to a large-scale electronic commerce platform, a financial industry application, an internet of things data platform, a medical health information system and an online education platform.

The large-scale e-commerce platform is usually deployed with a large-scale distributed database and bears massive data of users, orders, commodities and the like. The optimization strategy determination method can automatically find out performance problems such as data inclination, hot spot inquiry and the like, and provides a targeted optimization scheme to ensure that the E-commerce system can stably and efficiently run.

In one embodiment, there is provided an optimization strategy determination method applied to an online education platform, including the steps of:

1. Data acquisition and monitoring

The first step is to deploy an intelligent monitoring component, and deploy the intelligent monitoring component on a database server of the education platform. The component automatically collects various performance indexes including course access delay, user learning record inquiry time, CPU/memory usage, disk I/O and the like, and stores the collected data as a performance index data set in a time sequence database.

And secondly, developing a real-time performance monitoring instrument panel, building a real-time performance monitoring instrument panel, and displaying key performance indexes and trends. The instrument panel displays the delay of various queries, the use condition of system resources and the like, and supports an administrator to check the running state of the database in real time.

And thirdly, recording metadata, wherein the monitoring component records the metadata such as executed SQL sentences, table structures, index information and the like, and provides basic data for subsequent analysis.

2. Data skew analysis

Designing and deploying an analysis engine, developing an analysis engine based on machine learning, and deploying the analysis engine in a data center.

The analysis engine periodically obtains performance index data from the time series database.

And secondly, automatically identifying the data inclination problem, analyzing the duration of the mapping stage and the simplifying stage of the query by using an unsupervised learning algorithm, and performing correlation analysis on other performance indexes. Automatically identifying data skew problems during course access and examination, the system determines and alerts the administrator.

3. Automatic distribution key optimization

Step one, collecting and analyzing SQL execution conditions of a distributed database, and periodically collecting and analyzing executed SQL sentences by a system to record redistribution cost and non-redistribution cost.

Predicting the performance of each candidate distribution key according to SQL execution information, table structures and other characteristics by using a machine learning model, and outputting the distribution key with optimal prediction performance in each table by using the model.

And thirdly, dynamically applying the distribution key, automatically selecting and applying the optimal distribution key to the distributed database by the system, and optimizing the storage of course data and user learning records.

4. Adaptive performance optimization

The system continuously monitors the overall performance index of the database, and automatically triggers an optimization flow when performance deviation is detected.

And step two, executing an optimization flow, and dynamically adjusting measures such as distributing keys, adding indexes, adjusting data partitions and the like. And by utilizing a reinforcement learning technology, the optimal optimization strategy is learned and executed according to the history optimization experience, and the optimization efficiency is gradually improved.

5. Visualization and interactive optimization

Developing a management console, developing a visual management console, and displaying real-time performance monitoring data and optimization suggestions.

Step two, providing a manual intervention interface, wherein an administrator can check specific performance problem description and optimization schemes through a control console, and perform manual intervention and adjustment.

And thirdly, integrating business knowledge, enabling a control console to support an administrator to input business knowledge such as a data access mode, key business indexes and the like, and enabling a system to provide optimization suggestions closer to reality by combining the knowledge.

Furthermore, container editing tools such as Kubernetes or Docker Swart can be deployed to realize the elastic expansion of the application. And automatically expanding or shrinking the application instance according to the real-time traffic and the load.

Furthermore, automatic operation and maintenance tools such as Ansible, chef, puppet can be used, so that the operation and maintenance flow is simplified. And the automatic deployment, configuration management and fault repair are realized, and the workload of an operation and maintenance team is reduced.

Furthermore, tools such as Google analysis and Mixpanel can be integrated, and behavior data of the user including information such as course browsing, learning progress and examination results are collected to construct a personalized recommendation system. And finally, analyzing the user behavior data by utilizing big data analysis and machine learning technology, constructing a personalized recommendation algorithm, and recommending courses and learning resources according to learning habits and interests of the user.

In the embodiment, performance problems such as data inclination, hot spot inquiry and the like can be automatically found, and a targeted optimization scheme is provided. By means of measures such as automatic adjustment of distribution keys and addition of indexes, the performance bottleneck problem is effectively solved, and the query processing speed and the transaction response time are improved. It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides an optimization strategy determining device for realizing the above-mentioned optimization strategy determining method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the one or more optimization strategy determining devices provided below may be referred to the limitation of the optimization strategy determining method hereinabove, and will not be described herein.

In one exemplary embodiment, as shown in FIG. 3, an optimization strategy determination apparatus is provided, comprising a reading module 310, a tilt calculation module 320, a factor determination module 330, and a strategy generation module 340, wherein:

The reading module 310 is configured to read a performance index set of a target database, where the target database is a distributed database.

And the inclination calculation module 320 is configured to calculate a data inclination condition of the target database according to the performance index set.

The factor determining module 330 is configured to determine an anomaly factor that affects the data skew in a case where the target database is data skewed.

The policy generation module 340 is configured to generate a target optimization policy according to the anomaly factors.

In one embodiment, the tilt calculation module 320 includes:

the random tree is obtained by randomly selecting any one performance index from the performance index set as a root node and recursively dividing other performance indexes based on the root node.

And the path calculation unit is used for calculating the path length from the other performance indexes to the root node.

And the abnormal score calculating unit is used for calculating the abnormal score corresponding to the performance index set according to the path length.

And the abnormality judging unit is used for judging that the target database is inclined in the data under the condition that the abnormality score is larger than the abnormality condition threshold value.

In one embodiment, the factor determination module 330 includes:

and the correlation calculation unit is used for calculating correlation coefficients corresponding to the performance indexes in the performance index set.

The selecting unit is used for determining an abnormal factor according to the correlation coefficient and a preset abnormal threshold value.

In one embodiment, the policy generation module 340 includes:

And the record acquisition unit is used for acquiring the history optimization record.

The prediction unit is used for predicting according to the abnormal factors and the historical optimization to obtain an initial optimization strategy.

The optimizing unit is used for predicting the operation effect of the initial optimizing strategy and optimizing the initial optimizing strategy based on the operation effect until the initial optimizing strategy reaches the optimizing target, so as to obtain the target optimizing strategy.

In one embodiment, the policy generation module 340 further includes:

the detection unit is used for monitoring the performance index set;

The training unit is used for training an optimization model according to the performance index set if the variation degree of the performance index set reaches the optimization standard, and the optimization model is used for optimizing an initial optimization strategy according to the operation effect.

In one embodiment, the training unit includes:

And the sample acquisition subunit is used for acquiring the sample index.

And the model training subunit is used for inputting the sample indexes into the initial model for training to obtain a prediction optimization strategy.

And the model optimization subunit is used for generating a feedback result based on the prediction optimization strategy, optimizing the initial model according to the feedback result to obtain parameters until the iteration of the initial model is finished, and obtaining an optimized model.

In one embodiment, the apparatus further comprises:

and the test module is used for carrying out test verification on the target optimization strategy in the simulation test environment and applying the target optimization strategy to the target database after the test verification is passed.

The respective modules in the above-described optimization strategy determination device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one exemplary embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 4. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing performance index data. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an optimization strategy determination method.

It will be appreciated by persons skilled in the art that the architecture shown in fig. 4 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method of any of the embodiments described above when the computer program is executed.

In an embodiment, a computer readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method of any of the embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method of any of the embodiments described above.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile memory and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (RESISTIVE RANDOM ACCESS MEMORY, reRAM), magneto-resistive Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computation, an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) processor, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the present application.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of optimizing policy determination, the method comprising:

2. The method of claim 1, wherein said calculating a data tilt of said target database from said set of performance metrics comprises:

calculating path lengths from the other performance indexes to the root node;

and under the condition that the abnormality score is larger than an abnormality condition threshold value, judging that the target database is inclined in data.

3. The method of claim 1, wherein determining anomaly factors that affect data skew comprises:

4. The method of claim 1, wherein generating a target optimization strategy based on the anomaly factors comprises:

acquiring a history optimization record;

5. The method according to claim 4, wherein the method further comprises:

Monitoring the performance index set;

the training process of the optimization model comprises the following steps:

Acquiring a sample index;

and generating a feedback result based on the prediction optimization strategy, and optimizing the initial model according to the feedback result to obtain parameters until the initial model iteration is ended, so as to obtain the optimized model.

6. The method of claim 1, wherein after generating the target optimization strategy, further comprising:

7. An optimization strategy determination device, characterized in that the device comprises:

the system comprises a reading module, a processing module and a processing module, wherein the reading module is used for reading a performance index set of a target database, and the target database is a distributed database;

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.