WO2023206875A1 - Indicator distance-based indicator deduplication method and apparatus - Google Patents

Abstract

Disclosed in the embodiments of the present invention are an indicator distance-based indicator deduplication method and apparatus. Incremental data of indicators within a preset time is selected; pairwise comparison is performed on the indicators having the same time dimension in the incremental data, and the indicator distance of two indicators are calculated; and if the indicator distances in multiple continuous time periods are all smaller than an indicator repetition threshold value, the two indicators are proved to be synonymous, otherwise, the two indicators are non-synonymous. The present invention may help to eliminate repeated indicators for indicator management personnel, help to unify the indicator standard, and help to save the indicator storage space.

Description

Index deduplication method and device based on index distance Technical field

The present application relates to the field of Internet technology, specifically, to an index deduplication method, device, computer equipment and storage medium based on index distance.

Background technique

During the construction process of the indicator system, in order to balance the contradiction between the urgent need for applicable indicators and the slow speed of indicator development, indicator development work is usually carried out in different teams in the data department and business department. Due to the lack of unified indicator development specifications, a large number of duplicate indicators with synonymous names and different names will be generated. On the one hand, repeated indicators can easily lead to inconsistent indicator calibers, making people who use indicators afraid of trusting indicators; on the other hand, repeated indicators can take up a lot of storage space.

In view of the problem that repeated indicators in related technologies can easily lead to inconsistent indicator calibers and occupy a large amount of storage space, no effective solution has yet been proposed.

Contents of the invention

Embodiments of the present invention provide an index deduplication method, device, computer equipment and storage medium based on index distance to solve the problem in related technologies that repeated indicators easily lead to inconsistent index calibers and occupy a large amount of storage space.

In order to achieve the above object, a first aspect of the embodiment of the present invention provides an index deduplication method based on index distance, including:

Select the incremental data of the indicator within the preset time;

Compare indicators with the same time dimension in the incremental data in pairs, and calculate the indicator distance between the two indicators;

If the indicator distance is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous, otherwise they are not synonymous.

Optionally, in a possible implementation of the first aspect, the method further includes:

Generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information that refers to the indicator.

Optionally, a possible implementation of the first aspect includes:

Select a benchmark indicator for indicator merging, where the benchmark indicator refers to the indicator retained when two indicators are merged;

Based on the start and end time of the benchmark indicator, determine whether to complete the indicator data or delete the indicator data;

Replace the application information of non-benchmark indicators with benchmark indicators;

Remove non-benchmark indicators.

Optionally, in a possible implementation of the first aspect, determining whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator includes:

If the start and end time range of the benchmark indicator is greater than the start and end time range of the indicator to be merged, the data of the indicator to be merged will be automatically completed;

If the start and end time range of the benchmark indicator is smaller than the start and end time range of the indicators to be merged, part of the indicator data to be merged will be automatically deleted.

Optionally, in a possible implementation of the first aspect, comparing indicators with the same time dimension in the incremental data, and calculating the indicator distance between the two indicators, including:

Calculate the indicator difference between two indicators, as well as the mean and standard deviation of the indicator difference;

Based on the index difference, as well as the mean and standard deviation of the index difference, the index distance of the two indicators is calculated.

A second aspect of the embodiment of the present invention provides an indicator deduplication device based on indicator distance, including:

Incremental data selection module, used to select incremental data of indicators within a preset time;

An indicator distance calculation module is used to compare two indicators with the same time dimension in the incremental data and calculate the indicator distance between the two indicators;

The threshold judgment module is used to indicate that if the distance between the indicators is smaller than the indicator repetition threshold in multiple consecutive time periods, the two indicators are synonymous; otherwise, they are not synonymous.

Optionally, in a possible implementation of the second aspect, it is characterized in that the device further includes:

The indicator synthesis module is used to generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information of the referenced indicator.

Optionally, in a possible implementation of the second aspect, it is characterized in that the indicator synthesis module includes:

A selection unit is used to select a benchmark indicator for indicator merging, where the benchmark indicator refers to an indicator retained when two indicators are merged;

The judgment unit is used to judge whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator;

Replacement unit, used to replace application information of non-benchmark indicators with benchmark indicators;

Delete unit, used to delete non-benchmark indicators.

A third aspect of the embodiment of the present invention provides a computer device, including a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the above methods are implemented. Steps in Examples.

A fourth aspect of the embodiments of the present invention provides a readable storage medium. A computer program is stored in the readable storage medium. When the computer program is executed by a processor, it is used to implement the first aspect of the present invention and the first aspect thereof. Various possible designs of the steps of the method.

The index deduplication method, device, computer equipment and storage medium based on index distance provided by the present invention select the incremental data of the index within a preset time; compare the indexes with the same time dimension in the incremental data in pairs Yes, calculate the index distance between the two indicators; if the indicator distance is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous, otherwise they are not synonymous. The invention can help indicator managers eliminate duplicate indicators, unify indicator calibers, and save indicator storage space.

Description of the drawings

Figure 1 is a flow chart of the first implementation of the index deduplication method based on index distance;

Figure 2 is a structural diagram of a first implementation of an index deduplication device based on index distance.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments These are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to Describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the invention described herein are capable of being practiced in sequences other than those illustrated or described herein.

It should be understood that in various embodiments of the present invention, the size of the sequence numbers of each process does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be determined by the execution order of the embodiments of the present invention. The implementation process constitutes no limitation.

It should be understood that in the present invention, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or equipment that includes a series of steps or units is not necessarily limited to Those steps or elements that are expressly listed may instead include other steps or elements that are not expressly listed or that are inherent to the process, method, product or apparatus.

It should be understood that in the present invention, "plurality" means two or more. "And/or" is just an association relationship that describes related objects, indicating that three relationships can exist. For example, and/or B can mean: A alone exists, A and B exist simultaneously, and B alone exists. . The character "/" generally indicates that the related objects are in an "or" relationship. "Includes A, B and C" and "includes A, B, C" means that it includes all three of A, B and C, and "includes A, B or C" means that it includes one of A, B and C. "Including A, B and/or C" means including any one, any two or three of A, B and C.

It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponding to B" or "B corresponding to A" means that B is associated with A. According to A B can be determined. Determining B based on A does not mean determining B only based on A, but can also determine B based on A and/or other information. The matching between A and B means that the similarity between A and B is greater than or equal to the preset threshold.

Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determination" or "in response to detection."

The technical solution of the present invention will be described in detail below with specific examples. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

The present invention provides an index deduplication method based on index distance. The flow chart is shown in Figure 1, which includes:

Step S110: Select incremental data of the indicator within a preset time.

In this step, the length of the preset time is related to the time dimension of the indicator. If the time dimension of the indicator is day-level granularity, the newly added indicator data of the previous day will be generated every day. The indicator period is one day, and the preset time is n indicator periods, that is, the last n days; if the time dimension of the indicator is week-level granularity, the newly added indicator data of the previous week will be generated every week, and the indicator period is one week, and the preset time is n indicator periods, that is, the latest n weeks.

Step S120: Compare indicators with the same time dimension in the incremental data, and calculate the indicator distance between the two indicators.

In step S120, the newly added indicator data includes indicators in different time dimensions, including indicator data 1 that is added once a day, indicator data 2 that is added once a week, and indicator data that is added once at a specific time based on artificial settings. Indicator data 3; Therefore, when comparing indicators in pairs, it is necessary to first classify indicator data in the same time dimension. Then any two indicator data belonging to the same time dimension are calculated and compared, and their indicator distance is calculated.

Among them, the indicator distance refers to the distance between which the data difference of the indicator is calculated within n indicator periods. The calculation method of the indicator distance is as follows:

Step 1. Calculate the index difference between the two indicators, as well as the average and standard deviation of the index difference.

Indicator difference: xi=|Indicator Ai-Indicator Bi|, where i=1,2,...,n;

Average of indicator differences:

其中i＝1,2,......,n。 Standard deviation of indicator differences:

Where i=1,2,...,n.

Step 2: Calculate the index distance of the two indicators based on the index difference and the average and standard deviation of the index difference.

其中i＝1,2,......,n。

Where i=1,2,...,n.

Step S130: If the indicator distance is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous; otherwise, they are not synonymous.

In this step,

The time period is determined by the time dimension of the two indicators used to calculate the indicator distance. That is to say: if the time dimension of these two indicators is days, then the indicator distance within consecutive days needs to be less than the repetition threshold, indicating that the two The indicators are synonymous; or if the time dimension of the two indicators is week, then the indicator distance in consecutive weeks needs to be less than the repetition threshold to indicate that the two indicators are synonymous.

The indicator duplication threshold k is a threshold that measures whether two indicators are repeated. When dist(Indicator A, Indicator B)>k is calculated, it means that the two indicators are synonymous; when dist(Indicator A, Indicator B)≤k is calculated, it means that the two indicators are not synonymous.

In one embodiment, the method further includes:

Generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information that refers to the indicator.

In this step, since synonymous indicators are compared in pairs, the indicator names are obviously different, so the problem of synonymous indicators with different names arises. In order to solve this problem, the present invention also proposes to automatically generate indicator merging suggestions based on the dimension information of the indicator, the start and end time of the indicator data, the owner of the indicator, the department to which the indicator belongs, and the application (such as BI report) that refers to the indicator, and This serves as the basis for the indicator administrator to decide whether to merge indicators.

In one embodiment, the method further includes:

Select the benchmark indicator for indicator merging, which refers to the indicator retained when the two indicators are merged; determine whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator; replace the application information of non-benchmark indicators become benchmark indicators; delete non-benchmark indicators.

Determining whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator includes: if the start and end time range of the benchmark indicator is greater than the start and end time range of the indicator to be merged, automatically completing the indicator data to be merged; If the start and end time range is smaller than the start and end time range of the indicators to be merged, part of the indicator data to be merged will be automatically deleted.

In the steps, the indicator merging suggestions are mainly divided into the following three points:

First, you need to select an indicator from the indicators to be merged and use it as the base coordinate. The so-called base coordinate is also the indicator that is retained when the two indicators are merged. For example, indicator A and indicator B have synonyms and different names. When merging two indicators, , retain indicator A and delete indicator B, then indicator A is the benchmark indicator.

Secondly, the indicator data is supplemented or automatically deleted according to the start and end time of the benchmark indicator. That is, if the start and end time range of the benchmark indicator is greater than the start and end time range of the indicator to be merged, the indicator data to be merged will be automatically completed; if the start and end time of the benchmark indicator are If the range is smaller than the start and end time range of the indicators to be merged, some of the indicator data to be merged will be automatically deleted. For example: the start and end time of indicator A is from January to December, and the start and end time of indicator B is from May to December. When selecting indicator B as the base coordinate, specify the start and end time from May to December, then 1-4 will be automatically deleted. Indicator A within the monthly time range; when selecting indicator A as the base coordinate and specifying the start and end time as January to December, indicator B within the January to April time range will be automatically supplemented.

Finally, replace the application of non-benchmark indicators with benchmark indicators; and delete the non-benchmark indicator (you do not need to delete it, you just need to take the indicator offline and not display it to indicator users).

Based on the indicator merging suggestions and the user's choice, it is decided whether to merge duplicate indicators and how to merge duplicate indicators.

In one embodiment, the method further includes: after merging duplicate indicators, generating a difference in storage space after merging indicators - before merging indicators, and viewing the effect of the space saved after merging indicators through a visual interface, specifically as follows :

Space occupied before merging indicators: 3.2TB

Space occupied after merging indicators: 2.4TB

Space savings after metrics merged: 0.8TB (25%)

The index deduplication method based on index distance provided by the present invention selects the incremental data of the index within a preset time; compares the indexes with the same time dimension in the incremental data to calculate the index of the two indexes. distance; if the distance between the indicators is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous, otherwise they are not synonymous. The invention can help indicator managers eliminate duplicate indicators, unify indicator calibers, and save indicator storage space.

Technical effect:

1. Able to fully automate the identification of repeated indicators;

2. Users can adjust the repeated indicator identification method by setting the indicator period and indicator repetition threshold;

3. In the indicator merging proposal, sufficient upstream and downstream information of the indicators is provided as a basis for the indicator administrator to merge indicators;

4. A complete technical solution from identifying duplicate indicators, merging duplicate indicators to merging duplicate indicators to save storage space and visualize them.

Embodiments of the present invention also provide an indicator deduplication device based on indicator distance, as shown in Figure 2, including:

Incremental data selection module, used to select incremental data of indicators within a preset time;

An indicator distance calculation module is used to compare two indicators with the same time dimension in the incremental data and calculate the indicator distance between the two indicators;

The threshold judgment module is used to indicate that if the distance between the indicators is smaller than the indicator repetition threshold in multiple consecutive time periods, the two indicators are synonymous; otherwise, they are not synonymous.

In one embodiment, the device further includes:

The indicator synthesis module is used to generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information of the referenced indicator.

In one embodiment, the indicator synthesis module includes:

The selection unit is used to select the benchmark indicator used for indicator merging. The benchmark indicator refers to the indicator retained when the two indicators are merged;

The judgment unit is used to judge whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator;

Replacement unit, used to replace application information of non-benchmark indicators with benchmark indicators;

Delete unit, used to delete non-benchmark indicators.

The index deduplication device based on index distance provided by the present invention selects the incremental data of the index within a preset time; compares the indexes with the same time dimension in the incremental data to calculate the index of the two indexes. distance; if the distance between the indicators is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous, otherwise they are not synonymous. The invention can help indicator managers eliminate duplicate indicators, unify indicator calibers, and save indicator storage space.

The readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media can be any available media that can be accessed by a general purpose or special purpose computer. For example, a readable storage medium is coupled to a processor such that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and readable storage medium may be located in Application Specific Integrated Circuits (ASICs). Additionally, the ASIC can be located in the user equipment. Of course, the processor and the readable storage medium may also exist as discrete components in the communication device. Readable storage media can be read-only memory (ROM), random-access memory (RAM), CD-ROM, tapes, floppy disks, optical data storage devices, etc.

The present invention also provides a program product. The program product includes execution instructions, and the execution instructions are stored in a readable storage medium. At least one processor of the device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction to cause the device to implement the methods provided by the various embodiments described above.

In the above embodiments of the terminal or server, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, CPU), or other general-purpose processor, digital signal processor (English: Digital Signal Processor, DSP )wait. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present invention can be directly implemented by a hardware processor, or executed by a combination of hardware and software modules in the processor.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; and these modifications or substitutions do not deviate from the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention. scope.

Claims (10)

An indicator deduplication method based on indicator distance, which is characterized by including: Select the incremental data of the indicator within the preset time; Compare indicators with the same time dimension in the incremental data in pairs, and calculate the indicator distance between the two indicators; If the indicator distance is smaller than the indicator repetition threshold in multiple consecutive time periods, it means that the two indicators are synonymous, otherwise they are not synonymous. The indicator deduplication method based on indicator distance according to claim 1, characterized in that the method further includes: Generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information that refers to the indicator. The indicator deduplication method based on indicator distance according to claim 2, characterized in that it includes: Select a benchmark indicator for indicator merging, where the benchmark indicator refers to the indicator retained when two indicators are merged; Based on the start and end time of the benchmark indicator, determine whether to complete the indicator data or delete the indicator data; Replace the application information of non-benchmark indicators with benchmark indicators; Remove non-benchmark indicators. The index deduplication method based on index distance according to claim 3, characterized in that, judging whether to complete the index data or delete the index data according to the start and end time of the benchmark index includes: If the start and end time range of the benchmark indicator is greater than the start and end time range of the indicator to be merged, the data of the indicator to be merged will be automatically completed; If the start and end time range of the benchmark indicator is smaller than the start and end time range of the indicators to be merged, part of the indicator data to be merged will be automatically deleted. The indicator deduplication method based on indicator distance according to claim 1, characterized in that the step of comparing indicators with the same time dimension in the incremental data and calculating the indicator distance of the two indicators includes: Calculate the indicator difference between two indicators, as well as the mean and standard deviation of the indicator difference; Based on the index difference, as well as the mean and standard deviation of the index difference, the index distance of the two indicators is calculated. An indicator deduplication device based on indicator distance, which is characterized by including: Incremental data selection module, used to select incremental data of indicators within a preset time; An indicator distance calculation module is used to compare two indicators with the same time dimension in the incremental data and calculate the indicator distance between the two indicators; The threshold judgment module is used to indicate that if the distance between the indicators is smaller than the indicator repetition threshold in multiple consecutive time periods, the two indicators are synonymous; otherwise, they are not synonymous. The indicator deduplication device based on indicator distance according to claim 6, characterized in that the device further includes: The indicator synthesis module is used to generate indicator merging suggestions based on the indicator's dimension information, start and end time, owner, department, and application information of the referenced indicator. The indicator deduplication device based on indicator distance according to claim 7, characterized in that the indicator synthesis module includes: A selection unit is used to select a benchmark indicator for indicator merging, where the benchmark indicator refers to an indicator retained when two indicators are merged; The judgment unit is used to judge whether to complete the indicator data or delete the indicator data based on the start and end time of the benchmark indicator; Replacement unit, used to replace application information of non-benchmark indicators with benchmark indicators; Delete unit, used to delete non-benchmark indicators. A computer device, including a memory and a processor. The memory stores a computer program that can be run on the processor. It is characterized in that when the processor executes the computer program, it implements any one of claims 1 to 5. steps of the method described. A computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images