CN116680295A - Method, system and device for multi-database processing data - Google Patents

Abstract

The application discloses a method, a system and a device for processing data by a multi-database. Wherein the method comprises the following steps: receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one. The application solves the technical problems of low data query efficiency and low query response speed caused by poor throughput and low expandability of the traditional relational database in the related technology.

Description

Method, system and device for multi-database processing data

technical field

The present application relates to the field of data processing, in particular, to a method, system and device for processing data with multiple databases.

Background technique

In the telecom operator network, the performance data generated by the operation of network equipment usually has the characteristics of large data volume, high concurrency, and strong real-time performance. The data growth rate is extremely fast. When such data is stored and processed, the throughput and There are high requirements for scalability, which are difficult for traditional relational databases to meet. However, with the expansion of the scope of device management, the pressure of reading and writing of the database is increasing day by day, and the storage method that relies solely on OpenTSDB has gradually reached the performance bottleneck. The main problems are as follows:

1. The scale of existing network equipment is huge, there are many logical resources such as ports, sub-interfaces, and management domains, and the amount of data generated every day exceeds billions. At the same time, taking into account highly concurrent data writing and querying, the processing speed of the server CPU reaches the performance bottleneck. QPS can only reach dozens, and the query response speed becomes slower;

2. In OpenTSDB, the device IP and port names are stored in the metadata as tagv, but since the relationship between the device and the port cannot be stored in the tagv, when querying the specified port of the specified device, the program will search from all device IPs and ports The Cartesian product of the name matches the appropriate data one by one, and the query range increases exponentially, which greatly affects the data query efficiency;

3. In the usage scenario of device performance data, users generally pay more attention to the recently generated real-time data, which is used to monitor the current operation of the device, and have higher requirements for query response speed of hot data. For historical data stored for a long time, the query frequency of users is generally very low, and historical data is mainly used for visualization or predictive analysis, and the requirements for query response speed are not high. However, the current OpenTSDB does not have a hierarchical storage mechanism for hot and cold data. Hot and cold data share the same query storage mechanism, which reduces the query performance of hot data and also causes unnecessary storage costs and waste of resources for query storage of cold data. .

For the above problems, no effective solution has been proposed yet.

Contents of the invention

The embodiment of the present application provides a method, system and device for multi-database processing data, to at least solve the problem of low data query efficiency due to the poor throughput and low scalability of traditional relational databases in the related art , technical issues with slow query response times.

According to an aspect of an embodiment of the present application, a method for processing data in multiple databases is provided, including: receiving a data query request of a target object, wherein the data query request carries at least identification information of the data to be queried; and determining the data to be queried according to the identification information The target data type to which the data belongs, where the data type includes one of the following: hot data, cold data; determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format , where the target data type is in one-to-one correspondence with the data query interface.

Optionally, determining the target data type to which the data to be queried belongs according to the identification information includes: determining the query frequency or time range of the data to be queried corresponding to the identification information in a historical period, wherein the time range is used to indicate the last time the data to be queried is received The interval between the historical data query request and the data query request received this time; when the query frequency is greater than the predetermined frequency, or the time range is less than the preset time period, it is determined that the data to be queried is hot data; when the query frequency is less than If the predetermined frequency or the time range is greater than the preset time period, it is determined that the data to be queried is cold data.

Optionally, determine the data query interface corresponding to the target data type, and return the data queried by the data query interface to the target object, including: calling the data query interface of the OpenTSDB database when the target data type is cold data; The index passed in by the object and the selected time range are used to obtain all sub-table information of the index within the time range from the OpenTSDB database. Assign the device IP corresponding to the target object in the end parameter; judge whether there is a function calculation requirement in the parameters input from the target object, perform logical operations according to the calculation requirement, and return the calculation result to the target object.

Optionally, before obtaining all sub-table information of the indicator within the time range, the method also includes: assigning the IP of the device to query and assign values using the regular matching of the filter, and instead assigning values in the start parameter and end parameter, where the filter only Used to match port names.

Optionally, determine the data query interface corresponding to the target data type, and return the data queried by the data query interface to the target object, including: calling the data query interface of the Redis database when the target data is hot data; Whether there are function calculation requirements in the parameters input by the object, perform logical operations according to the calculation requirements, and return the calculation results to the terminal corresponding to the target object.

Optionally, before invoking the data query interface of the Redis database, the method also includes: determining the storage period of the Redis database; obtaining historical data written in the Redis database in historical periods, and deleting other data in the Redis database except for the storage period.

Optionally, after deleting other data in the Redis database except the storage period, the method further includes: determining the data requested by each object within the historical period, and analyzing the requested data within the historical period based on the page replacement algorithm , filter out new hot data and write it to the Redis database.

According to another aspect of the embodiment of the present application, there is also provided a multi-database data processing system, including: a data writing module, provided with a Redis database corresponding to storing hot data, and an OpenTSDB database used to store cold data; A data query module, configured to receive a data query request from a target object, wherein the data query request carries at least identification information of the data to be queried, and determine the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes one of the following : hot data, or cold data; determine the data query interface corresponding to the target data type, and return the structured data queried by the data query interface to the target object in a unified data format, where the target data type corresponds to the data query interface one by one .

Optionally, the scan function in the OpenTSDB database assigns the device IP corresponding to the target object in the start and end parameters.

According to another aspect of the embodiment of the present application, there is also provided an apparatus for processing data with multiple databases, including: a receiving module, configured to receive a data query request of a target object, wherein the data query request carries at least identification information of the data to be queried ; The query module is used to determine the target data type of the data to be queried according to the identification information, wherein the data type includes one of the following: hot data, cold data; the determination module is used to determine the data query interface corresponding to the target data type. The result data queried by the data query interface is returned to the target object in a unified data format, wherein the target data type is in one-to-one correspondence with the data query interface.

According to another aspect of the embodiment of the present application, there is also provided a non-volatile storage medium, the storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to execute any method for processing data with multiple databases .

According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a processor; wherein, the processor is configured to execute instructions to implement any method for processing data with multiple databases.

In this embodiment of the application, the hot and cold data are stored separately, and by receiving the data query request of the target object, wherein the data query request carries at least the identification information of the data to be queried; according to the identification information, the target to which the data to be queried belongs is determined Data type, where the data type includes one of the following: hot data, cold data; determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, where the target One-to-one correspondence between data types and data query interfaces achieves the purpose of classifying and storing hot and cold data based on different types of databases, thereby achieving the technical effect of improving the data processing throughput of the database, improving data query efficiency and response speed, and further It solves the technical problems of low data query efficiency and slow query response speed due to poor throughput and low scalability of traditional relational databases in related technologies.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

1 is a schematic flow diagram of a method for processing data in multiple databases according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a system for processing data with multiple databases according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a system for processing data with multiple databases in an embodiment of the present application;

4 is a schematic flow diagram of a method for processing data in multiple databases in an embodiment of the present application;

Fig. 5 is a schematic diagram of a flow diagram of real-time diversity storage of equipment operating data in an embodiment of the present application;

Fig. 6 is a real-time query flowchart of equipment operation data in the embodiment of the present application;

FIG. 7 is a schematic structural diagram of an apparatus for processing data with multiple databases according to an embodiment of the present application;

Fig. 8 is a structural block diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

In order for those skilled in the art to better understand the relevant embodiments of the present application, the technical terms or nouns that may be involved in the embodiments of the present application are now explained:

1.QPS: Queries Per Second means "query rate per second", which is the number of queries that a server can perform per second, and is a measure of how much traffic a specific query server handles within a specified time.

2. LRU is the abbreviation of Least Recently Used, that is, the least recently used. It is a commonly used page replacement algorithm, and the page that has not been used for the longest time is selected to be eliminated. This algorithm gives each page an access field, which is used to record the time t that has elapsed since a page was last accessed. When a page needs to be eliminated, choose the one with the largest t value among the existing pages, that is, the least recently used The page is eliminated.

According to an embodiment of the present application, an embodiment of a method for processing data with multiple databases is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and , although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

Fig. 1 is the method for processing data according to the multi-database embodiment of the present application, as shown in Fig. 1, this method comprises the following steps:

S102. Receive a data query request from the target object, where the data query request at least carries identification information of the data to be queried;

S104. Determine the target data type of the data to be queried according to the identification information, wherein the data type includes one of the following: hot data, cold data;

S106. Determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.

In the multi-database data processing method, the cold and hot data are stored separately, and by receiving the data query request of the target object, wherein the data query request at least carries the identification information of the data to be queried; The target data type, wherein the data type includes one of the following: hot data, cold data; determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, where , the target data type corresponds to the data query interface one by one, and achieves the purpose of classifying and storing hot and cold data based on different types of databases, thereby achieving the technical effect of improving the data processing throughput of the database, improving the efficiency of data query and the response speed , and further solve the technical problems of low data query efficiency and slow query response speed due to poor throughput and low scalability of traditional relational databases in related technologies.

In some embodiments of the present application, determining the target data type of the data to be queried according to the identification information can be achieved in the following manner: the query frequency or time range of the data to be queried corresponding to the identification information in the historical period can be determined, wherein the time range is used Indicates the interval between the historical data query request that received the data to be queried last time and the data query request received this time; when the query frequency is greater than the predetermined frequency, or the time range is less than the preset duration, determine the data to be queried It is hot data; when the query frequency is less than the predetermined frequency, or the time range is greater than the preset duration, it is determined that the data to be queried is cold data.

In order to improve the processing efficiency of cold data, as an optional implementation, determine the data query interface corresponding to the target data type, and return the data queried by the data query interface to the target object, including: when the target data type is cold data In the case of , call the data query interface of the OpenTSDB database; according to the indicators passed in by the target object and the selected time range, all sub-table information of the indicator within the time range is obtained from the OpenTSDB database. Among them, in the OpenTSDB database The scan function is used to assign the device IP corresponding to the target object to the start time start parameter and the end time end parameter; judge whether there is a function calculation requirement in the parameters input from the target object, perform logical operations according to the calculation requirement, and return the calculation result to the target audience.

In some embodiments of the present application, in order to avoid the Cartesian product problem, before obtaining all sub-table information of the indicator within the time range, the device IP can be queried and assigned using regular matching of the filter, and changed to the start parameter and end parameter, where filter is only used to match port names.

In order to improve the query efficiency of hot data, determine the data query interface corresponding to the target data type, and return the data queried by the data query interface to the target object, including: when the target data is hot data, call the data query of the Redis database Interface; determine whether there is a function calculation requirement in the parameters input from the target object, perform logical operations according to the calculation requirement, and return the calculation result to the terminal corresponding to the target object.

In order to reduce the occupation of the Redis database, before calling the data query interface of the Redis database, the storage period of the Redis database can also be determined; obtain the historical data written to the Redis database in the historical period, and delete other data in the Redis database except for the storage period .

Optionally, after deleting other data in the Redis database except the storage period, the method further includes: determining the data requested by each object within the historical period, and analyzing the requested data within the historical period based on the page replacement algorithm , filter out new hot data and write it to the Redis database.

Fig. 2 is a kind of multi-database processing data system according to the embodiment of the present application, as shown in Fig. 2, the system includes:

Data writing module 20 is provided with a Redis database for storing hot data, and an OpenTSDB database for storing cold data;

The data query module 22 is configured to receive a data query request from a target object, wherein the data query request at least carries identification information of the data to be queried, and determines the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes the following One: hot data or cold data; determine the data query interface corresponding to the target data type, and return the structural data queried by the data query interface to the target object in a unified data format, where the target data type and the data query interface are one by one correspond.

In a multi-database system for processing data, the data writing module 20 is provided with a corresponding Redis database for storing hot data and an OpenTSDB database for storing cold data; the data query module 22 is used to receive the data query request of the target object , wherein the data query request at least carries the identification information of the data to be queried, and determines the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes one of the following: hot data or cold data; The data query interface of the data query interface returns the structural data queried by the data query interface to the target object in a unified data format. Among them, the target data type corresponds to the data query interface one by one, which achieves hot and cold data based on different types of databases. The purpose of classified storage, thereby achieving the technical effect of improving the data processing throughput of the database, improving the efficiency of data query and response speed, and then solving the problem of poor throughput and low scalability of traditional relational databases in related technologies. , resulting in low data query efficiency and slow query response speed.

Optionally, the scan function in the OpenTSDB database assigns the device IP corresponding to the target object in the start and end parameters.

Fig. 3 is in an embodiment of the present application, the system of multi-database processing data, as shown in Fig. 3, this system comprises:

Data writing module: After the data is parsed into the database, it is simultaneously written to OpenTSDB and Redis for double-write storage;

OpenTSDB transformation module: According to the query/write index and time, obtain different sub-tables, if the sub-table does not exist, obtain the global lock, create the corresponding sub-table, optimize the Rowkey index at the bottom of HBASE, transform the scan function, and set the device IP From the regular matching query of the filter, to assigning values at start and end, the filter only matches the port name, thereby optimizing the Cartesian product problem; the api/query interface that comes with OpenTSDB is also modified according to the above rules;

Redis Transformation Module: Transform the Redis data structure based on the zset data set, eliminate the structural difference between OpenTSDB and Redis data query output; formulate a data elimination strategy, delete the historical data in Redis according to the data writing time, and use the LRU algorithm to delete the few query times Data: According to the characteristics of the data returned by OpenTSDB, define the Redis data query interface, including performance data query interface, multi-index item query interface, multi-time period query interface, and support sum and avg function calculations;

Data query module: encapsulates the query interface, judges whether the query belongs to cold data or hot data according to the time range and query frequency of user query data, obtains data from OpenTSDB or Redis respectively, and provides real-time data query services.

Fig. 4 is a schematic flow chart of a method for processing data by multiple databases in an embodiment of the present application. As shown in Fig. 4, the process includes:

S1, double-write the collected and analyzed data, and write it into OpenTSDB and Redis at the same time.

S2.1, enter the table according to the write index and write time, if the corresponding table does not exist, acquire the global lock to create the sub-table.

S2.2, optimize the Rowkey index at the bottom of HBASE, modify the scan function, change the device IP from the regular matching query of the filter to the assignment of start and end, and the filter only matches the port name, thereby avoiding the occurrence of the Cartesian product problem.

S2.2, the api/query interface that comes with OpenTSDB, is modified according to the rules in S2.1.

S3.1, transform the Redis data structure based on the zset data set, and eliminate the structural difference between OpenTSDB and Redis data query output.

S3.2, formulate a data elimination strategy, delete the historical data in Redis according to the data writing time, and use the LRU algorithm to delete the data with a small number of queries.

S3.3, according to the characteristics of the data returned by OpenTSDB, define the Redis data query interface, including performance data query interface, multi-index item query interface, multi-time period query interface, and support sum and avg function calculations.

S4, uniformly encapsulates the query interfaces of OpenTSDB and Redis.

S5. When receiving the user's data query request, it is judged whether it is hot data or cold data according to the time of querying data and the query frequency of the content being searched.

S5.1, if it is hot data, call the query interface of Redis.

S5.2, if it is cold data, call the query interface of OpenTSDB.

S5.3, the query may involve multiple sub-tables, then multiple tables are queried concurrently, and after all the returned data is obtained, aggregation or other processing logic is processed

S6, returning the query result to the user in a unified data format.

Now in conjunction with specific embodiment, the above-mentioned technical scheme of the present application is described further:

Figure 5 is a schematic diagram of the real-time diversity storage process of equipment operating data in the embodiment of the present application, as shown in Figure 5: after obtaining the equipment operating data from the telecom operator network, use the above-mentioned system to perform double-writing operations and automatically complete hierarchical storage. The specific process is as follows :

1. After the collection and analysis of equipment operation data is completed, the system writes the data to OpenTSDB and Redis simultaneously and in real time to perform data double-writing operations.

2. Among them, OpenTSDB will continue to write into tables, store the data with the longest storage period, and provide cold data storage and query services.

3. Redis will periodically delete historical data according to the configured storage period, and only store recent hot data.

4. After deleting the historical data in Redis, use the LRU algorithm to add the data with more user queries to Redis, and finally realize the storage of hot data and the elimination of cold data, and complete the hierarchical storage of hot and cold data.

Fig. 6 is a real-time query flowchart of equipment operation data in the embodiment of the present application, as shown in Fig. 6, the process includes:

After the application layer obtains the data call permission from the system, it can obtain the device operation and perform real-time query of the device operation data. After receiving the user's data call request, the specific program response process is as follows:

1. According to the time range and query content of the query request, it is judged whether the data to be queried is cold data or hot data.

2.1. If the time range is greater than the set hot data storage period, this query is cold data, and the query interface of OpenTSDB will be called.

2.2. Obtain all sub-table information according to the indicators and time range passed in by the user, and initiate concurrent queries

2.3. Call the modified scan function to obtain qualified data from HBASE.

2.4. Determine whether there is a function calculation requirement in the input parameter, and sum, average or return the result directly.

3.1. If the time range is less than the set hot data storage period, or the query content belongs to recent common query data, then this query is hot data, and the query interface of Redis will be called.

3.2. Obtain qualified data from Redis.

3.3. Determine whether there is a function calculation requirement in the input parameters, and sum, average or return the result directly.

4. Return the query results to the user in a unified data format.

Fig. 7 is a device for processing data with multiple databases according to an embodiment of the present application. As shown in Fig. 7, the device includes:

The receiving module 70 is configured to receive a data query request of a target object, wherein the data query request carries at least identification information of the data to be queried;

The query module 72 is configured to determine the target data type of the data to be queried according to the identification information, wherein the data type includes one of the following: hot data, cold data;

The determination module 74 is configured to determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.

In this device, the receiving module 70 is used to receive the data query request of the target object, wherein the data query request carries at least the identification information of the data to be queried; the query module 72 is used to determine the target data type to which the data to be queried belongs according to the identification information , wherein, the data type includes one of the following: hot data, cold data; the determination module 74 is used to determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format , where the target data type corresponds to the data query interface one by one, achieving the purpose of classifying and storing hot and cold data based on different types of databases, thereby achieving the goal of improving the data processing throughput of the database, improving the efficiency of data query and the response speed The technical effect further solves the technical problems of low data query efficiency and slow query response speed due to poor throughput and low scalability of traditional relational databases in related technologies.

According to another aspect of the embodiment of the present application, there is also provided a non-volatile storage medium, the storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to execute any method for processing data with multiple databases .

Specifically, the above-mentioned storage medium is used to store program instructions for the following functions, so as to realize the following functions:

Receiving a data query request from a target object, wherein the data query request carries at least identification information of the data to be queried; determining the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes one of the following: hot data, cold data; Determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.

Optionally, in this embodiment, the aforementioned storage medium may include, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of the aforementioned storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), optical fiber, compact disc read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

In an exemplary embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, any one of the methods for processing data with multiple databases described above is implemented.

Optionally, when the computer program is executed by the processor, the following steps can be realized:

Receiving a data query request from a target object, wherein the data query request carries at least identification information of the data to be queried; determining the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes one of the following: hot data, cold data; Determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.

According to an embodiment of the present application, an electronic device is provided, and the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions Executed by at least one processor, so that the at least one processor can execute any one of the methods for processing data in multiple databases described above.

Optionally, the electronic device may further include a transmission device and an input and output device, wherein the transmission device is connected to the processor, and the input device and output device are connected to the processor.

FIG. 8 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present application. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the applications described and/or claimed herein.

As shown in FIG. 8, the device 800 includes a computing unit 801 that can execute according to a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. Various appropriate actions and treatments. In the RAM 803, various programs and data necessary for the operation of the device 800 can also be stored. The computing unit 801 , ROM 802 , and RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to the bus 804 .

Multiple components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc. ; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the device 800 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The computing unit 801 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 801 executes various methods and processes described above, such as a method for processing data by multiple databases. For example, in some embodiments, the method for multi-database processing data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 800 via the ROM 802 and/or the communication unit 809 . When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the above-described method for processing data with multiple databases can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other appropriate way (for example, by means of firmware) to execute the method for processing data with multiple databases.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

The serial numbers of the above embodiments of the present application are for description only, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present application, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be realized in other ways. Wherein, the device embodiments described above are only illustrative. For example, the division of the units may be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrate into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of units or modules may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for enabling a computer device (which may be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes. .

The above description is only the preferred embodiment of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, some improvements and modifications can also be made. These improvements and modifications are also It should be regarded as the protection scope of this application.

Claims (12)

1. A method for multi-database processing data, comprising: receiving a data query request from a target object, wherein the data query request at least carries identification information of the data to be queried; Determine the target data type to which the data to be queried belongs according to the identification information, where the data type includes one of the following: hot data, cold data; Determine the data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type and the data query interface are one by one correspond. 2. The method according to claim 1, wherein determining the target data type to which the data to be queried belongs according to the identification information comprises: Determine the query frequency or time range of the data to be queried corresponding to the identification information in a historical period, wherein the time range is used to indicate the historical data query request received last time for the data to be queried and the data received this time The interval between query requests; When the query frequency is greater than a predetermined frequency, or the time range is less than a preset duration, determine that the data to be queried is the hot data; If the query frequency is less than the predetermined frequency, or the time range is greater than the preset duration, it is determined that the data to be queried is the cold data. 3. The method according to claim 1, wherein determining the data query interface corresponding to the target data type, and returning the data queried by the data query interface to the target object comprises: In the case where the target data type is the cold data, call the data query interface of the OpenTSDB database; According to the index passed in by the target object and the selected time range, all sub-table information of the index within the time range is obtained from the OpenTSDB database, wherein the scan function in the OpenTSDB database uses To assign the device IP corresponding to the target object to the start time start parameter and the end time end parameter; Judging whether there is a function calculation requirement in the parameters input from the target object, performing logic operations according to the calculation requirement, and returning the calculation result to the target object. 4. The method according to claim 3, characterized in that, before obtaining all sub-table information of the indicator within the time range, the method further comprises: The IP of the device is queried and assigned using the regular matching of the filter, and assigned in the start parameter and the end parameter, where the filter is only used to match the port name. 5. The method according to claim 1, wherein determining the data query interface corresponding to the target data type, and returning the data queried by the data query interface to the target object comprises: In the case where the target data is the hot data, call the data query interface of the Redis database; Judging whether there is a function calculation requirement in the parameters input from the target object, performing logical operations according to the calculation requirement, and returning the calculation result to the terminal corresponding to the target object. 6. the method according to claim 5, is characterized in that, before calling the data query interface of Redis database, described method also comprises: Determine the storage period of the Redis database; Obtain the historical data written in the Redis database in the historical period, and delete other data in the Redis database except the storage period. 7. The method according to claim 6, wherein, after deleting other data except the storage period in the Redis database, the method also includes: Determine the data requested by each object within the historical period, analyze the requested data within the historical period based on a page replacement algorithm, and filter out new hot data and write it into the Redis database. 8. A system for processing data in multiple databases, comprising: The data writing module is provided with a Redis database for storing hot data and an OpenTSDB database for storing cold data; A data query module, configured to receive a data query request from a target object, wherein the data query request carries at least identification information of the data to be queried, and determine the target data type to which the data to be queried belongs according to the identification information, wherein, The data type includes one of the following: the hot data or the cold data; determine the data query interface corresponding to the target data type, and return the structural data queried by the data query interface to the A target object, wherein the target data type is in one-to-one correspondence with the data query interface. 9. The system according to claim 8, wherein the scan function in the OpenTSDB database assigns the device IP corresponding to the target object in the start and end parameters. 10. A device for multi-database processing data, comprising: A receiving module, configured to receive a data query request of a target object, wherein the data query request carries at least identification information of the data to be queried; A query module, configured to determine the target data type to which the data to be queried belongs according to the identification information, wherein the data type includes one of the following: hot data, cold data; A determining module, configured to determine a data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, wherein the target data type is the same as the The data query interface corresponds one by one. 11. A non-volatile storage medium, characterized in that the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the program described in any one of claims 1 to 7. Describe the method of multi-database processing data. 12. An electronic device, characterized in that it comprises: processor; memory for storing said processor-executable instructions; Wherein, the processor is configured to execute the instructions, so as to realize the method for processing data with multiple databases according to any one of claims 1-7.