[go: up one dir, main page]

WO2018171289A1 - Method and device for storing data to database, middleware device, and server - Google Patents

Method and device for storing data to database, middleware device, and server Download PDF

Info

Publication number
WO2018171289A1
WO2018171289A1 PCT/CN2017/120189 CN2017120189W WO2018171289A1 WO 2018171289 A1 WO2018171289 A1 WO 2018171289A1 CN 2017120189 W CN2017120189 W CN 2017120189W WO 2018171289 A1 WO2018171289 A1 WO 2018171289A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
target database
module
database
consumer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/120189
Other languages
French (fr)
Chinese (zh)
Inventor
胡鹏
葛朋旭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Uc Network Technology Co Ltd
Original Assignee
Guangzhou Uc Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Uc Network Technology Co Ltd filed Critical Guangzhou Uc Network Technology Co Ltd
Publication of WO2018171289A1 publication Critical patent/WO2018171289A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures

Definitions

  • Embodiments of the present invention relate to the field of database technologies, and, more particularly, to a method for storing data of different data sources to a target database, a device for storing data of different data sources to a target database, a middleware device, and server.
  • data is generally collected, analyzed, processed, and the like. Finally, the processed data is provided to the application system for use. In each process, different analysis tools or system platforms may be used. Therefore, data conversion and transmission between different analysis tools or system platforms in accordance with the specified data format is required.
  • each big data service and/or various algorithms of the application system is different, and an algorithm may use data values of different fields in one or more tables of the database, so the application system needs to use Different data values.
  • each data in the database can be stored in the in-memory database according to a specified format and using different keywords (Key), so as to facilitate rapid acquisition of the application system. data.
  • the data When storing data to an in-memory database, the data may come from different data sources, for example, different databases and/or different tables in the database.
  • the producer-consumer model is typically used during data transfer or exchange.
  • producers generate data and put it into data-sharing memory, and consumers take data from the data-sharing memory and process the data.
  • the storage and reading of data can be asynchronous.
  • a method for storing data of different data sources to a target database comprising: setting configuration information, wherein the configuration information comprises: first from each data source a manner of conversion of data as it is stored to the target database; converting the first data to the second data based on the configuration information; and storing the second data to the target database.
  • an apparatus for storing data of different data sources to a target database comprising: a setting module for setting configuration information, wherein the configuration information represents each data a conversion mode of the first data of the source when stored in the target database; a conversion module, configured to convert the first data into the second data based on the configuration information set by the setting module; and a storage module, configured to store the The conversion module converts the obtained second data.
  • a middleware device comprising: means for storing data of a different data source to a target database, or for performing a method according to any of the embodiments, according to any one embodiment The operation in .
  • a server comprising a middleware device according to any one of the embodiments.
  • a server comprising a memory and a processor, wherein the memory is for storing an instruction, the instruction controlling the processor to execute according to any one when the server is running A method of storing data for different data sources to a target database of an embodiment.
  • a computer readable storage medium carrying one or more computer instruction programs, the computer instruction program being executed by one or more processors, the one or A plurality of processors execute a method for storing data of a different data source to a target database in accordance with any one of the embodiments.
  • the embodiment of the present invention provides a method, a device, a middleware device, and a server for storing data to a database.
  • the embodiment of the present invention sets a general data conversion rule according to the general data conversion rule. Converting the first data from each data source into the second data stored in the target database according to the set general data conversion rules, and storing in the target database, so that the data from different data sources can be The common format is quickly stored in the target database, which in turn increases the efficiency of storing data to the target database.
  • FIG. 1 shows a schematic flow chart of a method for storing data of different data sources to a target database in accordance with an embodiment of the present invention
  • FIG. 2 shows a schematic block diagram of a middleware device in accordance with an embodiment of the present invention
  • Figure 3 shows a schematic block diagram of a server in accordance with an embodiment of the present invention
  • FIG. 4 shows a schematic block diagram of a server according to an embodiment of the present invention
  • Figure 5 shows a schematic block diagram of an example of a system to which an embodiment of the invention may be applied
  • Figure 6 shows a schematic block diagram of an example of a producer-consumer mode that can be applied to embodiments of the present invention.
  • the different data sources are different tables in different databases or databases.
  • the database is, for example, a relational database.
  • the target database is an in-memory database, such as a distributed in-memory database. The application system can quickly access data from the in-memory database.
  • step S1100 configuration information is set, wherein the configuration information includes a conversion manner when first data from each data source is stored in a target database.
  • the configuration information includes task information indicating that data is stored in the target database, data source information indicating the data source, conversion rule information indicating a data conversion mode, and destination information indicating a data storage destination.
  • the storage of data from multiple data sources to the target database can be automated. This can improve the efficiency of data transfer.
  • step S1200 the first data is converted into the second data based on the configuration information.
  • first and second in the first data and the second data are only for distinguishing data at different processing stages.
  • first data and the second data may be in the same format or in different formats.
  • step S1300 the second data is stored to the target database.
  • the first data may be stored to a target database using a producer-consumer model.
  • the data producer module converts the first data into second data suitable for the target database and the second data in the data shared memory based on the configuration information.
  • the data consumer module reads the second data from the data shared memory and stores the second data to the target database.
  • duplicate data consumes resources from the database.
  • data from different data sources may have different formats, it is difficult to determine duplicate data.
  • the data producer module also calculates a hash value for each piece of second data.
  • the second data may be stored based on the hash value to avoid duplication of the stored second data.
  • the data duplication is determined based on the second data, and this problem can be avoided.
  • the duplication can be eliminated when storing the second data to the target database. For example, storing the hash value and each piece of second data in a data sharing memory in a corresponding manner, the data consumer module storing the second data to the target database with a hash value as a primary key of each second data .
  • the previous data entry can be overwritten by the subsequent data entry, or the subsequent data entry can be discarded.
  • the second data is stored in the data shared memory with the hash value as the primary key of each second data.
  • the data consumer can read the second data directly from the data shared memory and store the second data to the target database. This reduces subsequent processing.
  • the data consumer can read the second data only from the data shared memory without reading the hash value.
  • the data consumer module reads the second data and the corresponding hash value from the data shared memory and stores the second data and the corresponding hash value to the target database.
  • the target database and/or system application can also utilize the hash value.
  • the hash value can be utilized to calculate the address of the second data in the target database.
  • the hash value may also be utilized to verify the integrity of the second data.
  • a device that stores data of different data sources to a target database comprising various operations in a method for performing data for storing different data sources to a target database in accordance with the described embodiments.
  • the apparatus includes: means for setting configuration information, wherein the configuration information represents a manner of conversion of first data from each data source when stored to a target database; Means for converting data into second data; and means for storing the second data to a target database.
  • the embodiment can be applied to a middleware device.
  • the middleware device can be located in a server.
  • the middleware device can also be applied to a terminal device.
  • Figure 2 shows a schematic block diagram of a middleware device in accordance with one embodiment of the present invention.
  • the middleware device 2000 includes the device 2010 described above for storing data of different data sources to a target database.
  • a middleware device is also provided which is designed to perform the operations in the method described in FIG.
  • FIG. 3 shows a schematic block diagram of a server in accordance with another embodiment of the present invention.
  • the server 3000 may include the middleware device 2000 shown in FIG. 2.
  • the target database may be located inside the server 3000 or may be connected to the server 3000 by wire or wirelessly.
  • the server 3000 includes an in-memory database, wherein the middleware device 2000 uses the in-memory database as a target database and stores data of different data sources to the in-memory database.
  • Figure 4 shows a schematic block diagram of a server in accordance with another embodiment of the present invention.
  • the server 4000 can include a processor 4010, a memory 4020, an interface device 4030, a communication device 4040, a display device 4050, an input device 4060, a speaker 4070, a microphone 4080, and the like.
  • the processor 4010 can be, for example, a central processing unit CPU, a microprocessor MCU, or the like.
  • the memory 4020 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like.
  • the interface device 4030 includes, for example, a USB interface, a headphone jack, and the like.
  • the communication device 4040 can perform wired or wireless communication, for example.
  • the display device 4050 is, for example, a liquid crystal display, a touch display, or the like.
  • Input device 4060 can include, for example, a touch screen, a keyboard, and the like. The user can input/output voice information through the speaker 4070 and the microphone 4080.
  • the memory 4020 is configured to store instructions for controlling the processor 4010 to operate to perform the method of storing data for storing different data sources to a target database as described above with reference to FIG. Each operation in .
  • the present invention may relate only to some of the devices, such as processor 4010 and storage device 4020, and the like.
  • a technician can design instructions in accordance with the disclosed aspects of the present invention. How the instructions control the processor for operation is well known in the art and will not be described in detail herein.
  • Figure 5 shows a schematic block diagram of an example of a system to which embodiments of the invention may be applied.
  • server 5040 is coupled to databases 5020, 5030 via network 5010. Although network 5010 is shown in FIG. 5, those skilled in the art will appreciate that server 5040 can be directly coupled to databases 5020, 5030, or that databases 5020, 5030 can be located within server 5040.
  • the server 5040 is also connected to the memory server 5050.
  • the memory server 5050 can be located internal to the server 5040.
  • Embodiments in accordance with the present invention may be implemented in server 5040.
  • the server can be the server described above.
  • the configuration information may include task information indicating that data is stored in the target database, data source information indicating the data source, conversion rule information indicating a manner of data conversion, and destination information indicating a data storage destination.
  • the table below shows an example of the information.
  • the information can be converted into SQL statements for subsequent use according to different data synchronization/transmission tasks.
  • data synchronization (data storing different data sources to a target database) can be performed by the data synchronization executor based on the information.
  • the data synchronization executor can perform data reading, conversion, and storage to the target database.
  • a data synchronous executor can perform processing in a concurrent manner using multiple threads.
  • data sync executors employ a producer-consumer model. For example, it includes components: data producers, data sharing memory, and data consumers.
  • FIG. 6 shows a schematic block diagram of an example of a producer-consumer mode that can be applied to embodiments of the present invention.
  • the data producer reads the first data, for example, a data A table, a data B table, a data C table, and the like. For example, the data producer reads the corresponding data, tables, fields, etc. according to the data source information. The data producer converts the first data into the second data according to the conversion rule information.
  • the data producer can calculate the hash value of the second data.
  • every 5000 data is a batch, and the data is stored in the data sharing memory.
  • the second data may be stored in the data shared memory with the hash value as a primary key to prevent duplication.
  • the hash value and the second data may also be stored in the data sharing memory in a corresponding manner, and the repetition is eliminated when the second data is stored in the target database.
  • the data consumer reads the second data from the data shared memory.
  • the data consumer can read the second data and the corresponding hash value.
  • the second data is stored in a target database such as a Fooyum in-memory database with the hash value as a primary key to prevent data duplication.
  • the data consumer reads the second data for a batch of 5000 data and includes multiple threads to store the second data to the Fooyum in-memory database for use by the system.
  • the data shared memory can contain the status of the data producer, for example, during data reading, data has been read, data read abnormalities, and the like. When the status is "data has been read” and "read data is abnormal", the data producer (thread) will end. If the data in the data shared memory is consumed, or if the status is "read data abnormal", the data consumer will end.
  • information such as the running status of the data shared memory, the number of successful processing, the number of processing failures, the running time, and the processing duration can be saved to the database.
  • the data synchronization task can be automatically performed periodically according to the configuration information. In this way, different data can be read from different relational databases at regular intervals, and the data can be synchronized to the in-memory database according to the specified conversion rules.
  • the application system can quickly read relevant data from the in-memory database.
  • the application system can use, for example, a big data recommendation algorithm or the like.
  • embodiments of the invention may be applied to Hive data warehousing tools.
  • the embodiment of the present invention further provides a computer readable storage medium carrying one or more computer instruction programs thereon, when the computer instruction program is executed by one or more processors, one or more processors execute to implement one for A method for storing data of different data sources to a target database, comprising: setting configuration information, wherein the configuration information includes: a manner of converting the first data from each data source when storing to the target database; and based on the configuration information, the first The data is converted into the second data; and the second data is stored to the target database.
  • the foregoing program may be stored in a computer readable storage medium, and the program is executed when executed.
  • the foregoing storage medium includes: a mobile storage device, a random access memory (RAM), a read-only memory (ROM), a magnetic disk, or an optical disk.
  • RAM random access memory
  • ROM read-only memory
  • magnetic disk or an optical disk.
  • optical disk A medium that can store program code.
  • the above-described integrated unit of the embodiment of the present invention may be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as a stand-alone product.
  • the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for making A computer device (which may be a personal computer, server, or network device, etc.) performs all or part of the methods described in the various embodiments.
  • the foregoing storage medium includes various media that can store program codes, such as a mobile storage device, a RAM, a ROM, a magnetic disk, or an optical disk.
  • Embodiments of the invention may be apparatus, methods, and/or computer program products.
  • the computer program product can comprise a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement various aspects of the present invention.
  • the computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device.
  • the computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, with instructions stored thereon A raised structure in the hole card or groove, and any suitable combination of the above.
  • a computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.
  • the computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages.
  • the computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider to access the Internet) connection).
  • the customized electronic circuit such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions.
  • Computer readable program instructions are executed to implement various aspects of the present invention.
  • the computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams.
  • the computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, such that the computer readable medium storing the instructions includes An article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.
  • the computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process.
  • instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions.
  • Executable instructions can also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Embodiments of the invention disclose a method and device for storing data to a database, a middleware device, and a server. The method comprises: configuring configuration information, wherein the configuration information comprises a conversion method used to store, in a target database, first data from each data source; converting, on the basis of the configuration information, the first data into second data; and storing the second data to the target database. The embodiments of the invention provide the method and device for storing data to a database, the middleware device, and the server, thereby improving efficiency in storing data to a target database.

Description

向数据库存储数据的方法、设备、中间件设备和服务器Method, device, middleware device, and server for storing data to a database 技术领域Technical field

本发明实施例涉及数据库技术领域,并且更具体来说,涉及一种用于向目标数据库存储不同数据源的数据的方法、用于向目标数据库存储不同数据源的数据的设备、中间件设备和服务器。Embodiments of the present invention relate to the field of database technologies, and, more particularly, to a method for storing data of different data sources to a target database, a device for storing data of different data sources to a target database, a middleware device, and server.

背景技术Background technique

在数据库的应用过程中,通常会从一个数据库向另一个数据库转移数据。In the application process of the database, data is usually transferred from one database to another.

例如,在对大数据进行分析和应用的过程中,一般会对数据进行采集、分析、加工等处理。最后,将加工后的数据提供给应用系统,供使用。在每一个处理中,可能使用不同的分析工具或系统平台。因此,需要在不同的分析工具或系统平台之间按照指定数据格式进行数据转换和传输。For example, in the process of analyzing and applying big data, data is generally collected, analyzed, processed, and the like. Finally, the processed data is provided to the application system for use. In each process, different analysis tools or system platforms may be used. Therefore, data conversion and transmission between different analysis tools or system platforms in accordance with the specified data format is required.

通常,应用系统的各个大数据业务和/或各种算法所使用的数据是不一样的,并且一种算法可能使用数据库的一个或多个表中的不同字段的数据值,因此应用系统需要使用不同的数据值。为了使应用系统能够及时且快速地获取算法所需的数据,可以将数据库中的每一条数据,按照指定格式并使用不同的关键字(Key)存储到内存数据库中,以方便应用系统的快速获取数据。Usually, the data used by each big data service and/or various algorithms of the application system is different, and an algorithm may use data values of different fields in one or more tables of the database, so the application system needs to use Different data values. In order to enable the application system to acquire the data required by the algorithm in a timely and rapid manner, each data in the database can be stored in the in-memory database according to a specified format and using different keywords (Key), so as to facilitate rapid acquisition of the application system. data.

在将数据存储到内存数据库时,所述数据可能来自不同数据源,例如,不同的数据库和/或数据库中的不同表。When storing data to an in-memory database, the data may come from different data sources, for example, different databases and/or different tables in the database.

在数据传输或交换过程中,通常会使用生产者-消费者模式。在生产者-消费者模式中,生产者产生数据并将数据放入到数据共享内存中,消费者从数据共享内存取得数据并对数据进行处理。在生产者-消费者模式,数据的存储和读出可以是异步的。The producer-consumer model is typically used during data transfer or exchange. In the producer-consumer model, producers generate data and put it into data-sharing memory, and consumers take data from the data-sharing memory and process the data. In the producer-consumer mode, the storage and reading of data can be asynchronous.

因此,需要提供一种新的技术方案,针对上述现有技术中的至少一个 技术问题进行改进。Therefore, there is a need to provide a new technical solution for improving at least one of the above-mentioned prior art problems.

发明内容Summary of the invention

本发明实施例的一个目的是提供一种用于向目标数据库存储不同数据源的数据的新技术方案。It is an object of embodiments of the present invention to provide a new technical solution for storing data of different data sources to a target database.

根据本发明实施例的第一方面,提供了一种用于向目标数据库存储不同数据源的数据的方法,包括:设置配置信息,其中,所述配置信息包括:来自每个数据源的第一数据在向目标数据库存储时的转换方式;基于所述配置信息,将第一数据转换成第二数据;以及向目标数据库存储所述第二数据。According to a first aspect of an embodiment of the present invention, there is provided a method for storing data of different data sources to a target database, comprising: setting configuration information, wherein the configuration information comprises: first from each data source a manner of conversion of data as it is stored to the target database; converting the first data to the second data based on the configuration information; and storing the second data to the target database.

根据本发明实施例的第二方面,提供了一种用于向目标数据库存储不同数据源的数据的设备,包括:设置模块,用于设置配置信息,其中,所述配置信息表示来自每个数据源的第一数据在向目标数据库存储时的转换方式;转换模块,用于基于所述设置模块设置的配置信息将第一数据转换成第二数据;存储模块,用于向目标数据库存储所述转换模块转换得到的第二数据。According to a second aspect of the embodiments of the present invention, there is provided an apparatus for storing data of different data sources to a target database, comprising: a setting module for setting configuration information, wherein the configuration information represents each data a conversion mode of the first data of the source when stored in the target database; a conversion module, configured to convert the first data into the second data based on the configuration information set by the setting module; and a storage module, configured to store the The conversion module converts the obtained second data.

根据本发明实施例的第三方面,提供了一种中间件设备,包括根据任何一个实施例的用于向目标数据库存储不同数据源的数据的设备,或者用于执行根据任何一个实施例的方法中的操作。According to a third aspect of embodiments of the present invention, there is provided a middleware device comprising: means for storing data of a different data source to a target database, or for performing a method according to any of the embodiments, according to any one embodiment The operation in .

根据本发明实施例的第四方面,提供了一种服务器,包括根据任何一个实施例的中间件设备。According to a fourth aspect of an embodiment of the present invention, there is provided a server comprising a middleware device according to any one of the embodiments.

根据本发明实施例的第五方面,提供了一种服务器,包括存储器和处理器,其中,所述存储器用于存储指令,所述指令在所述服务器运行时控制所述处理器执行根据任何一个实施例的用于向目标数据库存储不同数据源的数据的方法。According to a fifth aspect of the embodiments of the present invention, there is provided a server comprising a memory and a processor, wherein the memory is for storing an instruction, the instruction controlling the processor to execute according to any one when the server is running A method of storing data for different data sources to a target database of an embodiment.

根据本发明实施例的第六方面,提供了一种计算机可读存储介质,其上承载一个或多个计算机指令程序,所述计算机指令程序被一个或多个处理器执行时,所述一个或多个处理器执行根据任何一个实施例的用于向目标数据库存储不同数据源的数据的方法。According to a sixth aspect of the present invention, there is provided a computer readable storage medium carrying one or more computer instruction programs, the computer instruction program being executed by one or more processors, the one or A plurality of processors execute a method for storing data of a different data source to a target database in accordance with any one of the embodiments.

本发明实施例提供了一种向数据库存储数据的方法、设备、中间件设备和服务器,与现有技术相比,本发明实施例设置通用的数据转换规则,并按照该通用的数据转换规则,将来自每个数据源的第一数据按照已设置的通用的数据转换规则,转换为在目标数据库中存储的第二数据,并在目标数据库中进行存储,从而可以将来自不同数据源的数据按照通用的格式快速存储至目标数据库,进而可以提高向目标数据库存储数据的效率。The embodiment of the present invention provides a method, a device, a middleware device, and a server for storing data to a database. Compared with the prior art, the embodiment of the present invention sets a general data conversion rule according to the general data conversion rule. Converting the first data from each data source into the second data stored in the target database according to the set general data conversion rules, and storing in the target database, so that the data from different data sources can be The common format is quickly stored in the target database, which in turn increases the efficiency of storing data to the target database.

通过以下参照附图对本发明的示例性实施例的详细描述,本发明的其它特征及其优点将会变得清楚。Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

附图说明DRAWINGS

被结合在说明书中并构成说明书的一部分的附图示出了本发明的实施例,并且连同其说明一起用于解释本发明的原理。The accompanying drawings, which are incorporated in FIG

图1示出了根据本发明实施例的用于向目标数据库存储不同数据源的数据的方法的示意性流程图;1 shows a schematic flow chart of a method for storing data of different data sources to a target database in accordance with an embodiment of the present invention;

图2示出了根据本发明实施例的中间件设备的示意性框图;2 shows a schematic block diagram of a middleware device in accordance with an embodiment of the present invention;

图3示出了是根据本发明实施例的服务器的示意性框图;Figure 3 shows a schematic block diagram of a server in accordance with an embodiment of the present invention;

图4示出了根据本发明实施例的服务器的示意性框图;FIG. 4 shows a schematic block diagram of a server according to an embodiment of the present invention; FIG.

图5示出了可以应用本发明实施例的系统的例子的示意性框图;Figure 5 shows a schematic block diagram of an example of a system to which an embodiment of the invention may be applied;

图6示出了可以应用于本发明实施例的生产者-消费者模式的例子的示意性框图。Figure 6 shows a schematic block diagram of an example of a producer-consumer mode that can be applied to embodiments of the present invention.

具体实施方式detailed description

现在将参照附图来详细描述本发明的各种示例性实施例。应注意到:除非另外具体说明,否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the invention unless otherwise specified.

以下对至少一个示例性实施例的描述实际上仅仅是说明性的,决不作为对本发明及其应用或使用的任何限制。The following description of the at least one exemplary embodiment is merely illustrative and is in no way

对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论,但在适当情况下,所述技术、方法和设备应当被视为说明书的一部分。Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

在这里示出和讨论的所有例子中,任何具体值应被解释为仅仅是示例性的,而不是作为限制。因此,示例性实施例的其它例子可以具有不同的值。In all of the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

应注意到:相似的标号和字母在下面的附图中表示类似项,因此,一旦某一项在一个附图中被定义,则在随后的附图中不需要对其进行进一步讨论。It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

下面,参照附图描述根据本发明的各个实施例和例子。Hereinafter, various embodiments and examples according to the present invention will be described with reference to the accompanying drawings.

<方法><method>

图1示出了根据本发明实施例的用于向目标数据库存储不同数据源的数据的方法的示意性流程图。例如,所述不同数据源是不同数据库或数据库中的不同表。所述数据库例如是关系型数据库。例如,所述目标数据库是内存数据库,例如,分布式内存数据库。应用系统可以快速地从内存数据库存取数据。1 shows a schematic flow diagram of a method for storing data of different data sources to a target database in accordance with an embodiment of the present invention. For example, the different data sources are different tables in different databases or databases. The database is, for example, a relational database. For example, the target database is an in-memory database, such as a distributed in-memory database. The application system can quickly access data from the in-memory database.

如图1所示,在步骤S1100,设置配置信息,其中,所述配置信息包括:来自每个数据源的第一数据在向目标数据库存储时的转换方式。As shown in FIG. 1, in step S1100, configuration information is set, wherein the configuration information includes a conversion manner when first data from each data source is stored in a target database.

例如,所述配置信息包括表示向目标数据库存储数据的任务信息、表示数据源的数据源信息、表示数据转换方式的转换规则信息和表示数据存储目标的目的地信息。For example, the configuration information includes task information indicating that data is stored in the target database, data source information indicating the data source, conversion rule information indicating a data conversion mode, and destination information indicating a data storage destination.

通过设置和使用配置信息,可以自动实现来自多个数据源的数据向目标数据库的存储。这可以提高数据转移的效率。By setting and using configuration information, the storage of data from multiple data sources to the target database can be automated. This can improve the efficiency of data transfer.

在步骤S1200,基于所述配置信息,将第一数据转换成第二数据。In step S1200, the first data is converted into the second data based on the configuration information.

在这里,第一数据和第二数据中的“第一”和“第二”仅是为了区分处于不同处理阶段的数据。例如,第一数据和第二数据可以是相同格式的或不同格式的。Here, the "first" and "second" in the first data and the second data are only for distinguishing data at different processing stages. For example, the first data and the second data may be in the same format or in different formats.

在步骤S1300,向目标数据库存储所述第二数据。In step S1300, the second data is stored to the target database.

例如,可以使用生产者-消费者模式向目标数据库存储所述第一数据。在生产者-消费者模式中,数据生产者模块基于所述配置信息将第一数据转换成适合目标数据库的第二数据以及将第二数据存储在数据共享内存中。数据消费者模块从数据共享内存读取第二数据并向目标数据库存储第二数 据。For example, the first data may be stored to a target database using a producer-consumer model. In the producer-consumer mode, the data producer module converts the first data into second data suitable for the target database and the second data in the data shared memory based on the configuration information. The data consumer module reads the second data from the data shared memory and stores the second data to the target database.

由于数据来自不同的数据源,因此,数据的内容可能存在重复。当将重复的数据存储到目标数据库中时,重复的数据会占用数据库的资源。但是,由于不同数据源的数据可能具有不同的格式,因此,很难确定重复的数据。Since the data comes from different data sources, the content of the data may be duplicated. When storing duplicate data in the target database, duplicate data consumes resources from the database. However, because data from different data sources may have different formats, it is difficult to determine duplicate data.

此外,当应用系统从目标数据读取重复的数据时,这会浪费应用系统的处理资源。In addition, when the application system reads duplicate data from the target data, this wastes the processing resources of the application system.

通常,由于存储资源的数量越来越多,价格越来越便宜,因此,很多设计人员认为重复的数据不会对系统造成很大影响。但是,在实际应用过程中,发明人发现,对于大数据应用来说,这部分影响是不可忽略的。Often, due to the increasing number of storage resources and the increasingly cheaper prices, many designers believe that duplicate data does not have a significant impact on the system. However, in the actual application process, the inventors found that for big data applications, this part of the impact is not negligible.

基于这些考虑,在一个实施例中,所述数据生产者模块还计算每条第二数据的哈希值。可以基于所述哈希值来存储所述第二数据以避免所存储的第二数据的重复。Based on these considerations, in one embodiment, the data producer module also calculates a hash value for each piece of second data. The second data may be stored based on the hash value to avoid duplication of the stored second data.

由于第一数据来自不同的数据源,因此,即使第一数据的内容相同,它们的格式也可能不同。因此,很难发现第一数据的重复。这里,基于第二数据确定数据重复,可以避免这个问题。Since the first data comes from different data sources, their format may be different even if the contents of the first data are the same. Therefore, it is difficult to find the repetition of the first data. Here, the data duplication is determined based on the second data, and this problem can be avoided.

可以在向目标数据库存储第二数据时消除所述重复。例如,以对应的方式将所述哈希值与每条第二数据存储在数据共享内存中,所述数据消费者模块以哈希值作为每条第二数据的主键向目标数据库存储第二数据。这样,在存储数据时,对于具有相同主键的数据条目,可以由在后的数据条目覆盖在先的数据条目,或者丢弃在后的数据条目。The duplication can be eliminated when storing the second data to the target database. For example, storing the hash value and each piece of second data in a data sharing memory in a corresponding manner, the data consumer module storing the second data to the target database with a hash value as a primary key of each second data . Thus, when storing data, for a data entry having the same primary key, the previous data entry can be overwritten by the subsequent data entry, or the subsequent data entry can be discarded.

还可以在数据同步过程中消除所述重复。例如,以所述哈希值作为每条第二数据的主键将所述第二数据存储在数据共享内存中。这样,数据消费者可以直接从数据共享内存读取第二数据并向目标数据库存储第二数据。这减轻了后续处理。It is also possible to eliminate the duplication during data synchronization. For example, the second data is stored in the data shared memory with the hash value as the primary key of each second data. In this way, the data consumer can read the second data directly from the data shared memory and store the second data to the target database. This reduces subsequent processing.

例如,数据消费者可以仅从数据共享内存读取第二数据而不读取所述哈希值。在一个例子中,所述数据消费者模块从数据共享内存读取第二数据及对应的哈希值并向目标数据库存储第二数据及对应的哈希值。这样,目标数据库和/或系统应用也可以利用所述哈希值。例如,可以利用所述哈 希值计算第二数据在目标数据库中的地址。可选地,还可以利用所述哈希值验证第二数据的完整性。For example, the data consumer can read the second data only from the data shared memory without reading the hash value. In one example, the data consumer module reads the second data and the corresponding hash value from the data shared memory and stores the second data and the corresponding hash value to the target database. In this way, the target database and/or system application can also utilize the hash value. For example, the hash value can be utilized to calculate the address of the second data in the target database. Optionally, the hash value may also be utilized to verify the integrity of the second data.

<设备><device>

本领域技术人员应当理解,在电子技术领域中,可以通过软件、硬件以及软件和硬件结合的方式,将上述方法体现在产品中本领域技术人员很容易基于上面公开的方法,产生一种用于向目标数据库存储不同数据源的数据的设备,所述设备包括用于执行根据所述实施例的用于向目标数据库存储不同数据源的数据的方法中的各个操作。例如,所述设备包括:用于设置配置信息的装置,其中,所述配置信息表示来自每个数据源的第一数据在向目标数据库存储时的转换方式;用于基于所述配置信息将第一数据转换成第二数据的装置;以及用于向目标数据库存储所述第二数据的装置。It should be understood by those skilled in the art that in the field of electronic technology, the above method can be embodied in a product by software, hardware, and a combination of software and hardware. Those skilled in the art can easily generate a method based on the method disclosed above. A device that stores data of different data sources to a target database, the device comprising various operations in a method for performing data for storing different data sources to a target database in accordance with the described embodiments. For example, the apparatus includes: means for setting configuration information, wherein the configuration information represents a manner of conversion of first data from each data source when stored to a target database; Means for converting data into second data; and means for storing the second data to a target database.

<中间件设备><Middleware device>

可以将所述实施例应用于中间件设备。所述中间件设备可以位于服务器中。例如,随着电子技术的发展,终端设备和服务器的区别越来越小。因此,所述中间件设备也可以应用于终端设备。The embodiment can be applied to a middleware device. The middleware device can be located in a server. For example, with the development of electronic technology, the difference between terminal devices and servers is getting smaller and smaller. Therefore, the middleware device can also be applied to a terminal device.

图2示出了根据本发明的一个实施例的中间件设备的示意性框图。Figure 2 shows a schematic block diagram of a middleware device in accordance with one embodiment of the present invention.

如图2所示,中间件设备2000包括前面所述的的用于向目标数据库存储不同数据源的数据的设备2010。As shown in FIG. 2, the middleware device 2000 includes the device 2010 described above for storing data of different data sources to a target database.

另外,对于电子信息技术领域的普通技术人员来说,更为直接和清楚地描述一个技术方案的方式是描述该方案中的各个操作。在知道所要执行的操作的情况下,本领域技术人员可以直接设计出期望的产品。从这个方面来说,在这个实施例中,还提供了一种中间件设备,它被设计成用于执行图1所述的方法中的操作。In addition, for those of ordinary skill in the art of electronic information technology, a more straightforward and clear way of describing a technical solution is to describe the various operations in the solution. Those skilled in the art can directly design the desired product without knowing the operation to be performed. In this regard, in this embodiment, a middleware device is also provided which is designed to perform the operations in the method described in FIG.

本领域技术人员公知的是,随着诸如大规模集成电路技术的电子信息技术的发展和软件硬件化的趋势,要明确划分计算机系统软、硬件界限已经显得比较困难了。因为,任何操作可以软件来实现,也可以由应当来实现。任何指令的执行可以由硬件完成,同样也可以由软件来完成。对于某一机器功能采用硬件实现方案还是软件实现方案,取决于价格、速度、可靠性、存储容量、变更周期等非技术性因素。对于技术人员来说,软件实 现方式和硬件实现方式是等同的。技术人员可以根据需要选择软件或硬件来实现上述方案。因此,这里不对具体的软件或硬件进行限制。It is well known to those skilled in the art that with the development of electronic information technology such as large scale integrated circuit technology and the trend of software hardware, it has become difficult to clearly define the software and hardware boundaries of computer systems. Because any operation can be implemented in software, it can also be implemented by it. Execution of any instruction can be done by hardware, as well as by software. Whether a hardware implementation or a software implementation is used for a certain machine function depends on non-technical factors such as price, speed, reliability, storage capacity, and change cycle. For the technician, the software implementation and hardware implementation are equivalent. The technician can select software or hardware as needed to implement the above solution. Therefore, no specific software or hardware is limited here.

<服务器><server>

图3示出了是根据本发明的另一个实施例的服务器的示意性框图。FIG. 3 shows a schematic block diagram of a server in accordance with another embodiment of the present invention.

如图3所示,服务器3000可以包括图2所示的中间件设备2000。As shown in FIG. 3, the server 3000 may include the middleware device 2000 shown in FIG. 2.

目标数据库可以位于服务器3000内部,也可以通过有线或无线的方式连接到服务器3000。例如,所述服务器3000包括内存数据库,其中,所述中间件设备2000以所述内存数据库作为目标数据库并向所述内存数据库存储不同数据源的数据。The target database may be located inside the server 3000 or may be connected to the server 3000 by wire or wirelessly. For example, the server 3000 includes an in-memory database, wherein the middleware device 2000 uses the in-memory database as a target database and stores data of different data sources to the in-memory database.

图4示出了根据本发明的另一个实施例的服务器的示意性框图。Figure 4 shows a schematic block diagram of a server in accordance with another embodiment of the present invention.

如图4所示,服务器4000可以包括处理器4010、存储器4020、接口装置4030、通信装置4040、显示装置4050、输入装置4060、扬声器4070、麦克风4080,等等。As shown in FIG. 4, the server 4000 can include a processor 4010, a memory 4020, an interface device 4030, a communication device 4040, a display device 4050, an input device 4060, a speaker 4070, a microphone 4080, and the like.

处理器4010例如可以是中央处理器CPU、微处理器MCU等。存储器4020例如包括ROM(只读存储器)、RAM(随机存取存储器)、诸如硬盘的非易失性存储器等。接口装置4030例如包括USB接口、耳机接口等。The processor 4010 can be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory 4020 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like. The interface device 4030 includes, for example, a USB interface, a headphone jack, and the like.

通信装置4040例如能够进行有有线或无线通信。The communication device 4040 can perform wired or wireless communication, for example.

显示装置4050例如是液晶显示屏、触摸显示屏等。输入装置4060例如可以包括触摸屏、键盘等。用户可以通过扬声器4070和麦克风4080输入/输出语音信息。The display device 4050 is, for example, a liquid crystal display, a touch display, or the like. Input device 4060 can include, for example, a touch screen, a keyboard, and the like. The user can input/output voice information through the speaker 4070 and the microphone 4080.

图4所示的服务器仅是解释性的,并且决不是为了要限制本发明、其应用或用途。The server shown in Figure 4 is merely illustrative and is in no way intended to limit the invention, its application or use.

在这个实施例中,所述存储器4020用于存储指令,所述指令用于控制所述处理器4010进行操作以执行前面参照图1所述的用于向目标数据库存储不同数据源的数据的方法中的各个操作。本领域技术人员应当理解,尽管在图4中示出了多个装置,但是,本发明可以仅涉及其中的部分装置,例如,处理器4010和存储装置4020等。技术人员可以根据本发明所公开方案设计指令。指令如何控制处理器进行操作,这是本领域公知,故在此不再详细描述。In this embodiment, the memory 4020 is configured to store instructions for controlling the processor 4010 to operate to perform the method of storing data for storing different data sources to a target database as described above with reference to FIG. Each operation in . It will be understood by those skilled in the art that although a plurality of devices are illustrated in FIG. 4, the present invention may relate only to some of the devices, such as processor 4010 and storage device 4020, and the like. A technician can design instructions in accordance with the disclosed aspects of the present invention. How the instructions control the processor for operation is well known in the art and will not be described in detail herein.

<例子><example>

图5示出了可以应用本发明的实施例的系统的例子的示意性框图。Figure 5 shows a schematic block diagram of an example of a system to which embodiments of the invention may be applied.

如图5所示,服务器5040通过网络5010与数据库5020、5030连接。尽管在图5中示出了网络5010,但是,本领域技术人员应当理解,服务器5040可以直接与数据库5020、5030连接,或者,数据库5020、5030位于服务器5040内部。As shown in FIG. 5, server 5040 is coupled to databases 5020, 5030 via network 5010. Although network 5010 is shown in FIG. 5, those skilled in the art will appreciate that server 5040 can be directly coupled to databases 5020, 5030, or that databases 5020, 5030 can be located within server 5040.

另外,在图5中,服务器5040还与内存服务器5050连接。本领域技术人员应当理解,内存服务器5050可以位于服务器5040内部。In addition, in FIG. 5, the server 5040 is also connected to the memory server 5050. Those skilled in the art will appreciate that the memory server 5050 can be located internal to the server 5040.

可以在服务器5040中实现根据本发明的实施例。例如,所述服务器可以是前面所述的服务器。Embodiments in accordance with the present invention may be implemented in server 5040. For example, the server can be the server described above.

首先,设置配置信息。例如,配置信息可以包括表示向目标数据库存储数据的任务信息、表示数据源的数据源信息、表示数据转换方式的转换规则信息和表示数据存储目标的目的地信息。First, set the configuration information. For example, the configuration information may include task information indicating that data is stored in the target database, data source information indicating the data source, conversion rule information indicating a manner of data conversion, and destination information indicating a data storage destination.

下面的表格示出了所述信息的例子。The table below shows an example of the information.

(1)任务信息(1) Task information

Figure PCTCN2017120189-appb-000001
Figure PCTCN2017120189-appb-000001

(2)数据源信息(2) Data source information

Figure PCTCN2017120189-appb-000002
Figure PCTCN2017120189-appb-000002

Figure PCTCN2017120189-appb-000003
Figure PCTCN2017120189-appb-000003

(3)输出目的地(3) Output destination

Figure PCTCN2017120189-appb-000004
Figure PCTCN2017120189-appb-000004

(4)转换规则信息(4) Conversion rule information

Figure PCTCN2017120189-appb-000005
Figure PCTCN2017120189-appb-000005

可以根据不同的数据同步/传输任务,将所述信息转换成SQL语句,供后续使用。The information can be converted into SQL statements for subsequent use according to different data synchronization/transmission tasks.

例如,可以由数据同步执行器基于所述信息执行数据同步(向目标数据库存储不同数据源的数据)。For example, data synchronization (data storing different data sources to a target database) can be performed by the data synchronization executor based on the information.

数据同步执行器可以执行数据的读取、转换、存储到目标数据库。数 据同步执行器可以采用多线程以并发方式执行处理。例如,数据同步执行器采用生产者-消费者模式。例如,它包括组件:数据生产者、数据共享内存、数据消费者。The data synchronization executor can perform data reading, conversion, and storage to the target database. A data synchronous executor can perform processing in a concurrent manner using multiple threads. For example, data sync executors employ a producer-consumer model. For example, it includes components: data producers, data sharing memory, and data consumers.

图6示出了可以应用于本发明的实施例的生产者-消费者模式的例子的示意性框图。FIG. 6 shows a schematic block diagram of an example of a producer-consumer mode that can be applied to embodiments of the present invention.

如图6所示,根据上面所述的配置信息,数据生产者读取第一数据,例如,数据A表、数据B表、数据C表等。例如,数据生产者按照数据源信息读取相应的数据、表和字段等。数据生产者按照转换规则信息将第一数据转换成第二数据。As shown in FIG. 6, according to the configuration information described above, the data producer reads the first data, for example, a data A table, a data B table, a data C table, and the like. For example, the data producer reads the corresponding data, tables, fields, etc. according to the data source information. The data producer converts the first data into the second data according to the conversion rule information.

这里,数据生产者可以计算第二数据的哈希值。Here, the data producer can calculate the hash value of the second data.

例如,每5000条数据为一个批次,将数据存储在数据共享内存中。可以以所述哈希值为主键将第二数据存储在数据共享内存中,以防止重复。可选地,也可以先以对应的方式将所述哈希值和第二数据存储在数据共享内存中,并在向目标数据库存储第二数据时消除重复。For example, every 5000 data is a batch, and the data is stored in the data sharing memory. The second data may be stored in the data shared memory with the hash value as a primary key to prevent duplication. Optionally, the hash value and the second data may also be stored in the data sharing memory in a corresponding manner, and the repetition is eliminated when the second data is stored in the target database.

数据消费者(线程)从数据共享内存读取第二数据。例如,数据消费者可以读取第二数据和对应的哈希值。以所述哈希值作为主键,将第二数据存储到诸如Fooyum内存数据库的目标数据库,以防止数据重复。The data consumer (thread) reads the second data from the data shared memory. For example, the data consumer can read the second data and the corresponding hash value. The second data is stored in a target database such as a Fooyum in-memory database with the hash value as a primary key to prevent data duplication.

例如,数据消费者以5000条数据为一个批次读取第二数据,并包含多个线程以将第二数据存储到Fooyum内存数据库,以供应用系统使用。For example, the data consumer reads the second data for a batch of 5000 data and includes multiple threads to store the second data to the Fooyum in-memory database for use by the system.

数据共享内存可以包含数据生产者的状态,例如,数据读取中、数据已读完、读入数据异常等。当所述状态为“数据已读完”和“读入数据异常”时,数据生产者(线程)将结束。如果数据共享内存中的数据被消费完,或者如果所述状态为“读入数据异常”,则数据消费者将结束。The data shared memory can contain the status of the data producer, for example, during data reading, data has been read, data read abnormalities, and the like. When the status is "data has been read" and "read data is abnormal", the data producer (thread) will end. If the data in the data shared memory is consumed, or if the status is "read data abnormal", the data consumer will end.

在数据生产者和数据消费者结束后,可以将数据共享内存的运行状态、处理成功的数量、处理失败的数量、运行时间和处理时长等信息保存到数据库中。After the data producer and the data consumer are finished, information such as the running status of the data shared memory, the number of successful processing, the number of processing failures, the running time, and the processing duration can be saved to the database.

可以定时自动根据所述配置信息执行数据同步任务。这样,可以定时从不同的关系型数据库读取不同的数据,并按指定的转换规则将数据同步到内存数据库中。应用系统可以快速地从内存数据库中读取相关数据。所 述应用系统例如可以使用大数据推荐算法等。The data synchronization task can be automatically performed periodically according to the configuration information. In this way, different data can be read from different relational databases at regular intervals, and the data can be synchronized to the in-memory database according to the specified conversion rules. The application system can quickly read relevant data from the in-memory database. The application system can use, for example, a big data recommendation algorithm or the like.

例如,本发明的实施例可以应用于Hive数据仓库工具。For example, embodiments of the invention may be applied to Hive data warehousing tools.

本发明实施例还提供一种计算机可读存储介质,其上承载一个或多个计算机指令程序,计算机指令程序被一个或多个处理器执行时,一个或多个处理器执行实现一种用于向目标数据库存储不同数据源的数据的方法,包括:设置配置信息,其中,配置信息包括:来自每个数据源的第一数据在向目标数据库存储时的转换方式;基于配置信息,将第一数据转换成第二数据;以及向目标数据库存储第二数据。The embodiment of the present invention further provides a computer readable storage medium carrying one or more computer instruction programs thereon, when the computer instruction program is executed by one or more processors, one or more processors execute to implement one for A method for storing data of different data sources to a target database, comprising: setting configuration information, wherein the configuration information includes: a manner of converting the first data from each data source when storing to the target database; and based on the configuration information, the first The data is converted into the second data; and the second data is stored to the target database.

本领域普通技术人员可以理解:实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成,前述的程序可以存储于一计算机可读取存储介质中,该程序在执行时,执行包括上述任意方法实施例的步骤;而前述的存储介质包括:移动存储设备、随机存取存储器(RAM,Random Access Memory)、只读存储器(ROM,Read-Only Memory)、磁碟或者光盘等各种可以存储程序代码的介质。A person skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by using hardware related to the program instructions. The foregoing program may be stored in a computer readable storage medium, and the program is executed when executed. The foregoing storage medium includes: a mobile storage device, a random access memory (RAM), a read-only memory (ROM), a magnetic disk, or an optical disk. A medium that can store program code.

或者,本发明实施例上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。基于这样的理解,本发明实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行申请各个实施例所述方法的全部或部分。而前述的存储介质包括:移动存储设备、RAM、ROM、磁碟或者光盘等各种可以存储程序代码的介质。Alternatively, the above-described integrated unit of the embodiment of the present invention may be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as a stand-alone product. Based on such understanding, the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for making A computer device (which may be a personal computer, server, or network device, etc.) performs all or part of the methods described in the various embodiments. The foregoing storage medium includes various media that can store program codes, such as a mobile storage device, a RAM, a ROM, a magnetic disk, or an optical disk.

本发明实施例可以是设备、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质,其上载有用于使处理器实现本发明的各个方面的计算机可读程序指令。Embodiments of the invention may be apparatus, methods, and/or computer program products. The computer program product can comprise a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement various aspects of the present invention.

计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列 表)包括:便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身,诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如,通过光纤电缆的光脉冲)、或者通过电线传输的电信号。The computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, with instructions stored thereon A raised structure in the hole card or groove, and any suitable combination of the above. A computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.

这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备,或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令,并转发该计算机可读程序指令,以供存储在各个计算/处理设备中的计算机可读存储介质中。The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .

用于执行本发明操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码,所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等,以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中,通过利用计算机可读程序指令的状态信息来个性化定制电子电路,例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA),该电子电路可以执行计算机可读程序指令,从而实现本发明的各个方面。Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, including object oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider to access the Internet) connection). In some embodiments, the customized electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

这里参照根据本发明实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本发明实施例的各个方面。应当理解,流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合,都可以由计算机可读程序指令实现。Aspects of embodiments of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams can be implemented by computer readable program instructions.

这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器,从而生产出一种机器,使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时,产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中,这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作,从而,存储有指令的计算机可读介质则包括一个制造品,其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。The computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams. The computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, such that the computer readable medium storing the instructions includes An article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.

也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上,使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤,以产生计算机实现的过程,从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。The computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process. Thus, instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.

附图中的流程图和框图显示了根据本发明的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分,所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个连续的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或动作的专用的基于硬件的系统来实现,或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是,通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions. Executable instructions. In some alternative implementations, the functions noted in the blocks may also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

以上已经描述了本发明的各实施例,上述说明是示例性的,并非穷尽性的,并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下,对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择,旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进,或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本发明的范围由所附权利要求来限定。The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the various embodiments of the embodiments, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (18)

一种用于向目标数据库存储不同数据源的数据的方法,包括:A method for storing data of different data sources to a target database, comprising: 设置配置信息,其中,所述配置信息包括:来自每个数据源的第一数据在向目标数据库存储时的转换方式;Setting configuration information, wherein the configuration information includes: a manner of converting the first data from each data source when stored in the target database; 基于所述配置信息,将第一数据转换成第二数据;以及Converting the first data into the second data based on the configuration information; 向目标数据库存储所述第二数据。The second data is stored to the target database. 根据权利要求1所述的方法,其中,不同数据源表征不同数据库或同一数据库中的不同表。The method of claim 1 wherein the different data sources characterize different databases or different tables in the same database. 根据权利要求1所述的方法,其中,所述目标数据库是内存数据库。The method of claim 1 wherein the target database is an in-memory database. 根据权利要求1所述的方法,其中,使用生产者-消费者模式向目标数据库存储所述第一数据。The method of claim 1 wherein the first data is stored to a target database using a producer-consumer mode. 根据权利要求4所述的方法,使用生产者-消费者模式向目标数据库存储所述第一数据,包括:The method of claim 4, storing the first data to a target database using a producer-consumer mode, comprising: 在生产者-消费者模式中,数据生产者模块基于所述配置信息将第一数据转换成适合目标数据库的第二数据以及将第二数据存储在数据共享内存中;以及,In the producer-consumer mode, the data producer module converts the first data into second data suitable for the target database and the second data in the data shared memory based on the configuration information; 数据消费者模块从数据共享内存读取第二数据并向目标数据库存储第二数据。The data consumer module reads the second data from the data shared memory and stores the second data to the target database. 根据权利要求5所述的方法,所述方法还包括:The method of claim 5, the method further comprising: 确定所述数据生产者模块中各个线程是否均结束,或者确定所述数据消费者中各个线程是否均结束;Determining whether each thread in the data producer module ends, or determining whether each thread in the data consumer ends; 当所述数据生产者模块中各个线程均结束,或者所述数据消费者中各个线程均结束时,将数据共享内存中存储的数据生产者状态信息或者数据共享内存中存储的数据消费者状态信息存储至所述目标数据库。Data producer status information stored in the data sharing memory or data consumer status information stored in the data sharing memory when each thread in the data producer module ends, or each thread in the data consumer ends Stored to the target database. 根据权利要求6所述的方法,所述数据生产者状态信息包括以下至少一项:The method of claim 6, the data producer status information comprising at least one of the following: 数据生产者模块中各个线程分别对应的运行状态信息;Running status information corresponding to each thread in the data producer module; 所述数据生产者模块中处理失败的线程数量;The number of threads that failed to process in the data producer module; 所述数据生产者模块中处理成功的线程数量;The number of threads successfully processed in the data producer module; 所述数据生产者模块中各个线程分别对应的运行时间;a running time corresponding to each thread in the data producer module; 所述数据生产者模块中各个线程分别对应的处理时间;Processing time corresponding to each thread in the data producer module; 其中,所述数据消费者状态信息包括以下至少一项:The data consumer status information includes at least one of the following: 数据消费者模块中各个线程分别对应的运行状态信息;Running status information corresponding to each thread in the data consumer module; 所述数据消费者模块中处理失败的线程数量;The number of threads that failed to process in the data consumer module; 所述数据消费者模块中处理成功的线程数量;The number of threads successfully processed in the data consumer module; 所述数据消费者模块中各个线程分别对应的运行时间;a running time corresponding to each thread in the data consumer module; 所述数据消费者模块中各个线程分别对应的处理时间。The processing time corresponding to each thread in the data consumer module. 根据权利要求4-7任一项所述的方法,其中,还通过所述数据生产者模块计算每条第二数据的哈希值,以及A method according to any one of claims 4 to 7, wherein a hash value of each piece of second data is also calculated by said data producer module, and 其中,基于所述哈希值存储所述第二数据。Wherein the second data is stored based on the hash value. 根据权利要求8所述的方法,其中,以对应的方式将所述哈希值与每条第二数据存储在数据共享内存中;以及The method of claim 8 wherein said hash value and each second data are stored in a data sharing memory in a corresponding manner; 其中,所述数据消费者模块以哈希值作为每条第二数据的主键向目标数据库存储第二数据。The data consumer module stores the second data to the target database with a hash value as a primary key of each second data. 根据权利要求9所述的方法,其中,以所述哈希值作为每条第二数据的主键将所述第二数据存储在数据共享内存中。The method of claim 9, wherein the second data is stored in the data shared memory with the hash value as a primary key of each of the second data. 根据权利要求10所述的方法,其中,所述数据消费者模块从数据共享内存读取第二数据及对应的哈希值并向目标数据库存储第二数据及对应的哈希值。The method of claim 10 wherein said data consumer module reads the second data and the corresponding hash value from the data shared memory and stores the second data and the corresponding hash value to the target database. 根据权利要求1所述的方法,其中,所述配置信息还包括:向目标数据库存储数据的任务信息、数据源信息、数据转换方式的转换规则以及数据存储目的地信息。The method according to claim 1, wherein the configuration information further comprises: task information for storing data to the target database, data source information, a conversion rule of the data conversion mode, and data storage destination information. 一种用于向目标数据库存储不同数据源的数据的设备,包括:A device for storing data of different data sources to a target database, comprising: 设置模块,用于设置配置信息,其中,所述配置信息表示来自每个数据源的第一数据在向目标数据库存储时的转换方式;a setting module, configured to set configuration information, wherein the configuration information represents a conversion manner of the first data from each data source when being stored in the target database; 转换模块,用于基于所述设置模块设置的配置信息将第一数据转换成第二数据;a conversion module, configured to convert the first data into the second data based on the configuration information set by the setting module; 存储模块,用于向目标数据库存储所述转换模块转换得到的第二数据。And a storage module, configured to store the second data converted by the conversion module to the target database. 一种中间件设备,包括根据权利要求13所述的用于向目标数据库存储不同数据源的数据的设备,或者用于执行根据权利要求1-9中的任何一个所述的方法中的操作。A middleware device comprising the apparatus for storing data of different data sources to a target database according to claim 13, or for performing the operations in the method according to any one of claims 1-9. 一种服务器,包括根据权利要求14所述的中间件设备。A server comprising the middleware device of claim 14. 根据权利要求15所述的服务器,还包括内存数据库,其中,所述中间件设备以所述内存数据库作为目标数据库并向所述内存数据库存储不同数据源的数据。The server of claim 15 further comprising an in-memory database, wherein said middleware device uses said in-memory database as a target database and stores data of different data sources to said in-memory database. 一种服务器,包括存储器和处理器,其中,所述存储器用于存储指令,所述指令在所述服务器运行时控制所述处理器执行根据权利要求1-12中的任何一项所述的用于向目标数据库存储不同数据源的数据的方法。A server comprising a memory and a processor, wherein the memory is for storing instructions that, when the server is running, control the processor to perform the use of any of claims 1-12 A method of storing data from different data sources to a target database. 一种计算机可读存储介质,其上承载一个或多个计算机指令程序,所述计算机指令程序被一个或多个处理器执行时,所述一个或多个处理器执行权利要求1至12任一项所述的方法。A computer readable storage medium having one or more computer program programs thereon, the one or more processors executing any one of claims 1 to 12 when the computer program program is executed by one or more processors The method described in the item.
PCT/CN2017/120189 2017-03-22 2017-12-29 Method and device for storing data to database, middleware device, and server Ceased WO2018171289A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710173926.4A CN107122401A (en) 2017-03-22 2017-03-22 To the method for data database storing, equipment, middleware equipment and server
CN201710173926.4 2017-03-22

Publications (1)

Publication Number Publication Date
WO2018171289A1 true WO2018171289A1 (en) 2018-09-27

Family

ID=59717416

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/120189 Ceased WO2018171289A1 (en) 2017-03-22 2017-12-29 Method and device for storing data to database, middleware device, and server

Country Status (2)

Country Link
CN (1) CN107122401A (en)
WO (1) WO2018171289A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107122401A (en) * 2017-03-22 2017-09-01 广州优视网络科技有限公司 To the method for data database storing, equipment, middleware equipment and server
CN109564567B (en) * 2018-10-17 2023-07-25 北京算能科技有限公司 Data storage method, device, electronic equipment and computer readable storage medium
CN112328675A (en) * 2020-11-25 2021-02-05 上海市计算技术研究所 Heterogeneous data conversion method, device, device and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060106856A1 (en) * 2004-11-04 2006-05-18 International Business Machines Corporation Method and system for dynamic transform and load of data from a data source defined by metadata into a data store defined by metadata
CN102142027A (en) * 2011-02-23 2011-08-03 南京智尚丰软件有限公司 Adaptive method for data integration
CN102591725A (en) * 2011-12-20 2012-07-18 浙江鸿程计算机系统有限公司 Method for multithreading data interchange among heterogeneous databases
CN104462121A (en) * 2013-09-18 2015-03-25 腾讯科技(深圳)有限公司 Data processing method, device and system
US9158827B1 (en) * 2012-02-10 2015-10-13 Analytix Data Services, L.L.C. Enterprise grade metadata and data mapping management application
CN107122401A (en) * 2017-03-22 2017-09-01 广州优视网络科技有限公司 To the method for data database storing, equipment, middleware equipment and server

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105824849A (en) * 2015-01-08 2016-08-03 中国移动通信集团河南有限公司 Data import method and adapter

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060106856A1 (en) * 2004-11-04 2006-05-18 International Business Machines Corporation Method and system for dynamic transform and load of data from a data source defined by metadata into a data store defined by metadata
CN102142027A (en) * 2011-02-23 2011-08-03 南京智尚丰软件有限公司 Adaptive method for data integration
CN102591725A (en) * 2011-12-20 2012-07-18 浙江鸿程计算机系统有限公司 Method for multithreading data interchange among heterogeneous databases
US9158827B1 (en) * 2012-02-10 2015-10-13 Analytix Data Services, L.L.C. Enterprise grade metadata and data mapping management application
CN104462121A (en) * 2013-09-18 2015-03-25 腾讯科技(深圳)有限公司 Data processing method, device and system
CN107122401A (en) * 2017-03-22 2017-09-01 广州优视网络科技有限公司 To the method for data database storing, equipment, middleware equipment and server

Also Published As

Publication number Publication date
CN107122401A (en) 2017-09-01

Similar Documents

Publication Publication Date Title
US11163777B2 (en) Smart content recommendations for content authors
US11194779B2 (en) Generating an index for a table in a database background
US20190102441A1 (en) Data standardization techniques
WO2022121171A1 (en) Similar text matching method and apparatus, and electronic device and computer storage medium
US11200291B2 (en) Automated generation of web API descriptions from usage data
CN113485781B (en) Report generation method, device, electronic device and computer readable medium
US10346381B2 (en) Atomic update operations in a data storage system
US10353874B2 (en) Method and apparatus for associating information
US9652368B2 (en) Using linked data to determine package quality
WO2018157850A1 (en) Multi-thread segmented downloading method, device, client device, electronic device and storage medium
US10915537B2 (en) System and a method for associating contextual structured data with unstructured documents on map-reduce
US9588952B2 (en) Collaboratively reconstituting tables
US8577848B2 (en) Converting two-tier resource mapping to one-tier resource mapping
CN113485763B (en) Data processing method, device, electronic device and computer readable medium
CN110209662A (en) A kind of method and apparatus of automation load data
WO2018171289A1 (en) Method and device for storing data to database, middleware device, and server
CN111581363A (en) Knowledge extraction method, device, equipment and storage medium
WO2018177032A1 (en) Method and device for processing response data, client device and electronic device
US9823905B2 (en) Event based code generation
WO2022121172A1 (en) Text error correction method and apparatus, electronic device, and computer readable storage medium
US10102029B2 (en) Extending a map-reduce framework to improve efficiency of multi-cycle map-reduce jobs
CN112148705A (en) Data migration method and device
US20170124214A1 (en) Determining data field offsets using a document object model representation
CN111142871A (en) Front-end page development system, method, equipment and medium
CN114612699B (en) A method and device for processing image data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17902325

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17902325

Country of ref document: EP

Kind code of ref document: A1