CN108984603A - Isomeric data acquisition method, equipment, storage medium and system - Google Patents

Abstract

The invention discloses isomeric data acquisition method, equipment, storage medium and systems.By carrying out the inquiry of data field to preset data catalogue in the present invention, to judge whether the preset data catalogue includes corresponding data field；When the preset data catalogue includes corresponding data field, program is realized from the interface that server-side obtains isomeric data to be collected by client, client is allowed directly to realize that program obtains corresponding data from data providing by the interface.Thus in multiple data collection client systems when acquiring all kinds of external isomeric datas, without development interface, only corresponding data-interface need to be obtained from server-side realize program, the external data can be directly acquired, the largely repeatedly interface exploitation work existing for each FTP client FTP when solving the problems, such as to be acquired isomeric data.

Description

Heterogeneous data collection method, device, storage medium and system

technical field

The present invention relates to the field of telecommunications, and in particular to heterogeneous data collection methods, equipment, storage media and systems.

Background technique

At present, the data service interfaces of various data providers in the data market are mostly defined by themselves, and they are not uniform in terms of interface interaction methods, call parameters, and data formats. Therefore, when each data demander collects external data, it needs to separate the interfaces of different providers, and also needs to localize the heterogeneous data, resulting in a lot of repetitive work. As the number of suppliers or demanders increases, the amount of duplication of work will also multiply.

The above content is only used to assist in understanding the technical solution of the present invention, and does not mean that the above content is admitted as prior art.

Contents of the invention

The main purpose of the present invention is to provide a heterogeneous data collection method, user equipment, storage medium and system, aiming to solve the problem of a lot of repetitive work in the prior art when collecting heterogeneous data.

For the above purpose, the present invention provides a method for collecting heterogeneous data, and the method for collecting heterogeneous data includes the following steps:

The data acquisition device responds to the data field query instruction input by the user, and determines whether the preset data directory contains the data field corresponding to the data field query instruction;

When the preset data directory includes a data field corresponding to the data field query instruction, the data acquisition device acquires an interface implementation program from the server;

The interface implementation program acquires the heterogeneous data to be collected from the data provider, and writes the collected heterogeneous data into the data field corresponding to the data field query instruction.

Preferably, after the judging whether the preset data directory contains the data field corresponding to the data field query instruction, the heterogeneous data collection method further includes:

When the data field does not exist in the default data directory, adding the data field corresponding to the data field query instruction to the default data directory;

The data acquisition device acquires an interface implementation program from the server;

The heterogeneous data to be collected is obtained from the data provider through the interface implementation program, and the collected heterogeneous data is written into the added data column.

Preferably, adding the data field corresponding to the data field query instruction to the preset data directory specifically includes:

Obtain an interface document of the data to be collected from the server, and judge whether the corresponding target data interface can be directly configured according to the interface document;

When the target data interface cannot be directly configured, send a special data interface command to the server to obtain the special data field returned by the server;

The feature data field is used as a data field corresponding to the data field query instruction, and the data field corresponding to the data field query instruction is updated to the default data directory.

Preferably, after determining whether the corresponding target data interface can be directly configured according to the interface document, the heterogeneous data collection method further includes:

When the target data interface can be directly configured, configure the general data interface, send an update command to the server, and obtain the general data field corresponding to the general interface returned by the server;

The general data field is used as the data field corresponding to the data field query instruction, and the data field corresponding to the data field query instruction is updated to the default data directory.

Preferably, the data acquisition device acquires an interface implementation program from the server, specifically including:

Receiving the target interface implementation program returned by the server, judging whether the target data interface configuration in the target interface implementation program and the data field are legal, and when the judgment result is legal, the data acquisition device obtains the Interface implementer.

Preferably, after determining whether the configuration of the target data interface in the target data interface and the data fields are legal, the method further includes:

When the judging result is invalid, modify the target data interface configuration and the data field.

Preferably, after writing the heterogeneous data to be collected into the data field corresponding to the data field query instruction, the method further includes:

The data field is output and displayed, so that the user can call the heterogeneous data to be collected.

Preferably, the data collection system includes:

Judgment module: used for the data acquisition device to respond to the data field query instruction input by the user, and judge whether the preset data directory contains the data field corresponding to the data field query instruction;

A control module: used for, when the preset data directory includes a data field corresponding to the data field query instruction, the data acquisition device acquires an interface implementation program from the server;

Acquisition module: used to obtain the heterogeneous data to be collected from the data provider through the interface implementation program, and write the collected heterogeneous data into the data field corresponding to the data field query instruction.

In addition, the present invention also proposes a data collection device, the data collection device includes: a memory, a processor, and a data collection program stored in the memory and operable on the processor, the data collection program is configured are the steps of the data collection method described above.

In addition, the present invention also proposes a storage medium, on which a data collection program is stored, and when the data collection program is executed by a processor, the steps of the above-mentioned heterogeneous data collection method are described.

The technical solution of the present invention checks the data field to determine whether the preset data directory contains the corresponding data field; when the preset data directory contains the corresponding data field, the data collection device will The terminal obtains the interface implementation program, and obtains the heterogeneous data to be collected from the data provider through the interface implementation program; writes the heterogeneous data to be collected into the corresponding data column, so that the data collection device is collecting external data ( For example, when using a server), there is no need to customize the development interface, just obtain the corresponding data interface from the server, implement the program to read the heterogeneous data to be collected through the interface, realize the localization processing of heterogeneous data, and solve the problem of collecting heterogeneous data There is a lot of duplication of work.

Description of drawings

FIG. 1 is a schematic structural diagram of a user equipment in a hardware operating environment involved in an embodiment of the present invention;

Fig. 2 is a schematic flow chart of the first embodiment of the heterogeneous data collection method of the present invention;

Fig. 3 is a schematic flow chart of the second embodiment of the heterogeneous data collection method of the present invention;

Fig. 4 is a schematic flow chart of the third embodiment of the heterogeneous data collection method of the present invention;

Fig. 5 is a block diagram of an embodiment of the data acquisition device of the present invention.

The purpose, functional features and advantages of the present invention will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The heterogeneous data collection method of the present invention is based on a data collection device, and the data collection device can be an electronic device such as a personal computer or a smart phone.

As shown in FIG. 1 , the data collection device 100 may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 can include a display unit (Display), an input unit such as an interactive interface. In the present invention, the data acquisition device 100 can interact with the user terminal during the software running process. When the data acquisition device 100 performs parameter setting or debugging, Testers or setup personnel can use the user interface 1003 to input data information, and the optional user interface 1003 can also include standard wired interfaces and wireless interfaces. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Based on the above hardware structure, an embodiment of the heterogeneous data collection method of the present invention is proposed.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a first embodiment of a method for collecting heterogeneous data according to the present invention.

In the first embodiment, the heterogeneous data collection method includes the following steps:

Step S10: In response to the data field query command input by the user, the data collection device determines whether the preset data directory contains the data field corresponding to the data field query command.

It can be understood that the execution subject of this embodiment is the data acquisition device. Considering that when users collect various external heterogeneous data through data collection equipment, in the data docking scenario, the data demander needs to carry out system docking according to the technical specifications (interface specifications, data specifications) of the data provider. The usual practice is that the demand side develops the interface by itself and deploys it in its own system environment. Whether the environment is in a self-built computer room or in a cloud computing environment, it is a must for the data demand side to develop the interface according to the technical specifications of the data provider. indispensable link. This development link always exists when the demand side or the provider is changed. In this embodiment, the data requester is responsible for the implementation of the interface, and the client obtains the interface implementation from the server and is responsible for corresponding data access and processing. In this way, the repeated development work of the client on the interface is reduced, and various types of data can be accessed quickly and flexibly.

It should be understood. When the data requirement opens a default input page, there are multiple data directories in the default input page, and different data directories correspond to different data fields. For example, the type of data directory can include judicial, identity information, Internet, vehicle, location, operator, bank card, travel, etc., and the data directory can also include subdirectories, such as further division of operator For China Unicom, China Mobile, China Telecom, and the entire network.

After selecting the type of data catalog to be queried, enter the field query page. In the field query input box, enter the name of the data field to be queried. By matching the input data field name with the data field in the default data directory, it can be determined whether the default data directory contains the data field corresponding to the data field query instruction based on the matching result.

The data field can be identity information, mobile phone number, name, etc.

Step S20: when the preset data directory includes a data field corresponding to the data field query instruction, the data acquisition device acquires an interface implementation program from the server;

It can be understood that when the name of the input data field matches the data field in the default data directory successfully, it means that the default data directory includes the data field corresponding to the data field query instruction. At this point, the required data fields can be directly seen on the preset page, and the data acquisition device downloads the corresponding interface implementation program from the server.

Step S30: Obtain the heterogeneous data to be collected from the data provider through the interface implementation program, and write the collected heterogeneous data into the data field corresponding to the data field query instruction.

It is easy to understand that the data collection device reads the heterogeneous data to be collected from the data provider through the interface implementation program. After the heterogeneous data is downloaded from the server, it is stored in the memory of the data collection device. Read heterogeneous data in the database, and write the read heterogeneous data into the data column for display, which is convenient for data demanders to view.

The technical solution of the present invention checks the data field to determine whether the preset data directory contains the corresponding data field; when the preset data directory contains the corresponding data field, the data collection device will The terminal obtains the interface implementation program, and obtains the heterogeneous data to be collected from the data provider through the interface implementation program; writes the heterogeneous data to be collected into the corresponding data column, so that the data collection device is collecting external data ( For example, when using a server), there is no need to customize the development interface, just obtain the corresponding data interface from the server, implement the program to read the heterogeneous data to be collected through the interface, realize the localization processing of heterogeneous data, and solve the problem of collecting heterogeneous data There is a lot of duplication of work.

Referring to FIG. 3 , in the second embodiment, after the determination of whether the preset data directory contains the data field corresponding to the data field query instruction, the heterogeneous data collection method further includes:

Step S40: When the data field does not exist in the default data directory, add the data field corresponding to the data field query instruction to the default data directory.

It is worth noting that the data fields can be manually configured by business personnel or automatically configured by the server. During manual configuration, business personnel configure on the preset page, define the data interface through a templated configuration process, and add the corresponding data fields to the data directory. In one embodiment, the templated configuration process includes the following steps: adding a process, adding an interface, adding a header parameter, adding a parameter, adding a field, adding a field mapping, adding decryption information, and adding a new Add analysis information. After passing these configurations, the data fields are added to the data directory accordingly.

When the server performs automatic configuration, the data acquisition device downloads the interface implementation program from the server, and configures and installs it on the data acquisition device.

S50. The data collection device acquires an interface implementation program from the server.

It is easy to understand that the data acquisition device downloads the corresponding interface implementation program from the server, configures and installs it on the data acquisition device, and the data acquisition device reads the heterogeneous data to be collected from the server through the interface implementation program.

S60. Obtain the heterogeneous data to be collected from the data provider through the interface implementation program, and write the collected heterogeneous data into the added data column. It is easy to understand that step S60 here is the same as step S30 in the first embodiment.

Referring to Fig. 4, in the third embodiment, the data field corresponding to the data field query instruction is added to the preset data directory, and the heterogeneous data collection method specifically includes:

Step S401: Obtain an interface document of the data to be collected from the server, and judge whether the corresponding target data interface can be directly configured according to the interface document.

It is understandable that in project development, the front-end and back-end of web projects are developed separately, and APP development requires front-end and back-end engineers to jointly define interfaces, write interface documents, and then develop based on this interface document.

Therefore, the method, uri, request parameters and return parameters of the target data interface are recorded in the interface document. The type of the target data interface can be known through the interface document.

In one embodiment, by checking the api interface document of the data to be collected (given by the data provider), it is judged according to the client access description of the api interface document whether the target data interface belongs to the general single-call type http or https Type interface, if so, it can be directly configured and implemented.

Step S402: When the target data interface cannot be configured directly, send a special data interface command to the server to obtain the special data field returned by the server.

It is worth noting that when the target data interface type is not a single-call type http or https, the data acquisition device sends a special data interface instruction to the server, and at the same time attaches the api interface document of the special data interface, and the server receives the special data interface When requested, implement the special data interface and special data field, and notify the data collection device. After receiving the notification, the data collection device downloads the special data fields.

Step S403: Use the feature data field as a data field corresponding to the data field query command, and update the data field corresponding to the data field query command to the default data directory.

In this way, when the target data interface cannot be directly configured, the special data field is received from the server to realize the collection of heterogeneous data, without the need to develop the data interface on the data collection device, thereby reducing the repetitive development work of the client on the interface, and quickly Flexible access to various types of data.

Further, after determining whether the corresponding target data interface can be directly configured according to the interface document, the heterogeneous data collection method further includes:

Step S405: When the target data interface can be directly configured, configure the general data interface, send an update command to the server, and obtain the general data field corresponding to the general interface returned by the server.

In an embodiment, when the target data interface is a single-call type http or https, it indicates that it can be directly configured. Through the manual configuration of the general interface by business personnel, the data acquisition device sends an update instruction to the server, the update instruction includes the information contained in the API interface document of the general interface, and the server implements the general data corresponding to the general interface according to the update instruction Field, the server notifies the data collection device to download.

Step S406: Using the common data field as the data field corresponding to the data field query instruction, updating the data field corresponding to the data field query instruction to the default data directory.

In this way, when the target data interface can be directly configured, the general data field is received from the server to realize the collection of heterogeneous data, and there is no need to develop the data interface on the data collection device, thereby reducing the repetitive development work of the client on the interface, and quickly Flexible access to various types of data.

Further, the data acquisition device acquires an interface implementation program from the server, specifically including:

Receiving the target interface implementation program returned by the server, judging whether the target data interface configuration in the target interface implementation program and the data field are legal, and when the judgment result is legal, the data acquisition device obtains the Interface implementer.

Specifically, by manually opening to a preset page, it is judged whether the target data interface configuration and the data field are correct, thereby determining whether it is legal.

It can be understood that, in this embodiment, by judging whether the data interface configuration and the data fields are legal, and then reading the data, the reliability and security of data collection can be improved.

Further, after judging whether the configuration of the target data interface in the target data interface and the data fields are legal, the method further includes that the heterogeneous data collection method may also include:

When the judging result is invalid, modify the target data interface configuration and the data field. It is worth noting that after the modification is completed, it is judged again whether the configuration of the target data interface and the data field are legal until it is judged to be legal.

In this way, the reliability of heterogeneous data collection can be improved.

To sum up, the technical solution of the present invention includes two software systems, one is a server system deployed on the Internet, and the other is a client system deployed in the environment of each data demander, both of which can communicate through the Internet, and also Communication can be carried out through the operator's dedicated line or VPN.

The server system is responsible for the data interface implementation program of the provider. The client obtains these data interface implementation programs from the server through the Internet, and each data interface implementation program runs in the client system, thereby directly connecting the client and each data provider. .

In the actual use process, the business personnel on the data demand side can directly obtain the corresponding interface implementation for the existing data by browsing and retrieving the contents of the data directory. If the required data does not exist in the directory, for the general provider api data interface, the business personnel of the data demand side can configure a new data provider interface in the client system and add new data fields; Configure the implemented api data interface, and business personnel can submit an interface implementation request to the server through the client, and obtain the interface implementation after the server is implemented, and add a new data field.

The data demand side only needs the unified data interface of the client system, that is, it can access various data fields in a unified format, process data values with custom data field processing logic, and quickly and flexibly collect external data into its own application system.

Referring to Fig. 5, based on the above heterogeneous data collection method, the present invention also proposes a heterogeneous data collection system, the data collection system includes:

Judgment module 10: used for the data acquisition device to respond to the data field query instruction input by the user, and judge whether the preset data directory contains the data field corresponding to the data field query instruction.

It should be noted that when a data requirement opens a preset input page, there are multiple data directories in the preset input page, and different data directories correspond to different data fields. For example, the type of data directory can include judicial, identity information, Internet, vehicle, location, operator, bank card, travel, etc., and the data directory can also include subdirectories, such as further division of operator For China Unicom, China Mobile, China Telecom, and the entire network.

After selecting the type of data catalog to be queried, enter the field query page. In the field query input box, enter the name of the data field to be queried. By matching the input data field name with the data field in the default data directory, it can be determined whether the default data directory contains the data field corresponding to the data field query instruction based on the matching result.

The control module 20 is configured to, when the preset data directory includes the data field corresponding to the data field query instruction, the data collection device acquires an interface implementation program from the server.

It can be understood that when the name of the input data field matches the data field in the default data directory successfully, it means that the default data directory includes the data field corresponding to the data field query instruction. At this point, you can directly see the required data fields on the preset page. The data acquisition device downloads the corresponding interface implementation program from the server, configures and installs it on the data acquisition device, and the data acquisition device realizes the program through the interface and the server. The data reading interface in reads the heterogeneous data to be collected from the server.

The collection module 30: used to obtain the heterogeneous data to be collected from the data provider through the interface implementation program, and write the collected heterogeneous data into the data field corresponding to the data field query instruction.

It is easy to understand that after the heterogeneous data is downloaded from the data provider, it is stored in the memory of the data acquisition device, and the data acquisition device reads the heterogeneous data from the memory and writes the read heterogeneous data into the data field for further processing. Display, to facilitate the data demand side to check.

To achieve the object of the present invention, the present invention also proposes a storage medium, on which a data collection program is stored, and when the data collection program is executed by a processor, the above-mentioned steps of the heterogeneous data collection method are described.

It should be noted that, as used herein, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or system. Without further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments. The use of the words first, second, and third etc. does not indicate any order and these words may be interpreted as designations.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general-purpose hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. A heterogeneous data collection method, characterized in that, the heterogeneous data collection method comprises the following steps: The data acquisition device responds to the data field query instruction input by the user, and determines whether the preset data directory contains the data field corresponding to the data field query instruction; When the preset data directory includes a data field corresponding to the data field query instruction, the data acquisition device acquires an interface implementation program from the server; The interface implementation program acquires the heterogeneous data to be collected from the data provider, and writes the collected heterogeneous data into the data field corresponding to the data field query instruction. 2. The method for collecting heterogeneous data according to claim 1, wherein after said judging whether the preset data directory contains the data field corresponding to the data field query instruction, said heterogeneous data collection Methods also include: When the data field does not exist in the default data directory, adding the data field corresponding to the data field query instruction to the default data directory; The data acquisition device acquires an interface implementation program from the server; The heterogeneous data to be collected is obtained from the data provider through the interface implementation program, and the collected heterogeneous data is written into the added data field. 3. The heterogeneous data collection method according to claim 2, wherein the adding the data field corresponding to the data field query instruction to the preset data directory specifically includes: Obtain an interface document of the data to be collected from the server, and judge whether the corresponding target data interface can be directly configured according to the interface document; When the target data interface cannot be directly configured, send a special data interface command to the server to obtain the special data field returned by the server; The feature data field is used as a data field corresponding to the data field query instruction, and the data field corresponding to the data field query instruction is updated to the default data directory. 4. The method for collecting heterogeneous data according to claim 3, wherein, after determining whether the corresponding target data interface can be directly configured according to the interface document, the method for collecting heterogeneous data further comprises: When the target data interface can be directly configured, configure the general data interface, send an update command to the server, and obtain the general data field corresponding to the general interface returned by the server; The general data field is used as the data field corresponding to the data field query instruction, and the data field corresponding to the data field query instruction is updated to the default data directory. 5. The heterogeneous data collection method according to any one of claims 1 to 4, wherein the data collection device acquires an interface implementation program from the server, specifically comprising: Receiving the target interface implementation program returned by the server, judging whether the target data interface configuration in the target interface implementation program and the data fields are legal, and when the judgment result is legal, obtain the Interface implementer. 6. The method for collecting heterogeneous data according to claim 5, wherein after the judging whether the target data interface configuration in the target data interface and the data field are legal, the method further comprises: When the judging result is invalid, modify the target data interface configuration and the data field. 7. The method for collecting heterogeneous data according to any one of claims 1 to 4, wherein the writing of the heterogeneous data to be collected into the data field corresponding to the data field query instruction Afterwards, the method also includes: The data field is output and displayed, so that the user can call the heterogeneous data to be collected. 8. A heterogeneous data acquisition system, characterized in that the data acquisition system comprises: Judging module: used for the data acquisition device to respond to the data field query instruction input by the user, and judge whether the preset data directory contains the data field corresponding to the data field query instruction; A control module: used for, when the preset data directory includes a data field corresponding to the data field query instruction, the data acquisition device acquires an interface implementation program from the server; Acquisition module: used to implement the program through the interface to obtain the heterogeneous data to be collected from the server, and write the collected heterogeneous data into the data field corresponding to the data field query instruction. 9. A data acquisition device, characterized in that, the data acquisition device comprises: a memory, a processor and a data acquisition program stored on the memory and operable on the processor, the data acquisition program is configured It is the step of the data acquisition method as described in any one of claims 1 to 7. 10. A storage medium, characterized in that, a data collection program is stored on the storage medium, and when the data collection program is executed by a processor, the heterogeneous data collection method according to any one of claims 1 to 7 A step of.