CN116011972A - Data processing system, method, electronic equipment and computer storage medium - Google Patents
Data processing system, method, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN116011972A CN116011972A CN202310032793.4A CN202310032793A CN116011972A CN 116011972 A CN116011972 A CN 116011972A CN 202310032793 A CN202310032793 A CN 202310032793A CN 116011972 A CN116011972 A CN 116011972A
- Authority
- CN
- China
- Prior art keywords
- data
- component
- query
- detail
- freight rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- General Engineering & Computer Science (AREA)
- Economics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing system, a data processing method, electronic equipment and a computer storage medium, wherein the data processing system comprises a data acquisition service component, a data communication component, a flow statistics query component and a cache component; the data acquisition service component is used for acquiring freight rate search data in freight rate search engine service, and processing the freight rate search data to obtain processed detail data; the detail data is sent to a flow statistics query component through a data communication component; the flow statistics query component is used for acquiring the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the detail data successfully received, and writing the statistical data obtained by processing into a cache component. According to the invention, the collected data is processed and then stored; so that the following user can inquire the corresponding data through the flow statistics inquiry component, and the processing efficiency and the processing accuracy can be improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing system, a data processing method, an electronic device, and a computer storage medium.
Background
In the context of the explosive development of the global civil aviation industry and the ever expanding economic globalization, the number of airlines and airports is increasing. Airlines and airports will generate more airlines, their corresponding tariffs, and more user group data. Resulting in more tariff search traffic and more complex traffic information data, and therefore how to store, analyze and utilize such data is a problem that has to be faced at present.
At present, the generated business information data is often stored, analyzed and utilized manually. However, the manual processing is prone to errors and longer processing time, resulting in lower processing rates and lower accuracy.
Disclosure of Invention
In view of the above, embodiments of the present invention provide a data processing system, a method, an electronic device, and a computer storage medium, so as to solve the problems of low processing rate and low accuracy in the prior art.
In order to achieve the above object, the embodiment of the present invention provides the following technical solutions:
a first aspect of an embodiment of the present invention shows a data processing system, the data processing system including a data acquisition service component, a data communication component, a traffic statistics query component, and a cache component;
the data acquisition service component is used for acquiring freight rate search data in freight rate search engine service, and processing the freight rate search data to obtain processed detail data; sending the detail data to the flow statistics query component through the data communication component;
the flow statistics query component is used for acquiring the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into a cache component.
Optionally, the data acquisition service component is further configured to:
in the process of sending the detail data to a flow statistics query component through the data communication component, detecting the network condition of a transmission network;
if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component;
if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted;
and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
Optionally, the data acquisition service component for processing the freight rate search data to obtain processed detail data is specifically configured to:
extracting service fields in the freight rate search data;
and sorting the service fields according to the format of a preset template to obtain detail data.
Optionally, the successfully received detail data is subjected to statistical processing, and the statistical data obtained by the processing is written into the flow statistical query component of the cache component, which is specifically used for:
carrying out statistical processing on service fields in the detail data to generate a freight rate search flow statistical identification name;
summarizing the traffic statistics identification names based on the freight rate searching flow statistics identification names to obtain statistics data;
and writing the statistical data into a cache component and a database.
Optionally, the flow statistics query component is further configured to:
if a data query instruction sent by a user terminal is received, verifying whether query request parameters in the data query instruction accord with preset rules;
if the query request parameters meet the preset rules, traversing the cache component to determine whether a cache result corresponding to the query request parameters exists;
and if the buffer memory results exist, feeding back the corresponding buffer memory results to the user terminal.
A second aspect of the embodiment of the present invention shows a data processing method, which is applied to the data processing system shown in the first aspect of the embodiment of the present invention, where the data processing system includes a data acquisition service component, a data communication component, a traffic statistics query component, and a cache component; the method comprises the following steps:
the data acquisition service component acquires freight rate search data in freight rate search engine service, and processes the freight rate search data to obtain processed detail data; sending the detail data to the flow statistics query component through the data communication component;
the flow statistics query component acquires the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into a cache component.
Optionally, the method further comprises:
the data acquisition service component detects the network condition of the transmission network;
if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component;
if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted;
and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
Optionally, the processing the freight rate search data to obtain processed detail data includes:
extracting service fields in the freight rate search data;
and sorting the service fields according to the format of a preset template to obtain detail data.
A third aspect of an embodiment of the present invention shows an electronic device, where the electronic device is configured to execute a program, where the program executes a data processing method as shown in the second aspect of the embodiment of the present invention.
A fourth aspect of the embodiment of the present invention shows a computer storage medium, where the storage medium includes a storage program, where the program, when executed, controls a device in which the storage medium is located to execute a data processing method as shown in the second aspect of the embodiment of the present invention.
Based on the data processing system, the method, the electronic device and the computer storage medium provided by the embodiment of the invention, the data processing system comprises a data acquisition service component, a data communication component, a flow statistics query component and a cache component; the data acquisition service component is used for acquiring freight rate search data in freight rate search engine service, and processing the freight rate search data to obtain processed detail data; sending the detail data to the flow statistics query component through the data communication component; the flow statistics query component is used for acquiring the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into a cache component. In the embodiment of the invention, the collected data is processed in advance and then stored, so that the service data of the freight rate search engine service can be transmitted reliably in real time, and the integrity, high availability and consistency of the data are ensured; so that the following user can inquire the corresponding data through the flow statistics inquiry component, and the processing efficiency and the processing accuracy can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another data processing system in accordance with an embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 4 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data push architecture according to an embodiment of the present invention;
FIG. 6 is a flow chart of a further data processing method according to an embodiment of the invention
FIG. 7 is a schematic diagram of a statistical data processing architecture according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a query data processing architecture according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
With reference now to FIG. 1, a block diagram illustrating a data processing system is depicted in accordance with an embodiment of the present invention, the system includes: a data acquisition service component 10, a data communication component 20, a traffic statistics query component 30, and a caching component 40.
The data acquisition service component 10 is connected with the data communication component 20, the data communication component 20 is connected with the flow statistics query component 30, and the flow statistics query component 30 is connected with the cache component 40.
The data acquisition service component 10 is configured to acquire freight rate search data in freight rate search engine service, and process the freight rate search data to obtain processed detail data; the detail data is sent to the flow statistics query component via the data communications component 20.
Optionally, the data acquisition service component 10 for processing the freight rate search data to obtain processed detail data is specifically configured to:
extracting service fields in the freight rate search data;
and sorting the service fields according to the format of a preset template to obtain detail data.
In the specific implementation of the data acquisition service component 10, the data acquisition service component 10 acquires the freight rate search data of the freight rate search engine service in real time, sorts the acquired freight rate search data according to service fields such as a navigation driver or agent user Id, freight rate search service type, freight rate search result and the like, writes the sorted service fields into a preset template according to the format of the preset template to obtain the detail data of freight rate search flow, and distributes the sorted detail data to the flow statistics query service 30 through the distributed publishing and subscribing message system kafka arranged in the data communication component 20.
It should be noted that the freight rate search service type may be domestic, international, etc., and the freight rate search result may be success, failure, error code, etc.
The preset template is set by a technician according to the format of the field, and can be specifically changed according to actual conditions.
The flow statistics query component 30 is configured to obtain the detail data that is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into the cache component 40.
Optionally, the successfully received detail data is subjected to statistical processing, and the statistical data obtained by the processing is written into the flow statistical query component 30 of the cache component, which is specifically configured to:
carrying out statistical processing on service fields in the detail data to generate a freight rate search flow statistical identification name; summarizing the traffic statistics identification names based on the freight rate searching flow statistics identification names to obtain statistics data; the statistics are written to the caching component 40, as well as to a database.
In the specific implementation of the flow statistics query component 30, the flow statistics query component 30 obtains the detail data of the freight rate searching flow through kafka, and generates freight rate searching flow statistics identification names formed by the navigation or agent user Id, freight rate searching service type, request result state and the like according to freight rate searching service field statistical calculation; summarizing the script in a memory database redis of the cache component 40 through a script language lua according to the freight rate search flow statistics identification name to obtain statistics data; and then starting a statistical data synchronization task according to a preset period, and synchronizing the statistical data into a edb database for persistence.
It should be noted that the preset period is preset by a technician according to the actual situation.
According to the invention, in redis, the statistical results of a plurality of freight rate searching and statistical query services are summarized through the lua script, so that the consistency of distributed data can be ensured, and the statistical calculation efficiency can be improved. Periodically synchronizing the redis summary data to edb, rather than directly summarizing edb, may reduce edb the data writing frequency and improve the data throughput of the overall system.
Data present in the caching component 40 is periodically purged, e.g., data is cached for more than 14 days.
Based on the data processing system shown in the above-mentioned embodiments of the present invention, referring to fig. 1 in combination with fig. 2, the system further includes:
a service dispatch management component 50 coupled to the traffic statistics query service 30, and a data persistence storage component 60 coupled to the caching component 40.
The service scheduling management component 50, i.e. the zookeeper service, is responsible for service registration of the plurality of traffic statistics query components 30 and detecting user terminals, i.e. user terminal discovery, configuration management of the traffic statistics query service 30, master-slave role election of the traffic statistics query service 30, and the like.
The data persistence storage component 60, i.e., the edb database, is responsible for persistence storing the summary statistics for providing the underlying data for subsequent query and archiving operations.
It should be noted that, the database shown in the application refers to edb database, the database is stored in a data table format, the data table is split horizontally on the premise of conforming to the BC format, so as to ensure that the data volume of each table is not too large, and an index table is additionally constructed for basic information in redis, so that data query, writing in and archiving operation can be improved through indexes.
Further, the basic information includes the aviator user, the agent user and the corresponding user Id.
Optionally, the traffic statistics query component 30 is further configured to periodically archive data three years ago in the database. Firstly, exporting data exceeding three years into a file in a structured form, namely a preset format is taken as a unit of month, compressing the exported data file into a compression package in a unit of year, naming the file in a form of year and version number, and uniformly uploading the file name into a distributed file storage system for archiving and storing.
It should be noted that the preset format is preset by a technician, for example, may be set in json form.
In the embodiment of the invention, the collected data is processed in advance and then stored, so that the service data of the freight rate search engine service can be transmitted reliably in real time, and the integrity, high availability and consistency of the data are ensured; so that the following user can inquire the corresponding data through the flow statistics inquiry component, and the processing efficiency and the processing accuracy can be improved.
Optionally, based on the data processing system shown in the foregoing embodiment of the present invention, the data acquisition service component 10 is further configured to:
if the detail data is transmitted to the flow statistics query component through the data communication component, detecting the network condition of the transmission network in real time; if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component; if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted; and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
In a specific implementation, the data collection service component 10 detects whether the network condition of the transmission network is abnormal in real time in the process of sending the detail data to the flow statistics query component 30 through the data communication component, if not, determines that no network cause causing data sending failure exists at the moment, and can normally send the detail data to the flow statistics query component 30 through the data communication component 20. If it is determined that the network condition is abnormal, if the data transmission fails due to the network abnormality, the detail data with failed transmission, that is, the traffic data with failed pushing, is converted into a structured data format, and is temporarily stored in the local disk of the data acquisition service assembly 10. If it is determined that the network condition is recovered from abnormal to normal, a compensation sending task thread is started after the network is recovered, and the freight rate searching flow detail data in the local disk is timely compensated and sent, that is, the temporarily stored detail data of the local disk is sent to the flow statistics inquiring component 30 through the data communication component according to preset time, so that the integrity of data transmission is ensured.
In the embodiment of the invention, the collected data is processed in advance, the processed data is transmitted, and the network condition of the transmission network is detected in real time in the process of sending the detail data to the flow statistics query component through the data communication component; if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component; if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted; and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
Optionally, based on the data processing system shown in the foregoing embodiment of the present invention, the traffic statistics query component 30 is further configured to:
if a data query instruction sent by a user terminal is received, verifying whether query request parameters in the data query instruction accord with preset rules; if the query request parameters meet the preset rules, traversing the cache component to determine whether a cache result corresponding to the query request parameters exists; and if the buffer memory results exist, feeding back the corresponding buffer memory results to the user terminal.
In a specific implementation, the flow statistics query component 30 detects whether a data query instruction sent by the user terminal is received or not in real time through an interface address of the flow statistics query component, and if not, the query request of the user terminal is refused; if received, the flow statistics query component 30 firstly checks whether query request parameters in the query request accord with preset formats in preset rules, verifies whether contents of the query request parameters accord with content requirements in the preset rules, traverses the cache component to determine whether cache results corresponding to the query request parameters exist, directly reads data in a redis cache and collates the data into structured preset data formats if the cache results exist, and then sends the structured preset data formats to the user terminal; if the query data does not exist in the cache component, traversing the edb database, acquiring statistical data corresponding to the query request parameters from the edb database, sorting the results into a structured preset data format, and then sending the structured preset data format to the user terminal, and simultaneously writing the query result data into the redis cache of the cache component, so that the next query can quickly acquire the result, and the influence on the service use caused by the overlarge request quantity of the database service can be avoided.
It should be noted that the user terminal may be a downstream service of the enterprise department.
For example: the flow statistics query component 30 detects and receives a data query instruction Q sent by a certain navigation user a through an own interface address in real time to query total data of a request flow generated in unit time of a certain freight rate search type, and the like, the flow statistics query component 30 firstly checks whether query request parameters 'the total data of the request flow generated in unit time of the certain freight rate search type' in the query request Q accord with a preset format b in preset rules, verifies whether contents of the query request parameters accord with a content requirement c in the preset rules, and traverses the cache component to determine the total data of the request flow generated in unit time of the certain freight rate search type if accord with the content requirement c, and collates the total data into a structured preset data format, so that the query request parameters are presented to a service department, namely, the user in a simple, visual, easy-to-understand and visual report mode; or directly transmitting the searched statistics to a business department, namely a user.
In the embodiment of the invention, if a data query instruction sent by a user terminal is received, whether query request parameters in the data query instruction accord with preset rules is verified; if the query request parameters meet the preset rules, traversing the cache component to determine whether a cache result corresponding to the query request parameters exists; and if the buffer memory results exist, feeding back the corresponding buffer memory results to the user terminal. The corresponding query result can be queried rapidly, and further the service efficiency is improved.
Referring to fig. 3, a flow chart of a data processing method according to an embodiment of the present invention is applied to the data processing system shown above, where the method includes:
step S301: freight rate search data in a freight rate search engine service is collected.
In the specific implementation process of step S301, the data collection service component collects the freight rate search data of the freight rate search engine service in real time.
Step S302: and processing the freight rate search data to obtain processed detail data.
It should be noted that, the process of step S302 is specifically implemented, including the following steps:
step S11: and extracting service fields in the freight rate search data.
In the specific implementation process of step S11, service fields such as the voyage or agent user Id, the freight rate search service type, and/or the freight rate search result in the freight rate search data are extracted.
It should be noted that the freight rate search service type may be domestic, international, etc., and the freight rate search result may be success, failure, error code, etc.
Step S12: and sorting the service fields according to the format of a preset template to obtain detail data.
In the specific implementation step S12, the collected freight rate search data are sorted according to service fields such as a voyage or agent user Id, freight rate search service types, freight rate search results and the like, and the sorted service fields are written into a preset template according to the format of the preset template to obtain detailed data of freight rate search flow.
It should be noted that, the preset template is set by a technician according to the format of the field, and can be specifically changed according to the actual situation.
Optionally, after the sorted detail data is obtained, the sorted detail data is distributed to the traffic statistics query service through a distributed publish-subscribe message system kafka set in the data communication component.
Step S303: and carrying out statistical processing on the successfully received detail data, and caching the statistical data obtained by processing.
It should be noted that, in the process of implementing step S303 specifically, the method includes the following steps:
step S21: and carrying out statistical processing on the service fields in the detail data to generate a freight rate search flow statistical identification name.
In the process of specifically implementing step S21, the flow statistics query component 30 obtains the detail data of the freight rate search flow through kafka, performs statistics calculation according to the freight rate search service field, and generates a freight rate search flow statistics identification name composed of the navigation or agent user Id, the freight rate search service type, the request result state, and the like.
Step S22: and summarizing based on the freight rate search flow statistics identification names to obtain statistics data.
In the specific implementation process of step S22, the statistical identification name is summarized in the memory database redis of the cache module 40 through the script language lua statistical script according to the freight rate search flow statistics, so as to obtain statistical data.
Step S23: and writing the statistical data into a cache component and a database.
In the specific implementation process of step S23, a statistical data synchronization task is started according to a preset period, and statistical data is synchronized into the edb database for persistent storage.
It should be noted that the preset period is preset by a technician according to the actual situation.
According to the invention, in redis, the statistical results of a plurality of freight rate searching and statistical query services are summarized through the lua script, so that the consistency of distributed data can be ensured, and the statistical calculation efficiency can be improved. Periodically synchronizing the redis summary data to edb, rather than directly summarizing edb, may reduce edb the data writing frequency and improve the data throughput of the overall system.
In the embodiment of the invention, the collected data is processed in advance and then stored, so that the service data of the freight rate search engine service can be transmitted reliably in real time, and the integrity, high availability and consistency of the data are ensured; so that the following user can inquire the corresponding data through the flow statistics inquiry component, and the processing efficiency and the processing accuracy can be improved.
The data processing apparatus according to the embodiment of the present invention is shown in fig. 4, which is another data processing apparatus according to the embodiment of the present invention, and the apparatus includes:
step S401: detecting the network condition of the transmission network, if the network condition is normal, executing step 402, and if the network condition is determined to be abnormal, executing step 403; if it is determined that the network condition is recovered from abnormality to normal, step S404 is executed.
In the specific implementation process of step S401, in the process of sending the detail data to the flow statistics query component through the data communication component, detecting whether the network condition of the transmission network is abnormal in real time, if not, executing step S402, and if determining that the network condition is abnormal, executing step S403; if it is determined that the network condition is recovered from abnormality to normal, step S404 is executed.
Step S402: and sending the detail data to a flow statistics query component through the data communication component.
In the specific implementation process of step S402, it is determined that there is no network reason that causes a failure in data transmission at this time, and the detailed data can be normally sent to the flow statistics query component through the data communication component.
Optionally, after step S402 is performed, the successfully transmitted detail data, i.e. the pushed data amount of the pushed detail data, is counted for the subsequent ledger.
Step S403: and temporarily storing the detail data being transmitted.
In the process of implementing step S403 specifically, if the data transmission fails due to network abnormality or the like, the details of the transmission failure, that is, the traffic data of the push failure, are converted into a structured data format and temporarily stored in the local disk.
It should be noted that, the data format is preset by a technician, for example, may be set to json format.
Step S404: and sending the temporarily stored detail data to the flow statistics query component through a data communication component.
In the specific implementation process of step S404, after the network is recovered, a compensation sending task thread is started, and the freight rate searching flow detail data in the local disk is sent in a timing compensation manner, that is, the temporarily stored detail data of the local disk is sent to the flow statistics query component through the data communication component according to the preset time, so that the integrity of data transmission is ensured.
Optionally, based on the specific implementation process of the step S401 to the step S404, the embodiment of the present invention correspondingly discloses a corresponding architecture, as shown in fig. 5.
In the embodiment of the invention, the collected data is processed in advance, the processed data is transmitted, and the network condition of the transmission network is detected in real time in the process of sending the detail data to the flow statistics query component through the data communication component; if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component; if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted; and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
The data processing apparatus according to the embodiment of the present invention is another data processing apparatus according to the embodiment of the present invention, as shown in fig. 6, and the apparatus includes:
step S601: whether a data query instruction sent by the user terminal is received is determined, if yes, step S602 is executed, and if no, step S601 is executed continuously.
Optionally, the traffic statistics query component registers in the distributed coordinator according to its role, i.e. master-slave role, and the user terminal, i.e. the downstream service, discovers the interface address of the traffic statistics query component through the coordinator registration node, accesses the master-rate search traffic statistics query service, accesses the slave-rate search traffic statistics query service if the master-rate search traffic statistics query service has a fault or is in a busy state, as shown in fig. 7, so as to collect the traffic rate search traffic statistics result from the cache redis through the lua script.
In the specific implementation process of step S601, the flow statistics query component detects in real time whether a data query instruction sent by the user terminal is received through its own interface address, if so, step S602 is executed, and if not, the query request of the user terminal is rejected.
Step S602: and verifying whether the query request parameters in the data query instruction meet the preset rules, if so, executing step S603, and if not, rejecting the query request of the user terminal.
In the specific implementation process of step S602, the flow statistics query component first verifies whether the query request parameters in the query request meet the preset format in the preset rule, verifies whether the content of the query request parameters meet the content requirement in the preset rule, if so, executes step S603, and if not, refuses the query request of the user terminal.
It should be noted that, the preset rule is set by a technician according to the actual situation, and specifically includes a requirement of a preset format, a requirement of content, and the like.
Step S603: and traversing the cache component to determine whether a cache result corresponding to the query request parameter exists, if yes, executing step S604, and if no, traversing edb database to acquire statistical data corresponding to the query request parameter from edb database, and arranging the result into a structured preset data format, and then transmitting to the user terminal, and simultaneously writing the query result data into a redis cache of the cache component, so that the next query can quickly acquire the result, and the influence on the service use caused by the overlarge request quantity of database service can be avoided.
It should be noted that, the preset data format is preset by a technician according to the actual situation, for example, the preset data format may be set as json data format.
Since the data of the query request parameters are stored in advance, there is no case where there is no cache result corresponding to the query request parameters in the edb database.
In the process of implementing step S603, it is first checked, according to the downstream query parameter, whether there is result data conforming to the query parameter in the redis cache, and if there is result data, step S604 is performed. If the query data does not exist in the cache component, traversing the edb database, acquiring statistical data corresponding to the query request parameters from the edb database, sorting the results into a structured preset data format, and then sending the structured preset data format to the user terminal, and simultaneously writing the query result data into a redis cache of the cache component, so that the next query can quickly acquire the result, and the influence on the service use caused by the overlarge request quantity of the database service can be avoided.
Step S604: and feeding back the corresponding caching result to the user terminal.
In the specific implementation process of step S604, the data in the redis buffer is directly read, and is organized into a structured preset data format, and then sent to the user terminal.
Optionally, based on the specific implementation process of the steps S601 to S604, the embodiment of the present invention correspondingly discloses a corresponding architecture, as shown in fig. 8.
Optionally, the first processing manner for cache penetration is to temporarily save the object which is not stored in the edb database in the form of empty object into the cache for a short time, so as to prevent a large amount of empty data requests from penetrating the cache, thereby causing excessive edb service pressure to affect the use; the second processing mode aiming at cache penetration is to distinguish hot spot data, prolong the dead time of the hot spot data in the cache, and simultaneously adopt a locking mode to avoid a great amount of same parameter requests in the same time, and concurrently break down the cache to access a background edb database, thereby causing edb the influence on the use due to overlarge pressure of the database. For cache avalanche, by setting the cache expiration time to have a small random difference, for example, the expiration time is random within 10 minutes to 15 minutes, a large number of caches are prevented from being simultaneously invalidated, and the cache avalanche phenomenon is prevented from occurring, so that a large number of requests bypass the caches, and the database service is not available due to overlarge access amount of the edb database.
In the embodiment of the invention, if a data query instruction sent by a user terminal is received, whether query request parameters in the data query instruction accord with preset rules is verified; if the query request parameters meet the preset rules, traversing the cache component to determine whether a cache result corresponding to the query request parameters exists; and if the buffer memory results exist, feeding back the corresponding buffer memory results to the user terminal. The corresponding query result can be queried rapidly, and further the service efficiency is improved.
The embodiment of the invention also discloses an electronic device which is used for running a database storage process, wherein the data processing method disclosed in the above figures 3 to 6 is executed when the database storage process is run.
The embodiment of the invention also discloses a computer storage medium, which comprises a database storage process, wherein the equipment where the storage medium is controlled to execute the data processing method disclosed in the figures 3 to 6 when the database storage process runs.
In the context of this disclosure, a computer storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing system, wherein the data processing system comprises a data acquisition service component, a data communication component, a flow statistics query component and a cache component;
the data acquisition service component is used for acquiring freight rate search data in freight rate search engine service, and processing the freight rate search data to obtain processed detail data; sending the detail data to the flow statistics query component through the data communication component;
the flow statistics query component is used for acquiring the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into a cache component.
2. The system of claim 1, wherein the data acquisition service component is further configured to:
in the process of sending the detail data to a flow statistics query component through the data communication component, detecting the network condition of a transmission network;
if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component;
if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted;
and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
3. The system according to claim 1, wherein the data acquisition service component for processing the freight rate search data to obtain processed detail data is specifically configured to:
extracting service fields in the freight rate search data;
and sorting the service fields according to the format of a preset template to obtain detail data.
4. The system according to claim 1, wherein the successfully received detail data is statistically processed, and the processed statistical data is written into the traffic statistics query component of the cache component, specifically for:
carrying out statistical processing on service fields in the detail data to generate a freight rate search flow statistical identification name;
summarizing the traffic statistics identification names based on the freight rate searching flow statistics identification names to obtain statistics data;
and writing the statistical data into a cache component and a database.
5. The system of claim 1, wherein the traffic statistics query component is further configured to:
if a data query instruction sent by a user terminal is received, verifying whether query request parameters in the data query instruction accord with preset rules;
if the query request parameters meet the preset rules, traversing the cache component to determine whether a cache result corresponding to the query request parameters exists;
and if the buffer memory results exist, feeding back the corresponding buffer memory results to the user terminal.
6. A data processing method, characterized by being applied to the data processing system shown in claim 1, wherein the data processing system comprises a data acquisition service component, a data communication component, a flow statistics query component and a cache component; the method comprises the following steps:
the data acquisition service component acquires freight rate search data in freight rate search engine service, and processes the freight rate search data to obtain processed detail data; sending the detail data to the flow statistics query component through the data communication component;
the flow statistics query component acquires the detail data which is successfully received and sent by the data communication component; and carrying out statistical processing on the successfully received detail data, and writing the statistical data obtained by processing into a cache component.
7. The method as recited in claim 6, further comprising:
the data acquisition service component detects the network condition of the transmission network;
if the network condition is determined to be abnormal, the detail data is sent to a flow statistics query component through the data communication component;
if the network condition is determined to be abnormal, temporarily storing the detail data which are being transmitted;
and if the network condition is determined to be recovered from the abnormality to be normal, the temporarily stored detail data is sent to the flow statistics query component through the data communication component.
8. The method of claim 6, wherein processing the freight rate search data to obtain processed detail data comprises:
extracting service fields in the freight rate search data;
and sorting the service fields according to the format of a preset template to obtain detail data.
9. An electronic device, characterized in that the electronic device is arranged to run a program, wherein the program when run performs the data processing method according to any of claims 6-8.
10. A computer storage medium, characterized in that the storage medium comprises a stored program, wherein the program, when run, controls a device in which the storage medium is located to perform the data processing method according to any one of claims 6-8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310032793.4A CN116011972A (en) | 2023-01-10 | 2023-01-10 | Data processing system, method, electronic equipment and computer storage medium |
| PCT/CN2023/113492 WO2024148817A1 (en) | 2023-01-10 | 2023-08-17 | Data processing system and method, and electronic device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310032793.4A CN116011972A (en) | 2023-01-10 | 2023-01-10 | Data processing system, method, electronic equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116011972A true CN116011972A (en) | 2023-04-25 |
Family
ID=86037106
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310032793.4A Pending CN116011972A (en) | 2023-01-10 | 2023-01-10 | Data processing system, method, electronic equipment and computer storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116011972A (en) |
| WO (1) | WO2024148817A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024148817A1 (en) * | 2023-01-10 | 2024-07-18 | 中国民航信息网络股份有限公司 | Data processing system and method, and electronic device and computer storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771594B (en) * | 2008-12-31 | 2012-07-18 | 鸿富锦精密工业（深圳）有限公司 | Data storage system and data storage method |
| WO2022067327A1 (en) * | 2020-09-25 | 2022-03-31 | Digital Realty Trust, Inc. | Pervasive data center architecture systems and methods |
| CN112307338A (en) * | 2020-10-30 | 2021-02-02 | 中国民航信息网络股份有限公司 | Flow control method, device, device and storage medium of freight rate search system |
| CN115314423A (en) * | 2022-07-19 | 2022-11-08 | 杭州安恒信息技术股份有限公司 | Traffic data statistical method, device and storage medium |
| CN116011972A (en) * | 2023-01-10 | 2023-04-25 | 中国民航信息网络股份有限公司 | Data processing system, method, electronic equipment and computer storage medium |
-
2023
- 2023-01-10 CN CN202310032793.4A patent/CN116011972A/en active Pending
- 2023-08-17 WO PCT/CN2023/113492 patent/WO2024148817A1/en not_active Ceased
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024148817A1 (en) * | 2023-01-10 | 2024-07-18 | 中国民航信息网络股份有限公司 | Data processing system and method, and electronic device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024148817A1 (en) | 2024-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240886B (en) | Exception handling method, exception handling device, computer equipment and storage medium | |
| CN113190423B (en) | Method, device and system for monitoring service data | |
| CN111030989B (en) | Flight operation data message analysis system and method | |
| CN110544164A (en) | Full link account checking method and system | |
| CN101188523A (en) | Generation method and generation system of alarm association rules | |
| CN101692668A (en) | Device and method for automatically downloading, resolving and storing aviation meteorological data | |
| CN113472881B (en) | Statistical method and device for online terminal equipment | |
| CN113392153A (en) | Data synchronization method and device, electronic equipment and computer storage medium | |
| CN116126552A (en) | Mass meteorological observation data processing method and device based on Storm | |
| CN111680104A (en) | Data synchronization method and device, computer equipment and readable storage medium | |
| CN116011972A (en) | Data processing system, method, electronic equipment and computer storage medium | |
| CN112860780B (en) | Data export method and device and terminal equipment | |
| CN119415143A (en) | Software configuration method and device | |
| CN103678423A (en) | Data file input system, device and method | |
| CN110689386B (en) | Electronic bill issuing method and system | |
| CN111061802A (en) | Power data management processing method and device and storage medium | |
| CN114020819B (en) | A method and device for synchronizing multi-system parameters | |
| CN115269424A (en) | Automatic regression testing method, device, equipment and storage medium for production flow | |
| CN117609315B (en) | Data processing method, device, equipment and readable storage medium | |
| CN110782380B (en) | Aviation change information management method and device and storage medium | |
| CN110888785A (en) | Method and device for monitoring alarm | |
| CN119961307A (en) | A Strategy Method for External Data Query | |
| CN111831697A (en) | Method and device for realizing verification based on JVM (Java virtual machine), electronic equipment and storage medium | |
| CN115473858B (en) | Data transmission method, stream data transmission system, computer device and storage medium | |
| CN118427260A (en) | Vehicle data processing method and device, vehicle and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |