[go: up one dir, main page]

WO2017067415A1 - 数据传输方法及装置 - Google Patents

数据传输方法及装置 Download PDF

Info

Publication number
WO2017067415A1
WO2017067415A1 PCT/CN2016/101981 CN2016101981W WO2017067415A1 WO 2017067415 A1 WO2017067415 A1 WO 2017067415A1 CN 2016101981 W CN2016101981 W CN 2016101981W WO 2017067415 A1 WO2017067415 A1 WO 2017067415A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
timestamp
value
fragment data
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/101981
Other languages
English (en)
French (fr)
Inventor
程峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to JP2018520613A priority Critical patent/JP2018536226A/ja
Priority to EP16856846.7A priority patent/EP3367634B1/en
Publication of WO2017067415A1 publication Critical patent/WO2017067415A1/zh
Priority to US15/959,047 priority patent/US10735295B2/en
Anticipated expiration legal-status Critical
Priority to US16/911,129 priority patent/US11201810B2/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W56/00—Synchronisation arrangements
    • H04W56/001—Synchronization between nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
    • H04L43/106—Active monitoring, e.g. heartbeat, ping or trace-route using time related information in packets, e.g. by adding timestamps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L61/00—Network arrangements, protocols or services for addressing or naming
    • H04L61/45—Network directories; Name-to-address mapping
    • H04L61/4505—Network directories; Name-to-address mapping using standardised directories; using standardised directory access protocols
    • H04L61/4511—Network directories; Name-to-address mapping using standardised directories; using standardised directory access protocols using domain name system [DNS]

Definitions

  • the present invention relates to the field of Internet technologies, and in particular, to a data transmission method and apparatus.
  • Some lawless elements may use the network platform to release information content that is harmful to society, such as pornography, gambling, violence, and terrorism, which will adversely affect society.
  • Network content may need to be monitored. When these unfavorable information is found, it can be shielded to minimize the spread of the unfavorable information.
  • a network content monitoring system generally includes two systems (which can be represented as two devices), for example, system A and system B, and system B needs to synchronize data in system A.
  • system A and system B each synchronization All data information needs to be synchronized, which may cause database read timeout, or system A may be caused when system A and system B are in different network environments, system A provides too much data, and system B memory is used too much. System A full-time update data frequently fails.
  • Embodiments of the present invention provide a data transmission method and apparatus to at least solve the technical problem of frequent failures when performing full amount of data update between systems.
  • a data transmission method including: a first device periodically receiving a request message from a second device, where the request message carries a time corresponding to the currently received request message
  • the first device determines whether the value of the timestamp is less than a preset threshold; when the value of the timestamp is less than the preset threshold, the correspondence between the timestamp and the fragmented data of the first device locally cached by the first device,
  • the fragment data corresponding to the time stamp is synchronized to the second device, wherein the fragment data is obtained by fragmenting the full amount of data recorded by the first device.
  • a data transmission method including: the second device sends a request message to the first device, where the request message carries a timestamp corresponding to the request message; If the value of the stamp is less than the preset threshold, the second device receives the fragment data sent by the first device;
  • the data is fragment data that is locally cached by the first device, and the fragment data is obtained by fragmenting the full amount of data recorded by the first device.
  • a data transmission apparatus is further provided, which is applied to a first device, and includes: a receiving module, configured to periodically receive a request message from a second device, where the request message carries a timestamp corresponding to the received request message; the first determining module is configured to determine whether the value of the timestamp is less than a preset threshold; and the first synchronization module is configured to: when the value of the timestamp is less than a preset threshold, Synchronizing the fragment data corresponding to the timestamp to the second device according to the correspondence between the timestamp and the fragmented data of the first device in the local cache, wherein the fragmented data is fragmented for the full amount of data recorded by the first device. get.
  • a data transmission apparatus is further provided, which is applied to a second device, and includes: a sending module, configured to send a request message to the first device, where the request message carries a request and a request a timestamp corresponding to the message; the first receiving module is configured to receive the fragment data sent by the first device if the value of the timestamp is less than a preset threshold; wherein the fragment data is corresponding to the timestamp
  • the fragment data of the device is locally cached, and the fragment data is obtained by fragmenting the full amount of data recorded by the first device.
  • the first device is used to synchronize the locally cached fragment data to the second device, where the fragment data is the full amount of data recorded by the first device.
  • the manner in which the fragmentation is obtained by means of buffering the full amount of data, and then sequentially synchronizing the fragmented data in the cache corresponding to the timestamp to the second device according to the timestamp in the received request message. It reduces the pressure of data reading, and solves the technical problem of frequent failures when updating data in full between systems, and improves the efficiency of data transmission.
  • FIG. 1 is a block diagram showing the hardware structure of a computer terminal of a data transmission method according to an embodiment of the present invention
  • FIG. 2 is a flowchart 1 of a data transmission method according to Embodiment 1 of the present invention.
  • FIG. 3 is a second flowchart of a data transmission method according to Embodiment 1 of the present invention.
  • FIG. 4 is a schematic flowchart of fragmenting a full amount of data of a first device according to an alternative embodiment of the present invention
  • FIG. 5 is a schematic flowchart diagram of a data transmission method according to an alternative embodiment of the present invention.
  • FIG. 6 is a flowchart 1 of a data transmission method according to Embodiment 2 of the present invention.
  • FIG. 7 is a second flowchart of a data transmission method according to Embodiment 2 of the present invention.
  • Figure 8 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 3 of the present invention.
  • FIG. 9 is a block diagram 2 of a structure of a data transmission apparatus according to Embodiment 3 of the present invention.
  • FIG. 10 is a block diagram 3 of a structure of a data transmission apparatus according to Embodiment 3 of the present invention.
  • FIG. 11 is a block diagram 4 of a structure of a data transmission apparatus according to Embodiment 3 of the present invention.
  • Figure 12 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 4 of the present invention.
  • FIG. 13 is a block diagram 2 of a structure of a data transmission apparatus according to Embodiment 4 of the present invention.
  • FIG. 14 is a block diagram 3 of a structure of a data transmission apparatus according to Embodiment 4 of the present invention.
  • FIG. 15 is a structural block diagram of a computer terminal according to an embodiment of the present invention.
  • the uniform resource locator url/domain name blocking record is read from the first device for the first time;
  • the read resource locator url/domain name blocking record is a full-time update mode, and the request sent by the second device is time-stamped, and the value is 0;
  • the first device After receiving the request with the timestamp parameter value of 0, the first device will read out all the url/domain name blocking records in the current database, and add the current time value (accurate to the minute), and return it to the second. device;
  • the second device After receiving the full url/domain name blocking record, the second device clears all blocking records in the system, and uses the received full url/domain name to block the record;
  • the second device determines whether the current time is a whole point or a half point. If yes, execute 2; if not, send a request to the first device, and the time stamped parameter is the time value returned by the first device in 3;
  • the first device After receiving the request with the timestamp parameter value not 0, the first device will read out the change data in the database from the specified time of the timestamp to the current time, and add the current time value (accurate to the minute). Return to the second device;
  • the second device After receiving the incremental url/domain name blocking record, the second device updates all blocked records in the system; then executes 5.
  • the url/domain name blocking data of the first device When the url/domain name blocking data of the first device reaches the tens of millions of levels, the following problem occurs: when the first device reads the url/domain name blocking data from the database in real time, the read database times out; when the first device In a different network environment with the second device, the network link is unstable; the url/domain name blocking record of 10 million levels will prolong the data transmission time, often causing the full data transmission failure; the second device server has limited memory when it uses its own memory. When there are more, the full url/domain name blocking record is swapped in and out, often causing a memory alarm.
  • Incremental synchronization mode System A maintains a batch of real-time changing data, System B needs to use the batch of data maintained by System A; when System B accesses System A to read data, System A only accesses System B last time and this time. The amount of data change between accesses is sent to System B. This method of data synchronization is called incremental synchronization.
  • Full-scale synchronization mode System A maintains a batch of real-time data changes, System B needs to use the batch of data maintained by System A; when System B accesses System A to read data, System A sends all current data to System B.
  • the data synchronization method is called full-quantity synchronization.
  • the embodiment of the invention optimizes the full update process of the blocking records of the first device and the second device, and uses the incremental synchronization mode to simulate the full amount synchronization mode to avoid the above problems.
  • the details are described below in conjunction with specific embodiments.
  • FIG. 1 is a block diagram showing the hardware structure of a computer terminal of a data transmission method according to an embodiment of the present invention.
  • computer terminal 10 may include one or more (only one shown) processor 102 (processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) a memory 104 for storing data, and a transmission module for communication functions 106.
  • processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA)
  • memory 104 for storing data
  • a transmission module for communication functions 106 a transmission module for communication functions 106.
  • FIG. 1 is merely illustrative and does not limit the structure of the above electronic device.
  • computer terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a different configuration than that shown in FIG.
  • the memory 104 can be used to store software programs and modules of application software, such as program instructions/modules corresponding to the data transfer method in the embodiment of the present invention, and the processor 102 executes various programs by running software programs and modules stored in the memory 104.
  • Functional application and data processing that is, a data transmission method for implementing the above application.
  • Memory 104 may include high speed random access memory, and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
  • memory 104 may further include memory remotely located relative to processor 102, which may be coupled to computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the transmission module 106 is configured to receive or transmit data via a network.
  • the network specific examples described above may include a wireless network provided by a communication provider of the computer terminal 10.
  • the transport module 106 includes a Network Interface Controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet.
  • the transmission module 106 can be a Radio Frequency (RF) module for communicating with the Internet wirelessly.
  • NIC Network Interface Controller
  • RF Radio Frequency
  • FIG. 2 is a flowchart 1 of a data transmission method according to Embodiment 1 of the present invention. As shown in FIG. 2, the method includes:
  • Step S202 the first device periodically receives a request message from the second device, where the request message carries a timestamp corresponding to the request message received this time;
  • the first device may be the computer terminal 10 in the above-mentioned operating environment, and the function of the first device is completed by the computer terminal 10 to implement the data transmission method in this embodiment.
  • the timing of periodically receiving the request message from the second device may be set according to actual conditions, and the time interval of the two received request messages may be fixed, for example, setting a request every other minute.
  • the message may also be unfixed.
  • the second request message is received after one minute, and the third request message is received after three minutes.
  • the specific information may be set according to actual conditions.
  • Step S204 The first device determines whether the value of the timestamp is less than a preset threshold.
  • first device and the second device may be represented by two physical entities, and may also be represented by two applications, but are not limited thereto.
  • the value of the above timestamp may be a simple serial number such as an Arabic numeral, or may be a time value, but is not limited thereto.
  • the value of the timestamp may be a specified value for indicating that the full amount of data in the first device is read.
  • the specified value may be 0, that is, the timestamp carried in the request message.
  • a value of 0 indicates that the second device should read the full amount of data of the first device.
  • step S204 it is required to determine whether the value of the timestamp is the specified value, and when the determination result is yes, trigger to synchronize the fragment data corresponding to the timestamp to the Second device.
  • the first device may be triggered to start synchronizing the first fragment data in the full amount of data to the second device.
  • Step S206 When the value of the timestamp is less than the preset threshold, the first device synchronizes the fragment data corresponding to the timestamp to the second device according to the correspondence between the timestamp and the fragmented data of the first device in the local cache.
  • the fragmentation data is obtained by fragmenting the full amount of data recorded by the first device.
  • the preset threshold may be the number of fragments obtained after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10;
  • the blocking record of the uniform resource locator url may also be a blocking record for the domain name, or a combination of the two, and is not limited thereto.
  • the fragment data may be obtained in the first device by: writing the fragment data corresponding to the first index n in the database of the first device to the local cache, and using n-1 as the fragment data in the first An index in the local cache of the device, where n is less than or equal to the total number of all the fragmented data N, n, and N obtained by fragmenting the full amount of data are positive integers.
  • the fragment data corresponding to the first index n in the database of the first device is written into the local cache, and n-1 is used as an index of the fragment data in the local cache of the first device.
  • the first device determines whether n is less than N; in the case that n is less than N, the first device reads the fragment data corresponding to n from the database, and attaches a value of n to the fragment data corresponding to n. a timestamp; in the case where n is equal to N, the first device reads the fragment data corresponding to n from the database, and attaches a timestamp value of the specified time to the fragment data corresponding to n; wherein The specified time is the time when the second device accesses the first device last time; the first device writes the fragment data corresponding to n to the local cache, and uses n-1 as an index of the fragment data in the local cache.
  • the manner of performing caching may be performing fragment caching according to the sending format. Therefore, when the second device needs to read, the first device does not need to perform format conversion on the fragment data, but directly returns to the first The second device reduces the synchronization time and improves the efficiency of synchronization.
  • each fragmented data cached in the cache has a corresponding index, and the index and the request message sent by the second device.
  • the timestamp in the file has a certain correspondence, and the full amount of data is divided by the correspondence between the timestamp and the fragmented data in the cache.
  • the slice data of the slice is synchronized to the second device.
  • the synchronization of the slice data corresponding to the timestamp to the second device in step S206 may be performed as follows: The value is used as an index to synchronize the specified fragment data corresponding to the index in the local cache of the first device to the second device.
  • the method of synchronizing data to the second device reduces the pressure of data reading, thereby solving the technical problem of frequent failures when updating data in full between the systems, and improving the efficiency of data transmission.
  • a data record of tens of millions of levels is usually recorded in the database of the first device.
  • the data record with less data records 80 records
  • the first device synchronizes all 80 records to the second device at one time.
  • the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces.
  • the fragmented data is sequentially synchronized to the second device by segmenting the full amount of data, that is, the full amount of data is synchronously synchronized to the second device, and the data reading pressure can be reduced relative to the scheme in the related art. , shorten the data transmission time and improve the data transmission efficiency.
  • FIG. 3 is a second flowchart of a data transmission method according to Embodiment 1 of the present invention. As shown in FIG. 3, the foregoing method further includes:
  • Step S302 When the value of the timestamp is equal to or greater than a preset threshold, the first device synchronizes the incremental data generated from the time indicated by the timestamp to the current time to the second device.
  • the first type of timestamps may be the timestamp in the embodiment shown in FIG. 2, and the timestamp may be used to read the points in the full amount of data.
  • the role of the index identification of the slice data which may be represented by a one-digit number such as 0, 1, 2, etc.
  • the second type of timestamp may be a real timestamp, which represents a real time, that is, the last access of the second device
  • the time of the first device which can be represented by a 9-digit number, that is, the second type of timestamp can both serve as an identifier or a real time.
  • the value of the second type of timestamp indicating the real timestamp is generally greater than the preset threshold (ie, the timestamp equal to or greater than the preset threshold in step S302), and thus the second type of timestamp can be used as a full amount.
  • step S302 the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request, and the second device needs to be accessed last time.
  • the incremental data between the time of the first device (ie, the time indicated by the time stamp described above) to the current time is transmitted to the second device, and the first device keeps the incremental data in the normal incremental synchronization manner in the subsequent time. Synchronizing to the second device until the point in time is an hour or a half point, that is, until an indication that the first device is synchronizing data to the second device in a full amount of synchronization is received, and so on.
  • the data communication between the first device and the second device may use the http protocol, but is not limited thereto.
  • the above method can be applied to the field of monitoring and shielding of network content.
  • the first device monitors the content of the network, such as yellow, poison, violence, and terrorism, or other devices monitor the content when monitoring.
  • the uniform resource locator of the webpage corresponding to the content or the domain name of the computer where the content appears is recorded, and then the data is synchronized to the second device by using the method of the embodiment of the present invention, and the second device can be further
  • the content is transmitted to other devices or the second device itself shields the content according to the obtained uniform resource locator or domain name.
  • the first device performs a fragmentation cache on the full url/domain name in the database of the first device every 30 minutes to solve the problem of read timeout caused by real-time reading of the record from the database.
  • FIG. 4 is a schematic flowchart of a first device fragmenting a full amount of data according to an alternative embodiment of the present invention. As shown in FIG. 4, the method includes the following steps:
  • Step S402 setting the number of fragments (for example, 4) of the full-quantity url/domain name blocking record, the number of fragments is less than 10;
  • Step S404 reading the current time, accurate to the minute, recorded in the timestamp
  • Step S406 read the total number of records in the current time database (for example, 81);
  • Step S408 calculating the number of records (for example, 20, 20, 20, 21) included in each slice of the full-quantity record according to the number of slices;
  • Step S410 setting a slice count index (that is, the first index n in the above embodiment), the value is 1;
  • Step S412 determining whether the current index value is smaller than the number of fragments (corresponding to the total number N of fragment data in the above embodiment); if (for example, 1), executing step S414; if not (for example, 4), executing step S422 ;
  • Step S414 reading data included in the specified fragment corresponding to the index from the database;
  • Step S420 the slice count index is increased by 1; returning to step S412 to execute;
  • Step S422 reading data included in the specified fragment corresponding to the index from the database
  • Step S424 attaching a timestamp, the value is timestamp (corresponding to the specified time in the above embodiment);
  • step S426 the fragment data and the time stamp are written into the cache, and the key value is index-1.
  • FIG. 5 is a schematic flowchart of a data transmission method according to an alternative embodiment of the present invention. As shown in FIG. 5, taking a 4-chip cache as an example, the method includes the following steps:
  • Step S501 the second device initiates a request to the first device, and reads the full amount of the blocked record, and the timestamp is 0;
  • Step S502 the first device determines that the timestamp is 0, is a full-time update blocking record, and reads the fragment data in the first slice cache in the cache of the full-blocking record with 0 as the key, and the timestamp is 1, Return to the second device;
  • Step S503 the second device clears the blocking record in the system, and uses the received full amount to block the fragmentation data in the first slice of the record;
  • Step S504 after 1 minute, the second device initiates a request to the first device, and reads the incremental blocking record, and the timestamp is 1 returned by the first device in step S502;
  • Step S505 the first device determines that the timestamp is less than 10, which is an uncompleted full-quantity update blocking record, and reads the fragment data in the second slice cache in the cache of the full-blocking record with 1 as the key, with timestamp 2, Return to the second device;
  • Step S506 the second device adds the fragmentation data in the received second slice buffer to the existing blocking record.
  • Step S507 after 1 minute, the second device initiates a request to the first device, and reads the incremental blocking record, and the timestamp is 2 returned by the first device in step S505;
  • Step S508 the first device determines that the timestamp is less than 10, which is an uncompleted full-quantity update blocking record, and reads the fragment data in the third-chip cache in the cache of the full-blocking record with 2 as the key, with timestamp 3, Return to the second device;
  • Step S509 the second device adds the fragmentation data in the received third slice buffer to the existing blocking record.
  • Step S510 after 1 minute, the second device initiates a request to the first device, and reads the incremental blocking record, the timestamp. 3 returned for the first device in step S508;
  • Step S511 the first device determines that the timestamp is less than 10, which is an uncompleted full-quantity update blocking record, and reads the fragment data in the fourth-chip cache in the buffer of the full-blocking record with the key of 3, with the timestamp being the full amount. Blocking the generation time of the record cache timestamp, returning to the second device;
  • Step S512 the second device adds the fragmentation data in the received fourth slice buffer to the existing blocking record.
  • Step S514 the first device Mrm determines that the timestamp is greater than 10, is a normal incremental blocking record request, reads the amount of the blocked record from the timestamp in the database to the current time, with the current time, and returns to the second device;
  • the second device performs an incremental blocking record request; if it is an hour or a half point, the second device does not perform an incremental blocking record request, but returns to step S501.
  • 1 minute in the optional embodiment shown in FIG. 5 of the present invention corresponds to the timing of the embodiment shown in FIG. 2 of the above embodiment, and the time stamp 0 is equivalent to the embodiment in the embodiment shown in FIG. 2 .
  • the value of the timestamp is a specified value to indicate that the full-time blocking record is read.
  • the timestamps in the above embodiment are 0, 1, 2, and 3, which are equivalent to the values of the timestamps in the embodiment shown in FIG. 2;
  • the timestamp is equivalent to the timestamp in which the value in the embodiment shown in FIG. 3 is equal to or greater than a preset threshold.
  • the synchronization is divided into two types, an incremental synchronization mode and a full-quantity synchronization mode, and the full-quantity synchronization mode synchronization data is implemented by: pre-scaling the full amount of data in advance and performing the reading on the second device.
  • the full-scale data synchronization is realized by N analog incremental synchronization methods, thereby reducing the data reading pressure, first in the first device.
  • the second synchronization is given to the second device, the synchronization fails because the amount of data is too large.
  • the data transmission method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases The former is a better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, can be embodied in the form of a software product.
  • Stored in a storage medium such as ROM/RAM, disk, optical disk
  • a terminal device which may be a mobile phone, a computer, a server, or a network device, etc.
  • FIG. 6 is a flowchart 1 of a data transmission method according to Embodiment 2 of the present invention. As shown in FIG. 6, the method includes:
  • Step S602 The second device sends a request message to the first device, where the request message carries a timestamp corresponding to the request message.
  • the second device may be the same computer terminal as the computer terminal 10 in the first embodiment, and the data transmission method in the embodiment is completed in the environment of the computer terminal, but is not limited thereto.
  • the foregoing sending request message may be a timed sending request message, and may be an unscheduled sending request message, and may be specifically set according to a specific situation.
  • Step S604 the second device receives the fragment data sent by the first device, where the value of the timestamp is less than the preset threshold, where the fragment data is cached locally on the first device corresponding to the timestamp.
  • the fragmentation data is obtained by fragmenting the full amount of data recorded by the first device.
  • the value of the timestamp may be a specified value, and is used to indicate that the second device reads the full amount of data in the first device.
  • the specified value may be 0, that is, carried in the request message. When the timestamp is 0, it indicates that the second device should read the full amount of data of the first device.
  • the method before the step S604, the method further includes: when the timestamp is a specified value, the second device triggers the first device to send the full amount of data, and the first device sends the corresponding value corresponding to the specified value. Fragmentation data.
  • the preset threshold may be the number of fragments obtained by the first device after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10; It may be a blocking record for the uniform resource locator url, a blocking record for the domain name, or a combination of the two, and is not limited thereto.
  • the first device performs fragmentation buffering on the full amount of data, that is, how the fragment data is obtained in the first device, which is the same as that in the first embodiment, and details are not described herein again.
  • the receiving, by the second device, the fragment data sent by the first device may be: the second device receives the specified fragment data corresponding to the index in the local cache of the first device; wherein the index is the timestamp; The second device receives the specified timestamp sent by the first device, where the value of the specified timestamp is obtained by adding 1 to the value of the timestamp.
  • the method for sending the time-stamped request message by the second device, and receiving, by the second device, the manner of sending the fragmented data in the cache of the first device corresponding to the timestamp according to the timestamp in the received request message That is, the second device reads the fragment data of the fragment cache in advance in the first device according to the time stamp, thereby reducing the pressure of data reading, thereby solving the technical problem of frequently failing to update the data in full during the system, and improving the technical problem.
  • the efficiency of data transmission is a method for sending the time-stamped request message by the second device, and receiving, by the second device, the manner of sending the fragmented data in the cache of the first device corresponding to the timestamp according to the timestamp in the received request message, That is, the second device reads the fragment data of the fragment cache in advance in the first device according to the time stamp, thereby reducing the pressure of data reading, thereby solving the technical problem of frequently failing to update the data in full during the system, and improving the technical problem.
  • a tens of millions of data records are usually recorded in the database of the first device.
  • a lesser data record 80 records
  • the second device sends a request carrying a timestamp of 0.
  • the second device will read the 80 records at a time.
  • the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces.
  • the full amount of data is fragmented by the first device, and the second device reads the fragment data in the cache of the first device in batches, which can reduce the reading pressure of the data compared to the scheme in the related art. Shorten the data transmission time and improve the data transmission efficiency.
  • FIG. 7 is a flowchart 2 of a data transmission method according to Embodiment 2 of the present invention. As shown in FIG. 7, the method further includes:
  • Step S702 If the value of the timestamp is equal to or greater than the preset threshold, the second device receives the incremental data generated by the first device from the time indicated by the timestamp to the current time.
  • the first type of timestamp may be the timestamp in the foregoing embodiment shown in FIG. 6, and the timestamp may be used to read the points in the full amount of data.
  • the role of the index identification of the slice data which may be represented by a one-digit number such as 0, 1, 2, etc.
  • the second type of timestamp may be a real timestamp, which represents a real time, that is, the last access of the second device
  • the time of the first device which can be represented by a 9-digit number, that is, the second type of timestamp can both serve as an identifier or a real time.
  • the preset threshold is generally fragmented data, it is not a large number, so the second type of timestamp, which can represent the real timestamp with 9 digits, is generally greater than the preset threshold (ie, in step S702). a timestamp equal to or greater than a preset threshold, so the second type of timestamp can be used as the end of a full data read Knowledge, or a trigger for a normal incremental data read.
  • step S702 the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request, and the second device needs to use the second device.
  • the incremental data between the time when the last access to the first device (ie, the time indicated by the timestamp described above) is up to the current time is read, and in the subsequent time, the second device is always read in the normal incremental synchronization manner. Incremental data until the point in time is an hour or a half point, that is, until the second device receives an indication to read the data in full synchronous mode, and so on.
  • the data communication between the first device and the second device may use the http protocol, but is not limited thereto.
  • the above method can be applied to the field of monitoring and shielding of network content.
  • the first device monitors the content of the network such as yellow, poison, violence, and terrorism, or other devices monitor the content.
  • the uniform resource locator of the webpage corresponding to the content or the domain name of the computer where the content is generated is recorded, and then the data is synchronized to the second device by using the method of the embodiment of the present invention.
  • the content is transmitted to other devices or the second device itself shields the content according to the obtained uniform resource locator or domain name.
  • FIG. 8 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 3 of the present invention, as shown in FIG. The device is applied to the first device, including:
  • the receiving module 80 is configured to receive a request message from the second device, where the request message carries a timestamp corresponding to the request message received this time;
  • timing here may be set according to actual conditions, and the time interval of the two received request messages may be fixed, for example, setting the request message every 1 minute, or may be unfixed. For example, after 1 minute, the second request message is received, and after 3 minutes, the third request message is received, and the specific request can be set according to the actual situation.
  • the first determining module 82 is connected to the receiving module 80, and is configured to determine whether the value of the timestamp is less than a preset threshold;
  • the value of the timestamp may be a specified value, which is used to indicate that the full amount of data in the first device is read.
  • the specified value may be 0, that is, the timestamp carried in the request message is 0, indicating the second The device should read the full amount of data for the first device.
  • FIG. 9 is a structural block diagram of a data transmission apparatus according to Embodiment 3 of the present invention. As shown in FIG. 9, the apparatus further includes:
  • the second determining module 90 is connected to the receiving module 80, and is configured to determine whether the value of the timestamp is a specified value, wherein the specified value is used to indicate that the full amount of data is read, wherein when the determining result is yes, the triggering is performed.
  • the fragment data corresponding to the time stamp is synchronized to the second device.
  • the first synchronization module 84 is connected to the first determining module 82, and is configured to: when the value of the timestamp is less than the preset threshold, according to the correspondence between the timestamp and the fragmented data of the first device in the local cache, The fragmented data corresponding to the stamp is synchronized to the second device, wherein the fragmented data is obtained by fragmenting the full amount of data recorded by the first device.
  • the preset threshold may be the number of fragments obtained after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10;
  • the blocking record of the uniform resource locator url may also be a blocking record for the domain name, or a combination of the two, and is not limited thereto.
  • the first synchronization module 84 is further configured to synchronize the specified fragment data corresponding to the index in the local cache of the first device to the second device by using the value of the timestamp as an index.
  • FIG. 10 is a structural block diagram 3 of a data transmission apparatus according to Embodiment 3 of the present invention. As shown in FIG. 10, the apparatus further includes:
  • the cache module 100 is connected to the first synchronization module 84, and is configured to write the fragment data corresponding to the first index n in the database of the first device into the local cache, and use n-1 as the fragment data in the first device.
  • the index in the local cache, where n is less than or equal to the total number of all the fragmented data N, n, and N obtained by fragmenting the full amount of data are positive integers.
  • the cache module 100 further includes:
  • the determining unit 1001 is configured to determine whether n is less than N;
  • the first reading unit 1002 is connected to the determining unit 1001, and is configured to read the fragment data corresponding to n from the database when n is smaller than N, and attach a value to the fragment data corresponding to n. a timestamp of n;
  • the second reading unit 1003 is connected to the determining unit 1001, and is configured to read the fragment data corresponding to n from the database when n is equal to N, and attach the fragment data corresponding to n
  • the upper value is a timestamp of the specified time; wherein the specified time is a time when the second device accesses the first device last time;
  • the buffer unit 1004 is connected to the first reading unit 1002 and the second reading unit 1003, and is configured to write the fragment data corresponding to n to the local cache, and use n-1 as the fragment data in the local cache. index.
  • the buffer unit 1004 may write the fragment data corresponding to n into the local cache according to the sending format, so that the first device does not need to format the fragment data when the second device needs to read. The change is directly returned to the second device, reducing the synchronization time and improving the efficiency of the synchronization.
  • the method of synchronizing data to the second device reduces the pressure of data reading, thereby solving the technical problem of frequent failures when updating data in full between the systems, and improving the efficiency of data transmission.
  • a data record of tens of millions of levels is usually recorded in the database of the first device.
  • the data record with less data records 80 records
  • the first device synchronizes all 80 records to the second device at one time.
  • the device in the first device fragments the 80 records, and the fragments can be equally distributed or distributed unevenly.
  • the equal distribution is divided into four pieces, each of which is divided into four pieces.
  • the slice has 20 records, the buffer unit 1004 writes the data contained in each slice into the cache, and sets the index 0, 1, 2, 3 corresponding to the 4 pieces of data in the cache; when the second device requests to read
  • the first time synchronization module 84 of the first device synchronizes the fragmented data indexed by 0 to the second device in the first device. (For example, 1 minute), the receiving module 80 of the first device receives the second request, and the second request carries the timestamp of 1, and the first synchronization module 84 of the first device indexes the index in the cache by 1.
  • the slice data is synchronized to the second device; and so on until the sliced data in the cache is read.
  • the above apparatus of the embodiment reduces the data by sequentially synchronizing the fragmented data to the second device by sharding the full amount of data, that is, synchronizing the full amount of data to the second device in batches, which can reduce the data compared to the scheme in the related art.
  • the reading pressure shortens the data transmission time and improves the data transmission efficiency.
  • FIG 11 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 3 of the present invention. As shown in Figure 11, the apparatus further includes:
  • the second synchronization module 1101 is connected to the first determining module 82, and is configured to synchronize the incremental data generated from the time indicated by the timestamp to the current time to the second when the value of the timestamp is equal to or greater than the preset threshold. device.
  • the first type of timestamps may be the timestamp in the foregoing embodiment shown in FIG. 8, and the timestamp may be used to read the points in the full amount of data.
  • the role of the index identification of the slice data which may be represented by a one-digit number such as 0, 1, 2, etc.
  • the second type of timestamp may be a real timestamp, which represents a real time, that is, the last access of the second device
  • the time of the first device which can be represented by a 9-digit number, that is, the second type of timestamp can both serve as an identifier or a real time.
  • the preset threshold is generally fragmented data, it may not be a large number, and thus the value of the second type of timestamp, which can represent the real timestamp with 9 digits, is generally greater than the preset threshold. Therefore, the second type of time stamp can be used as an identifier for the end of a full amount of data reading, or a trigger identifier for a normal incremental data reading.
  • the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request.
  • the second synchronization module 1101 needs to last the second device.
  • the incremental data between the time when the first device is accessed (ie, the time indicated by the timestamp described above) to the current time is transmitted to the second device, and the first device is always in the normal incremental data synchronization manner in the subsequent time.
  • the quantity data is synchronized to the second device until the point in time is an hour or a half point, that is, until an indication that the first device synchronizes the data to the second device in a full amount of read synchronization mode is received, and so on.
  • the data communication between the foregoing device and the second device in the first device may use the http protocol, but is not limited thereto.
  • the above device can be applied to the field of monitoring and shielding of network content.
  • the first device monitors content of the network such as yellow, poison, violence, and terrorism, or other devices monitor the content.
  • the uniform resource locator of the webpage corresponding to the content or the domain name of the computer where the content appears is recorded, and then the data is synchronized to the second device by using the device, and the second device can be transmitted to the other device.
  • the device or the second device itself shields the content according to the obtained uniform resource locator or domain name.
  • the second synchronization module 1101 of the first synchronization module 84 may be the same synchronization module in the first device, or may be a different synchronization module in the first device.
  • FIG. 12 is a block diagram of a structure of a data transmission apparatus according to Embodiment 4 of the present invention, as shown in FIG. The device is applied to the second device, including:
  • the sending module 1202 is configured to send a request message to the first device, where the request message carries a timestamp corresponding to the request message;
  • the foregoing sending request message may be a timed sending request message, and may be an unscheduled sending request message, and may be specifically set according to a specific situation.
  • the first receiving module 1204 is connected to the sending module 1202, and configured to receive the fragment data sent by the first device if the value of the timestamp is less than a preset threshold; wherein the fragment data is corresponding to the timestamp.
  • the fragment data that is locally cached by the first device is fragmented by the full amount of data recorded by the first device.
  • the timestamp may be a specified value, and is used to indicate that the second device reads the full amount of data in the first device.
  • the specified value may be 0, that is, the timestamp carried in the request message is 0, indicating that the second device should read the full amount of data of the first device.
  • the foregoing apparatus further includes: a triggering module, configured to: when the value of the timestamp is a specified value, trigger the first device to send the full amount of data; and the third receiving module is connected to the triggering module, And configured to receive the fragment data corresponding to the specified value sent by the first device, where the specified value is used to instruct the second device to read the full amount of data to the first device.
  • the preset threshold may be the number of fragments obtained by the first device after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10; It may be a blocking record for the uniform resource locator url, a blocking record for the domain name, or a combination of the two, and is not limited thereto.
  • the first device performs fragmentation buffering on the full amount of data, that is, how the fragment data is obtained in the first device, which is the same as that in the first embodiment, and details are not described herein again.
  • FIG. 13 is a block diagram of a structure of a data transmission apparatus according to Embodiment 4 of the present invention.
  • the first receiving module 1204 may include: a first receiving unit 1302. For receiving the specified fragment data corresponding to the index in the local cache of the first device; wherein the index is the value of the timestamp; the second receiving unit 1304 is connected to the first receiving unit 1302, and configured to Receiving a specified timestamp sent by the first device, where the value of the specified timestamp is obtained by adding 1 to the timestamp.
  • the sending module 1202 sends a request message carrying a timestamp through the sending module 1202, and the first receiving module 1204 receives the timestamp in the cache of the first device corresponding to the timestamp sent by the first device according to the timestamp in the received request message.
  • the method of the slice data that is, the device reads the fragment data of the slice buffer in advance in the first device according to the time stamp, thereby reducing the pressure of data reading, thereby solving the frequent failure of the system to update the data in full.
  • Technical issues have improved the efficiency of data transmission.
  • a tens of millions of data records are usually recorded in the database of the first device.
  • a lesser data record 80 records
  • the second device sends a request carrying a timestamp of 0.
  • the second device will read the 80 records at a time.
  • the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces.
  • the full amount of data is fragmented by the first device, and the device reads the fragment data in the cache of the first device in batches, which can reduce the reading pressure of the data and shorten the comparison compared with the scheme in the related art.
  • the data transmission time improves the data transmission efficiency.
  • FIG. 14 is a block diagram 3 of a structure of a data transmission apparatus according to Embodiment 4 of the present invention. As shown in FIG. 14, the apparatus further includes:
  • the second receiving module 1402, and the sending module 1204, is configured to receive, when the value of the timestamp is greater than or equal to the preset threshold, the incremental data generated by the first device from the time indicated by the timestamp to the current time.
  • the first type of timestamp may be the timestamp in the embodiment shown in FIG. 12, and the timestamp may be used to read the points in the full amount of data.
  • the role of the index identification of the slice data which may be represented by a one-digit number such as 0, 1, 2, etc.
  • the second type of timestamp may be a real timestamp, which represents a real time, that is, the last access of the second device
  • the time of the first device which can be represented by a 9-digit number, that is, the second type of timestamp can both serve as an identifier or a real time.
  • the preset threshold is generally fragmented data, it is not a large number, so the second type of timestamp, which can represent the real timestamp with 9 digits, is generally larger than the preset threshold, and thus the The second type of timestamp is used as an identifier for the end of a full data read, or an indication of the beginning of a normal incremental data read.
  • the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the device in the second device is a normal incremental data read request.
  • the device of the second device needs to be The incremental data between the time when the second device last accessed the first device (ie, the time indicated by the time stamp described above) to the current time is read, and in the subsequent time, the device is always synchronized with the normal incremental data.
  • the method reads the incremental data until the time point is the whole point or half point, that is, until the above device receives an instruction to read the data in a full-time read synchronous manner, and so on.
  • the data communication between the first device and the foregoing device in the second device may use the http protocol, but is not limited thereto.
  • the above device can be applied to the field of monitoring and shielding of network content.
  • the first device monitors content of the network such as yellow, poison, violence, and terrorism, or other devices monitor the content.
  • the uniform resource locator of the webpage corresponding to the content or the domain name of the computer where the content appears is recorded, and then the data is synchronized to the second device by using the device, and the second device can be transmitted to the other device.
  • the device or the second device itself is obtained according to These are blocked by a Uniform Resource Locator or Domain Name.
  • Embodiments of the present invention may provide a computer terminal, which may be any one of computer terminal groups.
  • the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.
  • the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.
  • the computer terminal may execute the program code of the steps in the data transmission method in the first embodiment of the application program.
  • the program code may be executed the program code of the steps in the data transmission method in the first embodiment of the application program.
  • FIG. 15 is a structural block diagram of a computer terminal according to an embodiment of the present invention.
  • the computer terminal A may include one or more (only one shown in the figure) processor 1502, memory 1506, and transmission device 1508.
  • the memory 1506 can be used to store software programs and modules, such as the security vulnerability detection method and the program instructions/modules corresponding to the device in the embodiment of the present invention.
  • the processor 1502 executes the software programs and modules stored in the memory, thereby executing each A functional application and data processing, that is, a detection method for implementing the above system vulnerability attack.
  • Memory 1506 can include high speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
  • the memory can further include a memory 1506 remotely located relative to the processor 1502, which can be connected to the terminal A via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the processor 1502 can call the information stored in the memory 1506 and the application through the transmission device 1408 to execute the application of the steps in Embodiment 1.
  • a solution of a computer terminal is provided.
  • the program code of the steps in the above Embodiment 1 is executed, the pressure of data reading is reduced, and the technical problem of frequent failures when updating data in full between the systems is solved, and the efficiency of data transmission is improved.
  • FIG. 15 is only for illustration, and the computer terminal can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, MID). ), PAD and other terminal devices.
  • Fig. 15 does not limit the structure of the above electronic device.
  • computer terminal A may also include more or fewer components (such as a network interface, display device, etc.) than shown in FIG. 15, or have a different configuration than that shown in FIG.
  • Embodiments of the present invention may provide a computer terminal, which may be any one of computer terminal groups.
  • the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.
  • the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.
  • the computer terminal may execute the program code of the steps in the data transmission method in the second embodiment of the application program.
  • the program code may be executed the program code of the steps in the data transmission method in the second embodiment of the application program.
  • the computer terminal in this embodiment is similar in structure to the computer terminal in the embodiment 5, and includes a processor, a memory, and a transmission device, wherein the memory is used to store the program code of the steps of the data transmission method in Embodiment 2, and the processing The program code for executing the information stored in the memory and the application program by the transfer device to execute the steps in the second embodiment.
  • the memory is used to store the program code of the steps of the data transmission method in Embodiment 2
  • the program code for executing the information stored in the memory and the application program by the transfer device to execute the steps in the second embodiment.
  • Embodiments of the present invention also provide a storage medium.
  • the foregoing storage medium may be used to save the program code executed by the data transmission method provided in Embodiment 1 above.
  • the foregoing storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.
  • the storage medium is configured to store program code for performing the steps of the data transmission method in Embodiment 1 or Embodiment 2.
  • program code for performing the steps of the data transmission method in Embodiment 1 or Embodiment 2.
  • the disclosed technical contents may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be used. Combinations can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or otherwise.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • the technical solution of the present invention which is essential or contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium.
  • a number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本发明公开了一种数据传输方法及装置。其中，该方法包括：第一设备定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的请求消息对应的时间戳；第一设备判断时间戳的取值是否小于预设阈值；在时间戳的取值小于预设阈值时，第一设备按照时间戳与第一设备在本地缓存的分片数据的对应关系，将与时间戳对应的分片数据同步至第二设备，其中，分片数据为对第一设备记录的全量数据进行分片得到。本发明解决了系统之间进行全量更新数据时频繁失败的技术问题。

Description

数据传输方法及装置
本申请要求2015年10月22日递交的申请号为201510696508.4、发明名称为“数据传输方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及互联网技术领域，具体而言，涉及一种数据传输方法及装置。
背景技术
随着网络技术的发展，有些不法分子可能借助于网络平台，发布一些危害社会的信息内容，比如涉黄、涉赌、涉暴以及涉恐内容等，这将给社会带来不利的影响，因而，可能需要对网络内容进行监控，当发现这些不利信息时，可以对其进行屏蔽，以尽量减少该不利信息的传播。
在相关技术中，网络内容监控系统，一般包括两个系统(可以表现为两个设备)，比如，系统A和系统B，系统B需要同步系统A中的数据，在相关技术中，每次同步需要同步所有的数据信息，这样可能会导致数据库读超时，或者而当系统A与系统B处于不同网络环境时、系统A提供的全量数据过大、系统B内存使用过多时，会造成系统B从系统A全量更新数据频繁失败。
针对上述的问题，目前尚未提出有效的解决方案。
发明内容
本发明实施例提供了一种数据传输方法及装置，以至少解决系统之间进行全量更新数据时频繁失败的技术问题。
根据本发明实施例的一个方面，提供了一种数据传输方法，包括：第一设备定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的请求消息对应的时间戳；第一设备判断时间戳的取值是否小于预设阈值；在时间戳的取值小于预设阈值时，第一设备按照时间戳与第一设备在本地缓存的分片数据的对应关系，将与时间戳对应的分片数据同步至第二设备，其中，分片数据为对第一设备记录的全量数据进行分片得到。
根据本发明实施例的另一方面，还提供了一种数据传输方法，包括：第二设备向第一设备发送请求消息，其中，该请求消息中携带有与请求消息对应的时间戳；在时间戳的取值小于预设阈值的情况下，第二设备接收第一设备发送的分片数据；其中，分片 数据是与时间戳对应的在第一设备本地缓存的分片数据,上述分片数据为对第一设备记录的全量数据进行分片得到。
根据本发明实施例的另一方面，还提供了一种数据传输装置，应用于第一设备，包括：接收模块，用于定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的请求消息对应的时间戳；第一判断模块，用于判断时间戳的取值是否小于预设阈值；第一同步模块，用于在时间戳的取值小于预设阈值时，按照时间戳与第一设备在本地缓存的分片数据的对应关系，将与时间戳对应的分片数据同步至第二设备，其中，分片数据为对第一设备记录的全量数据进行分片得到。
根据本发明实施例的另一方面，还提供了一种数据传输装置，应用于第二设备，包括：发送模块，用于向第一设备发送请求消息，其中，该请求消息中携带有与请求消息对应的时间戳；第一接收模块，用于在时间戳的取值小于预设阈值的情况下，接收第一设备发送的分片数据；其中，分片数据是与时间戳对应的在第一设备本地缓存的分片数据，上述分片数据为第一设备记录的全量数据进行分片得到。
在本发明实施例中，采用第一设备将时间戳对应的第一设备在本地缓存的分片数据同步到第二设备，其中，所述分片数据为对所述第一设备记录的全量数据进行分片得到的方式，即通过将全量数据进行缓存分片的方式，然后根据接收的请求消息中的时间戳，将该时间戳对应的缓存中的分片数据依次同步到第二设备的方式，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是本发明实施例的一种数据传输方法的计算机终端的硬件结构框图；
图2是根据本发明实施例1的数据传输方法的流程图一；
图3是根据本发明实施例1的数据传输方法的流程图二；
图4是根据本发明可选实施例的第一设备分片缓存全量数据的流程示意图；
图5是根据本发明可选实施例的数据传输方法的流程示意图；
图6是根据本发明实施例2的数据传输方法的流程图一；
图7是根据本发明实施例2的数据传输方法的流程图二；
图8是根据本发明实施例3的数据传输装置的结构框图一；
图9是根据本发明实施例3的数据传输装置的结构框图二；
图10是根据本发明实施例3的数据传输装置的结构框图三；
图11是根据本发明实施例3的数据传输装置的结构框图四；
图12是根据本发明实施例4的数据传输装置的结构框图一；
图13是根据本发明实施例4的数据传输装置的结构框图二；
图14是根据本发明实施例4的数据传输装置的结构框图三；
图15是根据本发明实施例的一种计算机终端的结构框图。
具体实施方式
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分的实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本发明保护的范围。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本发明的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
在相关技术中，在网络内容监控和屏蔽领域中，设备之间数据传输的主要步骤如下：
1.当第二设备启动时，第一次从第一设备读取统一资源定位符url/域名阻断记录；
2.该次读取统一资源定位符url/域名阻断记录是全量更新的方式，第二设备发送的请求会带时间戳参数，值为0；
3.第一设备收到时间戳参数值为0的请求后，会将当前数据库中所有url/域名阻断记录读出，并加上当前时间值(精确到分钟)，一并返回给第二设备；
4.第二设备收到全量url/域名阻断记录后，清空系统内的所有阻断记录，改为使用收到的全量url/域名阻断记录；
5.1分钟后，第二设备判断当前时间是否整点或半点。如果是，则执行2；如果不是，则发送请求给第一设备，带时间戳参数是3中第一设备返回的时间值；
6.第一设备收到时间戳参数值不为0的请求后，会将数据库中从时间戳指定时间到当前时间的变化数据读出，并加上当前时间值(精确到分钟)，一并返回给第二设备；
7.第二设备收到增量url/域名阻断记录后，对系统内的所有阻断记录做更新；然后执行5。
当上述第一设备的url/域名阻断数据到达千万级别时，会出现如下问题：第一设备实时从数据库全量读取url/域名阻断数据时，会造成读数据库超时；当第一设备与第二设备处于不同网络环境，网络链路不稳定；千万级别的全量url/域名阻断记录会延长数据传输时间，经常造成全量数据传输失败；第二设备服务器内存有限，当自身内存使用较多时，全量url/域名阻断记录的换入换出，经常造成内存报警。
为了方便理解本发明，以下对本发明实施例中所涉及的术语进行简单解释：
增量同步方式：系统A维护一批实时变化的数据，系统B需要使用系统A维护的这批数据；当系统B访问系统A读取数据时，系统A只将系统B上次访问与这次访问之间的数据变化量发送给系统B，这种数据同步方式叫做增量同步方式。
全量同步方式：系统A维护一批实时变化的数据，系统B需要使用系统A维护的这批数据；当系统B访问系统A读取数据时，系统A将当前所有数据发送给系统B，这种数据同步方式叫做全量同步方式。
本发明实施例对第一设备与第二设备的阻断记录的全量更新过程做了优化，使用增量同步方式来模拟全量同步方式，以避免出现以上问题。以下结合具体实施例详细说明。
实施例1
根据本发明实施例，还提供了一种数据传输的方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
本申请实施例1所提供的方法实施例可以在移动终端、计算机终端或者类似的运算装置中执行。以运行在计算机终端上为例，图1是本发明实施例的一种数据传输方法的计算机终端的硬件结构框图。如图1所示，计算机终端10可以包括一个或多个(图中仅示出一个)处理器102(处理器102可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)、用于存储数据的存储器104、以及用于通信功能的传输模块 106。本领域普通技术人员可以理解，图1所示的结构仅为示意，其并不对上述电子装置的结构造成限定。例如，计算机终端10还可包括比图1中所示更多或者更少的组件，或者具有与图1所示不同的配置。
存储器104可用于存储应用软件的软件程序以及模块，如本发明实施例中的数据传输方法对应的程序指令/模块，处理器102通过运行存储在存储器104内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的应用程序的数据传输方法。存储器104可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至计算机终端10。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输模块106用于经由一个网络接收或者发送数据。上述的网络具体实例可包括计算机终端10的通信供应商提供的无线网络。在一个实例中，传输模块106包括一个网络适配器(Network Interface Controller，NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输模块106可以为射频(Radio Frequency，RF)模块，其用于通过无线方式与互联网进行通讯。
在上述运行环境下，本申请提供了如图2所示的数据传输方法。图2是根据本发明实施例1的数据传输方法的流程图一，如图2所示，该方法包括：
步骤S202，第一设备定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的请求消息对应的时间戳；
可选地，第一设备可以为上述运行环境中的计算机终端10，通过上述计算机终端10来完成第一设备的功能进而实现本实施例中的数据传输方法。
需要说明的是，定时接收来自第二设备的请求消息中的定时可以是按照实际情况进行设定，两次接收的请求消息的时间间隔可以是固定的，比如设定每隔一分钟接收一次请求消息，也可以是不固定的，比如一分钟后接收第二次请求消息，再过三分钟后接收第三次请求消息，具体的可以根据实际情况进行设定。
步骤S204，第一设备判断时间戳的取值是否小于预设阈值；
需要说明的是，上述第一设备和第二设备可以表现为两个物理实体，也可以表现为两个应用，但不限于此。上述时间戳的取值可以是单纯的序号比如阿拉伯数字，也可以是时间值，但不限于此。
上述时间戳的取值可以是指定值，用于指示读取第一设备中的全量数据；在一个可选的实施例中，上述指定值可以是0，即在请求消息中携带的时间戳的取值为0时，指示第二设备应当读第一设备的全量数据。
在一个可选实施例中，在步骤S204之前，需要判断上述时间戳的取值是否为上述指定值，以及在判断结果为是时，触发将与该时间戳对应的分片数据同步至所述第二设备。在时间戳的取值为上述指定值的情况下，可以触发第一设备开始将全量数据中的第一个分片数据同步到第二设备。
步骤S206，在时间戳的取值小于预设阈值时，第一设备按照时间戳与第一设备在本地缓存的分片数据的对应关系，将与时间戳对应的分片数据同步至第二设备，其中，分片数据为对第一设备记录的全量数据进行分片得到。
需要说明的是，上述预设阈值可以是对上述全量数据进行分片后得到的分片数量，当然也可以是其他的值，一般该预设阈值可设置为小于10；上述全量数据可以是针对统一资源定位符url的阻断记录，也可以是针对域名的阻断记录，也可以是两者的组合，并不限于此。
上述分片数据可以通过以下方式在第一设备中得到：将第一设备的数据库中与第一索引n对应的分片数据，写入本地缓存，并将n-1作为分片数据在第一设备的本地缓存中的索引，其中，n小于或者等于对全量数据进行分片后得到的所有分片数据的总数量N，n、N均为正整数。在一个可选实施例中，将第一设备的数据库中与第一索引n对应的分片数据，写入本地缓存，并将n-1作为分片数据在第一设备的本地缓存中的索引具体为：第一设备判断n是否小于N；在n小于N的情况下，第一设备从数据库中读取n对应的分片数据，并在n对应的分片数据上附上值为n的时间戳；在n等于N的情况下，第一设备从数据库中读取与n对应的分片数据，并在与n对应的分片数据上附上值为指定时刻的时间戳；其中，该指定时刻为第二设备上一次访问第一设备的时刻；第一设备将与n对应的分片数据写入本地缓存，并将n-1作为分片数据在本地缓存中的索引。
需要说明的是，进行缓存的方式可以是按照发送格式进行分片缓存，因而在第二设备需要读取时，第一设备不需要再对分片数据进行格式的转变，而是直接返回给第二设备，降低了同步的时间，提高了同步的效率。
通过将第一设备中的全量数据进行分片后缓存在第一设备的本地缓存中，在缓存中缓存的每份分片数据都有对应的一个索引，该索引与第二设备发送的请求消息中的时间戳具有一定的对应关系，通过该时间戳与缓存中的分片数据的对应关系，将全量数据分 片的分片数据同步到第二设备中，在本发明一个可选实施例中，步骤S206中的将与时间戳对应的分片数据同步至第二设备可以表现为：第一设备以时间戳的取值作为索引，将第一设备的本地缓存中与索引对应的指定分片数据同步至第二设备。
通过上述步骤，通过将全量数据进行缓存分片的方式，然后根据接收的请求消息中的时间戳，按照时间戳与缓存中的分片数据的关系，将该时间戳对应的缓存中的分片数据依次同步到第二设备的方式，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
举个例子：第一设备的数据库中通常记录了千万级别的数据记录，此处为了方便理解，以较少数据记录(80条记录)为例，在第一设备接收到携带时间戳为0的请求消息后，按照相关技术中的方式，则第一设备将该80条记录一次性全部同步给第二设备。而在本实施例中，第一设备对该80条记录进行分片，分片可以进行均等分配也可以进行不均等分配，以均等分配为例，分为4片，每个分片有20条记录，将每个分片所包含的数据写入缓存，并设置缓存中的4个分片数据对应的索引0、1、2、3；当第二设备请求读取第一设备的全量数据时，第一次请求中携带的时间戳为0，则第一设备将缓存中以0为索引的分片数据同步至第二设备；一段时间后(比如1分钟)，第一设备接收第二次请求，该第二次请求中携带时间戳为1，则第一设备将缓存中以1为索引的分片数据同步至第二设备；以此类推，直至将缓存中的分片数据读取完毕。即本实施例通过将全量数据进行分片，将分片后的数据依次同步至第二设备，即将全量数据分批同步至第二设备，相对于相关技术中的方案能够降低数据的读取压力，缩短数据的传送时间，提高了数据的传输效率。
在一个可选实施例中，图3是根据本发明实施例1的数据传输方法的流程图二，如图3所示，上述方法还包括：
步骤S302，在时间戳的取值等于或者大于预设阈值时，第一设备将从时间戳指示的时间至当前时间产生的增量数据同步至第二设备。
需要说明的是，在本发明实施例中，存在两类时间戳，第一类时间戳可以是上述图2所示实施例中的时间戳，该时间戳可以起到读取全量数据中的分片数据的索引标识的作用，其可以是用个位数字比如0、1、2等表示；第二类时间戳可以是真实的时间戳，其表示一个真实的时间，即第二设备最近一次访问第一设备的时间，其可以通过一个9位的数字表示，即第二类时间戳既可以起到标识的作用，也可以表示一个真实的时间。由于上述预设阈值一般为分片数据，其不可能是个很大的数字，因而可以用9位数字表 示真实时间戳的第二类时间戳的取值一般是大于上述预设阈值的(即步骤S302中的等于或者大于预设阈值的时间戳)，因而可以将该第二类时间戳作为一次全量数据读取结束的标识，或者是一个正常的增量数据读取的触发标识。
需要说明的是，步骤S302中，时间戳的取值大于或者等于预设阈值，表示第二设备发送的请求是一次正常的增量数据读取请求，此时，需要将第二设备最近一次访问第一设备的时刻(即上述的时间戳指示的时间)至当前时间之间的增量数据传输至第二设备，在后续的时间里第一设备一直以正常的增量同步方式将增量数据同步至第二设备，直到时间点为整点或者半点为止，即直到收到第一设备以全量同步方式将数据同步至第二设备的指示为止，以此类推。
需要说明的是，在本发明的实施例中，上述第一设备与第二设备之间的数据通信可以使用http协议，但不限于此。上述方法可以应用于网络内容的监控与屏蔽领域，比如，第一设备对网络的内容比如涉黄、涉毒、涉暴以及涉恐内容等进行监控，或者其他设备对该内容进行监控，当监控到这些内容后，将该内容对应的网页的统一资源定位符或者该内容出现的计算机所在的域名记录下来，然后将这些数据通过本发明实施例的方法同步到第二设备，第二设备可以再传输给其他设备或者第二设备本身根据获得到统一资源定位符或者域名对这些内容进行屏蔽。
为了更好的理解本发明，以下结合可选的实施例对本发明做进行一步的解释。
在一个可选实施例中，第一设备每30分钟对第一设备的数据库中的全量url/域名进行一次分片缓存，来解决实时从数据库全量读取记录时造成的读取超时等问题。图4是根据本发明可选实施例的第一设备分片缓存全量数据的流程示意图，如图4所示，包括以下步骤：
步骤S402，设定该次全量url/域名阻断记录的分片数量(例如4)，该分片数量小于10；
步骤S404，读取当前时间，精确到分钟，记录在timestamp中；
步骤S406，根据timestamp的值，读取当前时间数据库中全量记录数量(例如81)；
步骤S408，根据分片数量，计算全量记录的每个分片中包含的记录数(例如20、20、20、21)；
步骤S410，设置分片计数index(即上述实施例中的第一索引n)，值为1；
步骤S412，判断当前index值是否小于分片数量(相当于上述实施例中的分片数据的总数量N)；是(例如1)，则执行步骤S414；不是(例如4)，则执行步骤S422；
步骤S414，从数据库中读取与index对应的指定分片包含的数据；
步骤S416，附上时间戳，值为index；
步骤S418，将分片数据与时间戳写入缓存，key值为index–1(相当于上述实施例中的n-1)；
步骤S420，分片计数index增加1；返回步骤S412执行；
步骤S422，从数据库中读取与index对应的指定分片包含的数据；
步骤S424，附上时间戳，值为timestamp(相当于上述实施例中的指定时刻)；
步骤S426，将分片数据与时间戳写入缓存，key值为index-1。
第二设备从第一设备读取全量阻断记录时，第一设备将已有全量阻断记录缓存分批返回给第二设备，以解决大数据量一次性传送时可能导致的网络问题与第二设备内存使用等问题。图5是根据本发明可选实施例的数据传输方法的流程示意图，如图5所示，以4片缓存为例，该方法包括以下步骤：
步骤S501，第二设备向第一设备发起请求，读全量阻断记录，时间戳为0；
步骤S502，第一设备判断时间戳为0，是全量更新阻断记录，以0为key在全量阻断记录的缓存中读取第一片缓存中的分片数据，并附带时间戳为1，返回给第二设备；
步骤S503，第二设备清空系统内阻断记录，并使用收到的全量阻断记录的第一片缓存中的分片数据；
步骤S504，1分钟后，第二设备向第一设备发起请求，读取增量阻断记录，时间戳为步骤S502中第一设备返回的1；
步骤S505，第一设备判断时间戳小于10，是未完的全量更新阻断记录，以1为key在全量阻断记录的缓存中读取第二片缓存中的分片数据，附带时间戳2，返回给第二设备；
步骤S506，第二设备将收到的第二片缓存中的分片数据，增加到已有阻断记录中；
步骤S507，1分钟后，第二设备向第一设备发起请求，读取增量阻断记录，时间戳为步骤S505中第一设备返回的2；
步骤S508，第一设备判断时间戳小于10，是未完的全量更新阻断记录，以2为key在全量阻断记录的缓存中读取第三片缓存中的分片数据，附带时间戳3，返回给第二设备；
步骤S509，第二设备将收到的第三片缓存中的分片数据，增加到已有阻断记录中；
步骤S510，1分钟后，第二设备向第一设备发起请求，读取增量阻断记录，时间戳 为步骤S508中第一设备返回的3；
步骤S511，第一设备判断时间戳小于10，是未完的全量更新阻断记录，以3为key在全量阻断记录的缓存中读取第四片缓存中的分片数据，附带时间戳为全量阻断记录缓存的生成时间timestamp，返回给第二设备；
步骤S512，第二设备将收到的第四片缓存中的分片数据，增加到已有阻断记录中；
步骤S513，1分钟后，第二设备向Mrm发起请求，读取增量阻断记录，时间戳为步骤S511中第一设备返回的timestamp；
步骤S514，第一设备Mrm判断时间戳大于10，是一次正常的增量阻断记录请求，读数据库中timestamp到当前时间的阻断记录变化量，附带当前时间，返回给第二设备；
后续每分钟，第二设备都做增量阻断记录请求；如果是整点或半点，则第二设备不做增量阻断记录请求，而返回执行步骤S501。
需要说明的是，在本发明图5所示的可选实施例中的1分钟相当于上述实施例图2所示实施例的定时，上述时间戳为0相当于图2所示实施例中的时间戳的取值为指定值，来指示读全量阻断记录，上述实施例中的时间戳为0、1、2、3相当于图2所示实施例中的时间戳的取值；上述key相当于图2所示实施例中的索引，上述timestamp相当于图3所示实施例中的取值等于或者大于预设阈值的时间戳。
在上述实施例中，同步分为两种，增量同步方式和全量同步方式，全量同步方式同步数据是这样实现的：通过预先将全量数据提前进行分片缓存，在第二设备进行读取的时候，直接使用分片缓存中的分片数据，即根据全量数据分片的数量N，通过N次模拟增量同步方式实现全量数据同步的，降低了数据读取压力，在第一设备第一次同步给第二设备时，也不会因为数据量过大而同步失败。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本发明并不受所描述的动作顺序的限制，因为依据本发明，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本发明所必须的。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的数据传输方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品 存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。
实施例2
根据本发明实施例，还提供了一种数据传输方法，图6是根据本发明实施例2的数据传输方法的流程图一，如图6所示，该方法包括：
步骤S602，第二设备向第一设备发送请求消息，其中，该请求消息中携带有与请求消息对应的时间戳；
需要说明的是，上述第二设备可以是与上述实施例1中的计算机终端10的结构相同的一个计算机终端，在该计算机终端的环境中完成本实施例中的数据传输方法，但不限于此。上述发送请求消息可以是定时发送请求消息，可以是不定时发送请求消息，可以根据具体的情况进行具体设定。
步骤S604，在时间戳的取值小于预设阈值的情况下，第二设备接收第一设备发送的分片数据；其中，分片数据是与时间戳对应的在所述第一设备本地缓存的分片数据,上述分片数据为对第一设备记录的全量数据进行分片得到。
上述时间戳的取值可以是指定值，用于指示第二设备读取第一设备中的全量数据；在一个可选的实施例中，上述指定值可以是0，即在请求消息中携带的时间戳为0时，指示第二设备应当读第一设备的全量数据。在一个可选的实施例中，在步骤S604之前，上述方法还包括：在时间戳为指定值的情况下，第二设备触发第一设备发送全量数据，接收第一设备发送的与指定值对应的分片数据。
需要说明的是，上述预设阈值可以是第一设备对上述全量数据进行分片后得到的分片数量，当然也可以是其他的值，一般该预设阈值可设置为小于10；上述全量数据可以是针对统一资源定位符url的阻断记录，也可以是针对域名的阻断记录，也可以是两者的组合，并不限于此。
具体地，第一设备是如何对全量数据进行分片缓存的，即上述分片数据是如何在第一设备中得到的，与上述实施例1中的相同，此处不再赘述。
上述步骤S604中，第二设备接收第一设备发送的分片数据可以表现为：第二设备接收在第一设备的本地缓存中与索引对应的指定分片数据；其中，索引为上述时间戳；第二设备接收第一设备发送的指定时间戳；其中，指定时间戳的取值为由上述时间戳的取值加1得到。
通过上述步骤，通过第二设备发送携带时间戳的请求消息，接收第一设备根据接收的请求消息中的时间戳，发送的该时间戳对应的第一设备的缓存中的分片数据的方式，即第二设备按照时间戳读取在第一设备中预先进行分片缓存的分片数据，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
举个例子：第一设备的数据库中通常记录了千万级别的数据记录，此处为了方便理解，以较少数据记录(80条记录)为例，第二设备发送携带时间戳为0的请求消息后，按照相关技术中的方式，则第二设备将一次性读取该80条记录。而在本实施例中，第一设备对该80条记录进行分片，分片可以进行均等分配也可以进行不均等分配，以均等分配为例，分为4片，每个分片有20条记录，将每个分片所包含的数据写入缓存，并设置缓存中的4个分片数据对应的索引0、1、2、3；当第二设备请求读取第一设备的全量数据时，第一次请求中携带的时间戳为0，则第二设备读取第一设备的缓存中以0为索引的分片数据；一段时间后(比如1分钟)，第二设备发送第二次请求，该第二次请求中携带时间戳为1，则第二设备读取第一设备的缓存中以1为索引的分片数据；以此类推，直至第二设备将第一设备的缓存中的分片数据读取完毕。即本实施例通过第一设备将全量数据进行分片，第二设备分批读取该第一设备的缓存中的分片数据，相比于相关技术中的方案能够降低数据的读取压力，缩短数据的传送时间，提高了数据的传输效率。
在一个可选实施例中，图7是根据本发明实施例2的数据传输方法的流程图二，如图7所示，上述方法还包括：
步骤S702，在时间戳的取值等于或者大于预设阈值的情况下，第二设备接收第一设备发送的从时间戳指示的时间至当前时间产生的增量数据。
需要说明的是，在本发明实施例中，存在两类时间戳，第一类时间戳可以是上述图6所示实施例中的时间戳，该时间戳可以起到读取全量数据中的分片数据的索引标识的作用，其可以是用个位数字比如0、1、2等表示；第二类时间戳可以是真实的时间戳，其表示一个真实的时间，即第二设备最近一次访问第一设备的时间，其可以通过一个9位的数字表示，即第二类时间戳既可以起到标识的作用，也可以表示一个真实的时间。由于上述预设阈值一般为分片数据，其并不是很大的数字，因而作为可以用9位数字表示真实时间戳的第二类时间戳一般是大于上述预设阈值的(即步骤S702中的等于或者大于预设阈值的时间戳)，因而可以将该第二类时间戳作为一次全量数据读取结束的标 识，或者是一个正常的增量数据读取的触发标识。
需要说明的是，步骤S702中，时间戳的取值大于或者等于预设阈值，表示第二设备发送的请求是一次正常的增量数据读取请求，此时，第二设备需要将第二设备最近一次访问第一设备的时刻(即上述的时间戳指示的时间)至当前时间之间的增量数据读取过来，在后续的时间里，第二设备一直以正常的增量同步方式读取增量数据，直到时间点为整点或者半点为止，即直到第二设备收到以全量同步方式读取数据的指示为止，以此类推。
需要说明的是，在本发明的实施例中，上述第一设备与第二设备之间的数据通信可以使用http协议，但不限于此。上述方法可以应用于网络内容的监控与屏蔽领域，比如，第一设备对网络的内容比如涉黄、涉毒、涉暴、以及涉恐内容等进行监控，或者其他设备对该内容进行监控，当监控到这些内容后，将该内容对应的网页的统一资源定位符或者该内容出现的计算机所在的域名记录下来，然后将这些数据通过本发明实施例的方法同步到第二设备，第二设备可以再传输给其他设备或者第二设备本身根据获得到统一资源定位符或者域名对这些内容进行屏蔽。
为了更好的理解本发明，可以结合实施例1中的图4和图5所示的可选的实施例对本发明做进行一步的解释，此处不再赘述。
实施例3
根据本发明实施例，还提供了一种用于实施上述实施例1中的数据传输方法的装置，图8是根据本发明实施例3的数据传输装置的结构框图一，如图8所示，该装置，应用于第一设备，包括：
接收模块80，用于定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的请求消息对应的时间戳；
需要说明的是，这里的定时可以是按照实际情况进行设定，两次接收的请求消息的时间间隔可以是固定的，比如设定每隔1分钟接收一次请求消息，也可以是不固定的，比如1分钟后接收第二次请求消息，再过3分钟后接收第三次请求消息，具体的可以根据实际情况进行设定。
第一判断模块82，与上述接收模块80连接，用于判断时间戳的取值是否小于预设阈值；
上述时间戳的取值可以是指定值，用于指示读取第一设备中的全量数据；在一个可选的实施例中，上述指定值可以是0，即在请求消息中携带的时间戳为0时，指示第二 设备应当读第一设备的全量数据。
在一个可选实施例中，图9是根据本发明实施例3的数据传输装置的结构框图二，如图9所示，上述装置还包括：
第二判断模块90，与接收模块80连接，用于判断时间戳的取值是否为指定值，其中，该指定值用于指示读取全量数据，其中，在判断结果为是时，触发将与时间戳对应的分片数据同步至第二设备。
第一同步模块84，与上述第一判断模块82连接，用于在时间戳的取值小于预设阈值时，按照时间戳与第一设备在本地缓存的分片数据的对应关系，将与时间戳对应的分片数据同步至第二设备，其中，分片数据为对第一设备记录的全量数据进行分片得到。
需要说明的是，上述预设阈值可以是对上述全量数据进行分片后得到的分片数量，当然也可以是其他的值，一般该预设阈值可设置为小于10；上述全量数据可以是针对统一资源定位符url的阻断记录，也可以是针对域名的阻断记录，也可以是两者的组合，并不限于此。
上述第一同步模块84还用于以时间戳的取值作为索引，将第一设备的本地缓存中与该索引对应的指定分片数据同步至第二设备。
在一个可选的实施例中，图10是根据本发明实施例3的数据传输装置的结构框图三，如图10所示，上述装置还包括：
缓存模块100，与第一同步模块84连接，用于将第一设备的数据库中与第一索引n对应的分片数据，写入本地缓存，并将n-1作为分片数据在第一设备的本地缓存中的索引，其中，n小于或者等于对全量数据进行分片后得到的所有分片数据的总数量N，n、N均为正整数。
具体地，上述缓存模块100还包括：
判断单元1001，用于判断n是否小于N；
第一读取单元1002，与上述判断单元1001连接，用于在n小于N的情况下，从所述数据库中读取n对应的分片数据，并在n对应的分片数据上附上值为n的时间戳；
第二读取单元1003，与上述判断单元1001连接，用于在n等于N的情况下，从所述数据库中读取与n对应的分片数据，并在与n对应的分片数据上附上值为指定时刻的时间戳；其中，所述指定时刻为所述第二设备上一次访问所述第一设备的时刻；
缓存单元1004，与上述第一读取单元1002和第二读取单元1003连接，用于将与n对应的分片数据写入本地缓存，并将n-1作为分片数据在本地缓存中的索引。
需要说明的是，上述缓存单元1004可以是按照发送格式将与n对应的分片数据写入本地缓存，这样，在第二设备需要读取时，第一设备不需要再对分片数据进行格式的转变，而是直接返回给第二设备，降低了同步的时间，提高了同步的效率。
通过上述装置，通过将全量数据进行缓存分片的方式，然后根据接收的请求消息中的时间戳，按照时间戳与缓存中的分片数据的关系，将该时间戳对应的缓存中的分片数据依次同步到第二设备的方式，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
举个例子：第一设备的数据库中通常记录了千万级别的数据记录，此处为了方便理解，以较少数据记录(80条记录)为例，在第一设备接收到携带时间戳为0的请求消息后，按照相关技术中的方式，则第一设备将该80条记录一次性全部同步给第二设备。而在本实施例中，第一设备中的上述装置对该80条记录进行分片，分片可以进行均等分配也可以进行不均等分配，以均等分配为例，分为4片，每个分片有20条记录，缓存单元1004将每个分片所包含的数据写入缓存，并设置缓存中的4个分片数据对应的索引0、1、2、3；当第二设备请求读取第一设备的全量数据时，第一次请求中携带的时间戳为0，则第一设备的第一同步模块84将缓存中以0为索引的分片数据同步至第二设备；一段时间后(比如1分钟)，第一设备的接收模块80接收第二次请求，该第二次请求中携带时间戳为1，则第一设备的第一同步模块84将缓存中以1为索引的分片数据同步至第二设备；以此类推，直至将缓存中的分片数据读取完毕。即本实施例的上述装置通过将全量数据进行分片，将分片后的数据依次同步至第二设备，即将全量数据分批同步至第二设备，相比于相关技术中的方案能够降低数据的读取压力，缩短数据的传送时间，提高了数据的传输效率。
图11是根据本发明实施例3的数据传输装置的结构框图四，如图11所示，上述装置还包括：
第二同步模块1101，与上述第一判断模块82连接，用于在时间戳的取值等于或者大于预设阈值时，将从时间戳指示的时间至当前时间产生的增量数据同步至第二设备。
需要说明的是，在本发明实施例中，存在两类时间戳，第一类时间戳可以是上述图8所示实施例中的时间戳，该时间戳可以起到读取全量数据中的分片数据的索引标识的作用，其可以是用个位数字比如0、1、2等表示；第二类时间戳可以是真实的时间戳，其表示一个真实的时间，即第二设备最近一次访问第一设备的时间，其可以通过一个9位的数字表示，即第二类时间戳既可以起到标识的作用，也可以表示一个真实的时间。 在由于上述预设阈值一般为分片数据，其不可能是个很大的数字，因而作为可以用9位数字表示真实时间戳的第二类时间戳的取值一般是大于上述预设阈值的，因而可以将该第二类时间戳作为一次全量数据读取结束的标识，或者是一个正常的增量数据读取的触发标识。
需要说明的是，时间戳的取值大于或者等于预设阈值，表示第二设备发送的请求是一次正常的增量数据读取请求，此时，第二同步模块1101需要将第二设备最近一次访问第一设备的时刻(即上述的时间戳指示的时间)至当前时间之间的增量数据传输至第二设备，在后续的时间里第一设备一直以正常的增量数据同步方式将增量数据同步至第二设备，直到时间点为整点或者半点为止，即直到收到第一设备以全量读取同步方式将数据同步至第二设备的指示为止，以此类推。
需要说明的是，在本发明的实施例中，第一设备中的上述装置与第二设备之间的数据通信可以使用http协议，但不限于此。上述装置可以应用于网络内容的监控与屏蔽领域，比如，第一设备对网络的内容比如涉黄、涉毒、涉暴、以及涉恐内容等进行监控，或者其他设备对该内容进行监控，当监控到这些内容后，将该内容对应的网页的统一资源定位符或者该内容出现的计算机所在的域名记录下来，然后将这些数据通过上述装置同步到第二设备，第二设备可以再传输给其他设备或者第二设备本身根据获得到统一资源定位符或者域名对这些内容进行屏蔽。
需要说明的是，上述模块第一同步模块84第二同步模块1101可以是第一设备中的同一个同步模块，也可以是第一设备中的不同的同步模块。
实施例4
根据本发明实施例，还提供了一种用于实施上述实施例2中的数据传输方法的装置，图12是根据本发明实施例4的数据传输装置的结构框图一，如图12所示，该装置，应用于第二设备，包括：
发送模块1202，用于向第一设备发送请求消息，其中，该请求消息中携带有与请求消息对应的时间戳；
需要说明的是，上述发送请求消息可以是定时发送请求消息，可以是不定时发送请求消息，可以根据具体的情况进行具体设定。
第一接收模块1204，与上述发送模块1202连接，用于在时间戳的取值小于预设阈值的情况下，接收第一设备发送的分片数据；其中，分片数据是与时间戳对应的在第一设备本地缓存的分片数据，上述分片数据为第一设备记录的全量数据进行分片得到。
上述时间戳可以是指定值，用于指示第二设备读取第一设备中的全量数据；在一个可选的实施例中，上述指定值可以是0，即在请求消息中携带的时间戳为0时，指示第二设备应当读第一设备的全量数据。在一个可选实施例中，上述装置还包括：触发模块，用于在时间戳的取值为指定值的情况下，触发第一设备发送全量数据；第三接收模块，与该触发模块连接，用于接收第一设备发送的与指定值对应的分片数据，其中，指定值用于指示第二设备向所述第一设备读取全量数据。
需要说明的是，上述预设阈值可以是第一设备对上述全量数据进行分片后得到的分片数量，当然也可以是其他的值，一般该预设阈值可设置为小于10；上述全量数据可以是针对统一资源定位符url的阻断记录，也可以是针对域名的阻断记录，也可以是两者的组合，并不限于此。
具体地，第一设备是如何对全量数据进行分片缓存的，即上述分片数据是如何在第一设备中得到的，与上述实施例1中的相同，此处不再赘述。
在本发明的一个可选实施例中，图13是根据本发明实施例4的数据传输装置的结构框图二，如图13所示，上述第一接收模块1204可以包括：第一接收单元1302，用于接收在所述第一设备的本地缓存中与索引对应的指定分片数据；其中，索引为上述时间戳的取值；第二接收单元1304，与上述第一接收单元1302连接，用于接收第一设备发送的指定时间戳；其中，指定时间戳的取值为由上述时间戳的取值加1得到。
通过上述装置，通过发送模块1202发送携带时间戳的请求消息，第一接收模块1204接收第一设备根据接收的请求消息中的时间戳，发送的该时间戳对应的第一设备的缓存中的分片数据的方式，即上述装置按照时间戳读取在第一设备中预先进行分片缓存的分片数据，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
举个例子：第一设备的数据库中通常记录了千万级别的数据记录，此处为了方便理解，以较少数据记录(80条记录)为例，第二设备发送携带时间戳为0的请求消息后，按照相关技术中的方式，则第二设备将一次性读取该80条记录。而在本实施例中，第一设备对该80条记录进行分片，分片可以进行均等分配也可以进行不均等分配，以均等分配为例，分为4片，每个分片有20条记录，将每个分片所包含的数据写入缓存，并设置缓存中的4个分片数据对应的索引0、1、2、3；当第二设备中上述装置中的发送模块1202请求读取第一设备的全量数据时，第一次请求中携带的时间戳为0，则上述装置中的第一接收模块1204读取第一设备的缓存中以0为索引的分片数据；一段时间后 (比如1分钟)，第二设备上述装置中的发送模块1202发送第二次请求，该第二次请求中携带时间戳为1，则上述装置中的第一接收模块1204读取第一设备的缓存中以1为索引的分片数据；以此类推，直至第二设备中的上述装置将第一设备的缓存中的分片数据读取完毕。即本实施例通过第一设备将全量数据进行分片，上述装置分批读取该第一设备的缓存中的分片数据，相比于相关技术中的方案能够降低数据的读取压力，缩短数据的传送时间，提高了数据的传输效率。
图14是根据本发明实施例4的数据传输装置的结构框图三，如图14所示，上述装置还包括：
第二接收模块1402，与发送模块1204用于在时间戳的取值大于或者等于预设阈值的情况下，接收第一设备发送的从时间戳指示的时间至当前时间产生的增量数据。
需要说明的是，在本发明实施例中，存在两类时间戳，第一类时间戳可以是上述图12所示实施例中的时间戳，该时间戳可以起到读取全量数据中的分片数据的索引标识的作用，其可以是用个位数字比如0、1、2等表示；第二类时间戳可以是真实的时间戳，其表示一个真实的时间，即第二设备最近一次访问第一设备的时间，其可以通过一个9位的数字表示，即第二类时间戳既可以起到标识的作用，也可以表示一个真实的时间。在由于上述预设阈值一般为分片数据，其并不是很大的数字，因而作为可以用9位数字表示真实时间戳的第二类时间戳一般是大于上述预设阈值的，因而可以将该第二类时间戳作为一次全量数据读取结束的标识，或者是一个正常的增量数据读取的开始的标识。
需要说明的是，时间戳的取值大于或者等于预设阈值，表示第二设备中的上述装置发送的请求是一次正常的增量数据读取请求，此时，第二设备的上述装置需要将第二设备最近一次访问第一设备的时刻(即上述的时间戳指示的时间)至当前时间之间的增量数据读取过来，在后续的时间里，上述装置一直以正常的增量数据同步方式读取增量数据，直到时间点为整点或者半点为止，即直到上述装置收到以全量读取同步方式读取数据的指示为止，以此类推。
需要说明的是，在本发明的实施例中，上述第一设备与第二设备中的上述装置之间的数据通信可以使用http协议，但不限于此。上述装置可以应用于网络内容的监控与屏蔽领域，比如，第一设备对网络的内容比如涉黄、涉毒、涉暴、以及涉恐内容等进行监控，或者其他设备对该内容进行监控，当监控到这些内容后，将该内容对应的网页的统一资源定位符或者该内容出现的计算机所在的域名记录下来，然后将这些数据通过上述装置同步到第二设备，第二设备可以再传输给其他设备或者第二设备本身根据获得到 统一资源定位符或者域名对这些内容进行屏蔽。
实施例5
本发明的实施例可以提供一种计算机终端，该计算机终端可以是计算机终端群中的任意一个计算机终端设备。可选地，在本实施例中，上述计算机终端也可以替换为移动终端等终端设备。
可选地，在本实施例中，上述计算机终端可以位于计算机网络的多个网络设备中的至少一个网络设备。
在本实施例中，上述计算机终端可以执行应用程序的实施例1中的数据传输方法中的步骤的程序代码，具体详见实施例1，此处不再赘述。
可选地，图15是根据本发明实施例的一种计算机终端的结构框图。如图15所示，该计算机终端A可以包括：一个或多个(图中仅示出一个)处理器1502、存储器1506、以及传输装置1508。
其中，存储器1506可用于存储软件程序以及模块，如本发明实施例中的安全漏洞检测方法和装置对应的程序指令/模块，处理器1502通过运行存储在存储器内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的系统漏洞攻击的检测方法。存储器1506可包括高速随机存储器，还可以包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器可进一步包括相对于处理器1502远程设置的存储器1506，这些远程存储器1506可以通过网络连接至终端A。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
处理器1502可以通过传输装置1408调用存储器1506存储的信息及应用程序，以执行实施例1中的步骤的应用程序。
采用本发明实施例，提供了一种计算机终端的方案。通过该计算机终端，执行上述实施例1中的步骤的程序代码，降低了数据读取的压力，进而解决了系统之间进行全量更新数据时频繁失败的技术问题，提高了数据传输的效率。
本领域普通技术人员可以理解，图15所示的结构仅为示意，计算机终端也可以是智能手机(如Android手机、iOS手机等)、平板电脑、掌声电脑以及移动互联网设备(Mobile Internet Devices，MID)、PAD等终端设备。图15其并不对上述电子装置的结构造成限定。例如，计算机终端A还可包括比图15中所示更多或者更少的组件(如网络接口、显示装置等)，或者具有与图15所示不同的配置。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令终端设备相关的硬件来完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：闪存盘、只读存储器(Read-Only Memory，ROM)、随机存取器(Random Access Memory，RAM)、磁盘或光盘等。
实施例6
本发明的实施例可以提供一种计算机终端，该计算机终端可以是计算机终端群中的任意一个计算机终端设备。可选地，在本实施例中，上述计算机终端也可以替换为移动终端等终端设备。
可选地，在本实施例中，上述计算机终端可以位于计算机网络的多个网络设备中的至少一个网络设备。
在本实施例中，上述计算机终端可以执行应用程序的实施例2中的数据传输方法中的步骤的程序代码，具体详见实施例2，此处不再赘述。
在本实施例中的计算机终端与实施例5中的计算机终端的结构类似，包括处理器、存储器以及传输装置，其中，存储器用于存储实施例2中的数据传输方法的步骤的程序代码，处理器用于通过传输装置调用存储器存储的信息及应用程序，以执行实施例2中的步骤的程序代码。具体步骤详见实施例2中的方法，此处不再赘述。
实施例7
本发明的实施例还提供了一种存储介质。可选地，在本实施例中，上述存储介质可以用于保存上述实施例一所提供的数据传输方法所执行的程序代码。
可选地，在本实施例中，上述存储介质可以位于计算机网络中计算机终端群中的任意一个计算机终端中，或者位于移动终端群中的任意一个移动终端中。
可选地，在本实施例中，存储介质被设置为存储用于执行实施例1或者实施例2中的数据传输方法的步骤的程序代码，具体步骤详见实施例1或者实施例2，此处不再赘述。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
在本发明的上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的技术内容，可通过其它的方式实现。其中，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以 结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，单元或模块的间接耦合或通信连接，可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。

Claims (24)

  1. 一种数据传输方法，其特征在于，包括：
    第一设备定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的所述请求消息对应的时间戳；
    所述第一设备判断所述时间戳的取值是否小于预设阈值；
    在所述时间戳的取值小于所述预设阈值时，所述第一设备按照所述时间戳与所述第一设备在本地缓存的分片数据的对应关系，将与所述时间戳对应的所述分片数据同步至所述第二设备，其中，所述分片数据为对所述第一设备记录的全量数据进行分片得到。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在所述时间戳的取值等于或者大于所述预设阈值时，所述第一设备将从所述时间戳指示的时间至当前时间产生的增量数据同步至所述第二设备。
  3. 根据权利要求1所述的方法，其特征在于，所述第一设备判断所述时间戳的取值是否小于预设阈值之前，所述方法还包括：
    所述第一设备判断所述时间戳的取值是否为指定值，其中，该指定值用于指示读取所述全量数据；并且，在判断结果为是时，触发将与所述时间戳对应的所述分片数据同步至所述第二设备。
  4. 根据权利要求1所述的方法，其特征在于，将与所述时间戳对应的所述分片数据同步至所述第二设备包括：
    所述第一设备以所述时间戳的取值作为索引，将所述第一设备的本地缓存中与所述索引对应的指定分片数据同步至所述第二设备。
  5. 根据权利要求1所述的方法，其特征在于，所述分片数据通过以下方式在所述第一设备中得到：
    所述第一设备将所述第一设备的数据库中与第一索引n对应的分片数据，写入所述本地缓存，并将n-1作为所述分片数据在所述第一设备的本地缓存中的索引，其中，n小于或者等于对所述全量数据进行分片后得到的所有分片数据的总数量N，n、N均为正整数。
  6. 根据权利要求5所述的方法，其特征在于，所述第一设备将所述第一设备的数据库中与所述第一索引n对应的分片数据，写入所述本地缓存，并将n-1作为所述分片数据在所述第一设备的本地缓存中的索引包括：
    所述第一设备判断n是否小于N；
    在n小于N的情况下，所述第一设备从所述数据库中读取n对应的分片数据，并在n对应的分片数据上附上值为n的时间戳；
    在n等于N的情况下，所述第一设备从所述数据库中读取与n对应的分片数据，并在与n对应的分片数据上附上值为指定时刻的时间戳；其中，所述指定时刻为所述第二设备上一次访问所述第一设备的时刻；
    所述第一设备将与n对应的分片数据写入所述本地缓存，并将n-1作为所述分片数据在所述本地缓存中的索引。
  7. 根据权利要求1至6中任一项所述的方法，其特征在于，所述全量数据包括以下至少之一：针对统一资源定位符的阻断记录，针对域名的阻断记录。
  8. 一种数据传输方法，其特征在于，包括：
    第二设备向第一设备发送请求消息，其中，该请求消息中携带有与所述请求消息对应的时间戳；
    在所述时间戳的取值小于预设阈值的情况下，所述第二设备接收所述第一设备发送的分片数据；其中，所述分片数据是与所述时间戳对应的在所述第一设备本地缓存的分片数据,所述分片数据为对所述第一设备记录的全量数据进行分片得到。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    在所述时间戳的取值等于或者大于所述预设阈值的情况下，所述第二设备接收所述第一设备发送的从所述时间戳指示的时间至当前时间产生的增量数据。
  10. 根据权利要求8所述的方法，其特征在于，所述第二设备接收所述第一设备发送的与所述时间戳对应的所述第一设备在本地缓存的分片数据之前，所述方法还包括：
    在所述时间戳的取值为指定值的情况下，所述第二设备触发所述第一设备发送所述全量数据，接收所述第一设备发送的与所述指定值对应的所述分片数据，其中，所述指定值用于指示所述第二设备向所述第一设备读取所述全量数据。
  11. 根据权利要求8所述的方法，其特征在于，所述第二设备接收所述第一设备发送的分片数据包括：
    所述第二设备接收在所述第一设备的本地缓存中与索引对应的指定分片数据；其中，所述索引为所述时间戳的取值；
    所述第二设备接收所述第一设备发送的指定时间戳；其中，所述指定时间戳的取值为由所述时间戳的取值加1得到。
  12. 根据权利要求8至11中任一项所述的方法，其特征在于，所述全量数据包括以 下至少之一：针对统一资源定位符的阻断记录，针对域名的阻断记录。
  13. 一种数据传输装置，应用于第一设备，其特征在于，包括：
    接收模块，用于定时接收来自第二设备的请求消息，其中，该请求消息中携带有与本次接收的所述请求消息对应的时间戳；
    第一判断模块，用于判断所述时间戳的取值是否小于预设阈值；
    第一同步模块，用于在所述时间戳的取值小于所述预设阈值时，按照所述时间戳与所述第一设备在本地缓存的分片数据的对应关系，将与所述时间戳对应的所述分片数据同步至所述第二设备，其中，所述分片数据为对所述第一设备记录的全量数据进行分片得到。
  14. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    第二同步模块，用于在所述时间戳等于或者大于所述预设阈值时，将从所述时间戳指示的时间至当前时间产生的增量数据同步至所述第二设备。
  15. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    第二判断模块，用于判断所述时间戳的取值是否为指定值，其中，该指定值用于指示读取所述全量数据，其中，在判断结果为是时，触发将与所述时间戳对应的所述分片数据同步至所述第二设备。
  16. 根据权利要求13所述的装置，其特征在于，所述第一同步模块，还用于以所述时间戳的取值作为索引，将所述第一设备的本地缓存中与所述索引对应的指定分片数据同步至所述第二设备。
  17. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    缓存模块，用于将所述第一设备的数据库中与第一索引n对应的分片数据，写入所述本地缓存，并将n-1作为所述分片数据在所述第一设备的本地缓存中的索引，其中，n小于或者等于对所述全量数据进行分片后得到的所有分片数据的总数量N，n、N均为正整数。
  18. 根据权利要求17所述的装置，其特征在于，所述缓存模块包括：
    判断单元，用于判断n是否小于N；
    第一读取单元，用于在n小于N的情况下，从所述数据库中读取n对应的分片数据，并在n对应的分片数据上附上值为n的时间戳；
    第二读取单元，用于在n等于N的情况下，从所述数据库中读取与n对应的分片数据，并在与n对应的分片数据上附上值为指定时刻的时间戳；其中，所述指定时刻为所 述第二设备上一次访问所述第一设备的时刻；
    缓存单元，用于将与n对应的分片数据写入所述本地缓存，并将n-1作为所述分片数据在所述本地缓存中的索引。
  19. 根据权利要求13至18中任一项所述的装置，其特征在于，所述全量数据包括以下至少之一：针对统一资源定位符的阻断记录，针对域名的阻断记录。
  20. 一种数据传输装置，应用于第二设备，其特征在于，包括：
    发送模块，用于向第一设备发送请求消息，其中，该请求消息中携带有与所述请求消息对应的时间戳；
    第一接收模块，用于在所述时间戳的取值小于预设阈值的情况下，接收所述第一设备发送的分片数据；其中，所述分片数据是与所述时间戳对应的在所述第一设备本地缓存的分片数据，所述分片数据为所述第一设备记录的全量数据进行分片得到。
  21. 根据权利要求20所述的装置，其特征在于，所述装置还包括：
    第二接收模块，用于在所述时间戳的取值等于或者大于所述预设阈值的情况下，接收所述第一设备发送的从所述时间戳指示的时间至当前时间产生的增量数据。
  22. 根据权利要求20所述的装置，其特征在于，所述装置还包括：
    触发模块，用于在所述时间戳的取值为指定值的情况下，触发所述第一设备发送所述全量数据；
    第三接收模块，用于接收所述第一设备发送的与所述指定值对应的所述分片数据，其中，所述指定值用于指示所述第二设备向所述第一设备读取所述全量数据。
  23. 根据权利要求20所述的装置，其特征在于，所述第一接收模块包括：
    第一接收单元，用于接收在所述第一设备的本地缓存中与索引对应的指定分片数据；其中，所述索引为所述时间戳的取值；
    第二接收单元，用于接收所述第一设备发送的指定时间戳；其中，所述指定时间戳的取值为由所述时间戳的取值加1得到。
  24. 根据权利要求20至23中任一项所述的装置，其特征在于，所述全量数据包括以下至少之一：针对统一资源定位符的阻断记录，针对域名的阻断记录。
PCT/CN2016/101981 2015-10-22 2016-10-13 数据传输方法及装置 Ceased WO2017067415A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2018520613A JP2018536226A (ja) 2015-10-22 2016-10-13 データ伝送方法及び装置
EP16856846.7A EP3367634B1 (en) 2015-10-22 2016-10-13 Data transmission method and device
US15/959,047 US10735295B2 (en) 2015-10-22 2018-04-20 Data transmission method and apparatus
US16/911,129 US11201810B2 (en) 2015-10-22 2020-06-24 Data transmission method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510696508.4 2015-10-22
CN201510696508.4A CN106612308B (zh) 2015-10-22 2015-10-22 数据传输方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/959,047 Continuation US10735295B2 (en) 2015-10-22 2018-04-20 Data transmission method and apparatus

Publications (1)

Publication Number Publication Date
WO2017067415A1 true WO2017067415A1 (zh) 2017-04-27

Family

ID=58556711

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/101981 Ceased WO2017067415A1 (zh) 2015-10-22 2016-10-13 数据传输方法及装置

Country Status (5)

Country Link
US (2) US10735295B2 (zh)
EP (1) EP3367634B1 (zh)
JP (1) JP2018536226A (zh)
CN (1) CN106612308B (zh)
WO (1) WO2017067415A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114338806A (zh) * 2022-02-28 2022-04-12 湖南云畅网络科技有限公司 一种同步消息处理方法及系统
CN115174651A (zh) * 2022-06-24 2022-10-11 浪潮工业互联网股份有限公司 一种多主机与一从机的通信方法、设备及介质

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106612308B (zh) 2015-10-22 2021-04-16 阿里巴巴集团控股有限公司 数据传输方法及装置
CN110555012B (zh) * 2018-05-14 2022-03-25 杭州海康威视数字技术股份有限公司 数据迁移方法及装置
CN108965402A (zh) * 2018-06-21 2018-12-07 杭州领智云画科技有限公司 基于cdn的缓存方法、装置及系统
CN111221469B (zh) * 2018-11-27 2021-05-11 杭州海康威视数字技术股份有限公司 同步缓存数据的方法、装置和系统
CN110716984B (zh) * 2019-10-10 2023-06-20 北京字节跳动网络技术有限公司 数据处理方法、装置、电子设备及存储介质
CN111245548B (zh) * 2020-01-14 2024-08-20 平安国际智慧城市科技股份有限公司 基于时间戳的数据同步方法、装置和计算机设备
CN113590017B (zh) * 2020-04-30 2023-11-17 伊姆西Ip控股有限责任公司 用于处理数据的方法、电子设备和计算机程序产品
CN111625605B (zh) * 2020-06-02 2024-05-10 北京金堤科技有限公司 信息同步方法和装置、以及存储介质和电子设备
CN113220321B (zh) * 2021-04-13 2023-09-26 山东英信计算机技术有限公司 一种cpld的高效能升级方法、系统及介质
CN113052543B (zh) * 2021-05-31 2022-05-03 广州宸祺出行科技有限公司 一种区域订单和司机热力的统计储存方法、装置及应用
CN113723813A (zh) * 2021-08-31 2021-11-30 平安普惠企业管理有限公司 业绩排行方法、装置、电子设备及可读存储介质
CN114328427A (zh) * 2021-12-27 2022-04-12 未来电视有限公司 数据同步方法、装置、服务器及存储介质
CN116192759B (zh) * 2022-12-13 2024-08-30 网易（杭州）网络有限公司 数据传输量的调整方法、装置、电子设备及存储介质
CN116074282A (zh) * 2023-01-17 2023-05-05 北京有竹居网络技术有限公司 数据报文的响应方法、装置、电子设备及存储介质
CN119149643B (zh) * 2024-11-14 2025-03-18 腾讯科技（深圳）有限公司 数据同步方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070208918A1 (en) * 2006-03-01 2007-09-06 Kenneth Harbin Method and apparatus for providing virtual machine backup
CN103812849A (zh) * 2012-11-15 2014-05-21 腾讯科技（深圳）有限公司 一种本地缓存更新方法、系统、客户端及服务器
CN104506643A (zh) * 2014-12-30 2015-04-08 北京奇虎科技有限公司 分布式存储系统的服务器及服务器之间的数据同步方法
CN104967640A (zh) * 2014-07-31 2015-10-07 腾讯科技（深圳）有限公司 一种数据存储方法、装置和系统

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6701329B1 (en) * 2000-09-14 2004-03-02 Microsoft Corporation Aging and scavenging of DNS resource records
US7500020B1 (en) * 2003-12-31 2009-03-03 Symantec Operating Corporation Coherency of replicas for a distributed file sharing system
US8606926B2 (en) * 2006-06-14 2013-12-10 Opendns, Inc. Recursive DNS nameserver
US8621108B2 (en) * 2008-05-08 2013-12-31 Dialogic Corporation System and method for monitoring user interface connectivity state
US20090282169A1 (en) * 2008-05-09 2009-11-12 Avi Kumar Synchronization programs and methods for networked and mobile devices
JP4939568B2 (ja) * 2009-04-28 2012-05-30 インターナショナル・ビジネス・マシーンズ・コーポレーション データベース間でデータを同期するための方法、並びにそのコンピュータ・システム及びコンピュータ・プログラム
CN101867605A (zh) 2010-05-21 2010-10-20 中兴通讯股份有限公司 提高rss业务同步效率的方法、装置和系统
US11200287B2 (en) * 2011-09-30 2021-12-14 Google Llc Global address list
US20150006485A1 (en) * 2013-06-26 2015-01-01 Eric Alan Christiansen High Scalability Data Management Techniques for Representing, Editing, and Accessing Data
US9509747B2 (en) * 2014-01-23 2016-11-29 Dropbox, Inc. Content item synchronization by block
CN106612308B (zh) 2015-10-22 2021-04-16 阿里巴巴集团控股有限公司 数据传输方法及装置
US10530852B2 (en) * 2016-05-19 2020-01-07 Level 3 Communications, Llc Network mapping in content delivery network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070208918A1 (en) * 2006-03-01 2007-09-06 Kenneth Harbin Method and apparatus for providing virtual machine backup
CN103812849A (zh) * 2012-11-15 2014-05-21 腾讯科技（深圳）有限公司 一种本地缓存更新方法、系统、客户端及服务器
CN104967640A (zh) * 2014-07-31 2015-10-07 腾讯科技（深圳）有限公司 一种数据存储方法、装置和系统
CN104506643A (zh) * 2014-12-30 2015-04-08 北京奇虎科技有限公司 分布式存储系统的服务器及服务器之间的数据同步方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114338806A (zh) * 2022-02-28 2022-04-12 湖南云畅网络科技有限公司 一种同步消息处理方法及系统
CN114338806B (zh) * 2022-02-28 2022-06-21 湖南云畅网络科技有限公司 一种同步消息处理方法及系统
CN115174651A (zh) * 2022-06-24 2022-10-11 浪潮工业互联网股份有限公司 一种多主机与一从机的通信方法、设备及介质
CN115174651B (zh) * 2022-06-24 2023-10-27 浪潮工业互联网股份有限公司 一种多主机与一从机的通信方法、设备及介质

Also Published As

Publication number Publication date
US20180241658A1 (en) 2018-08-23
US11201810B2 (en) 2021-12-14
US20200328959A1 (en) 2020-10-15
EP3367634B1 (en) 2020-07-22
EP3367634A4 (en) 2019-07-17
CN106612308A (zh) 2017-05-03
CN106612308B (zh) 2021-04-16
EP3367634A1 (en) 2018-08-29
US10735295B2 (en) 2020-08-04
JP2018536226A (ja) 2018-12-06

Similar Documents

Publication Publication Date Title
WO2017067415A1 (zh) 数据传输方法及装置
CN104104707B (zh) 网页数据更新方法、终端、缓存服务器及系统
CN110377570B (zh) 节点切换方法、装置、计算机设备及存储介质
CN104301161B (zh) 业务质量指标的计算方法、计算装置以及通信系统
CN107181686B (zh) 路由表的同步方法、装置及系统
US9596313B2 (en) Method, terminal, cache server and system for updating webpage data
CN111694836B (zh) 遥测数据错误检测
AU2015213307A1 (en) Method for setting heartbeat timer, terminal and server
US20210326211A1 (en) Data backup method, apparatus, and system
KR20140145972A (ko) 단말 및 그 단말에서 애플리케이션 동기화 방법
TW201737107A (zh) 資料的同步方法和裝置
US20130080818A1 (en) Conversion of timestamps between multiple entities within a computing system
CN110196780B (zh) 确定服务器状态的方法、装置、存储介质和电子装置
CN103139820A (zh) 一种链路检测方法及网元
CN111130951B (zh) 设备状态检测方法、装置及存储介质
CN110830138A (zh) 一种信息处理方法、电子设备及服务器
CN113472469B (zh) 一种数据同步方法、装置、设备及存储介质
TWI738709B (zh) 資料傳輸方法及裝置
WO2012071810A1 (zh) 网管性能非粒度数据快速存储的方法及装置
JPWO2014199568A1 (ja) 永続記憶装置へのデータ書込制御方法
CN103117883B (zh) 一种分组设备运行状态同步方法
CN107038195B (zh) 数据同步方法和装置
CN109474442B (zh) 日志处理方法、电子设备及存储介质
CN115150933A (zh) 一种切片确定方法、装置和存储介质
CN114331785A (zh) 基于以太坊的行程监管方法、系统及可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16856846

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2018520613

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016856846

Country of ref document: EP