CN114390117B - High-speed continuous data stream storage processing device and method based on FPGA - Google Patents
High-speed continuous data stream storage processing device and method based on FPGA Download PDFInfo
- Publication number
- CN114390117B CN114390117B CN202111448601.5A CN202111448601A CN114390117B CN 114390117 B CN114390117 B CN 114390117B CN 202111448601 A CN202111448601 A CN 202111448601A CN 114390117 B CN114390117 B CN 114390117B
- Authority
- CN
- China
- Prior art keywords
- data
- framing
- storage
- file
- segment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 title claims abstract description 32
- 238000013500 data storage Methods 0.000 claims abstract description 100
- 238000009432 framing Methods 0.000 claims abstract description 79
- 230000005540 biological transmission Effects 0.000 claims abstract description 43
- 239000005441 aurora Substances 0.000 claims description 16
- 102100032306 Aurora kinase B Human genes 0.000 claims description 12
- 108090000749 Aurora kinase B Proteins 0.000 claims description 12
- 108090000461 Aurora Kinase A Proteins 0.000 claims description 9
- 102100026630 Aurora kinase C Human genes 0.000 claims description 8
- 108090000805 Aurora kinase C Proteins 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 8
- 230000001502 supplementing effect Effects 0.000 claims description 8
- 238000003672 processing method Methods 0.000 claims description 7
- 102100032311 Aurora kinase A Human genes 0.000 claims description 5
- 102000004000 Aurora Kinase A Human genes 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 abstract description 2
- 230000000630 rising effect Effects 0.000 description 6
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009469 supplementation Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a high-speed continuous data stream storage processing device and method based on an FPGA, and belongs to the field of data stream storage processing. The method is used for finishing three-level framing of parallel data streams in the face of multipath parallel high-speed data, and the data after framing can finish data searching according to three sizes of files, data segments and data blocks; the method adopts an FPGA full parallel scheme to group frame data streams to finish the data stream processing process; the method provides a high-speed transmission port data transmission control scheme for the high-speed data stream after framing, completes on-line switching of the data storage board through interactive control between the data stream framing buffer end and the data storage end, and ensures continuous data storage processing.
Description
Technical Field
The invention belongs to the field of data stream storage processing, and particularly relates to a high-speed continuous data stream storage processing device and method based on an FPGA.
Background
The prior data stream framing method based on FPGA utilizes the time information in the original data and GPS module to combine with the second pulse to complete framing, the data stream contains the original data and the time information, and the framing mode is a single-stage framing mode. The additional information contained in the data stream is limited, and the data stream is formed by adopting a single-stage mode group frame data stream, so that flexible searching cannot be performed on the data, and the data query efficiency is low. The DDR cache data is adopted, the data is finally stored in a hard disk, and when a plurality of memory modules are carried in a system by expanding the memory space of the system, an effective control mechanism is often lacked, so that the data can be continuously stored at high speed in the process of switching the memory modules.
High-speed continuous data stream storage processing is a key technology in the field of data storage processing.
With the rapid development of communication technology, data with higher sampling rate needs to be acquired and processed, and the instantaneous data processing rate needs to be matched with the processing speed of tens of Gb per second. How to complete the data stream framing of the high-speed parallel data ensures the flexibility of the subsequent data search, and continuously stores the high-speed data stream becomes an important subject in the field of data processing business. In order to improve flexibility and efficiency of subsequent data searching, optimization of data stream framing is needed to meet the requirement of flexible retrieval of data blocks. In addition, in order to store high-speed continuous data, a reliable data storage processing scheme needs to be designed, so that the data is ensured to be reliably and continuously stored in the storage module.
In the prior art, the data stream framing principle based on the FPGA is shown in fig. 1, time information is obtained from a time system such as a GPS module based on the FPGA algorithm, time synchronization is performed by using second pulses, whole second time information is obtained, and then decimal second time information is generated by using a system clock. The data stream is generated in the framing format of fig. 1 in combination with the original data and the time data. The data stream generated by the scheme is analyzed, time information can be analyzed in the frame header, and the first data point and the subsequent data of the data stream collected at the corresponding moment can be analyzed after the frame header.
And writing the data generated by framing into a storage module to finish data storage, and writing the data stream into a DDR module for caching in order to adapt to the variable data writing rate of the storage module. And finally, caching the data from the DDR module, and writing the data into the data storage module to finish data storage.
In the prior art, a single-stage mode is adopted to complete data stream framing, and in the subsequent data query process, the mode of searching the interested data is single, so that the efficiency is low. Meanwhile, the data flow lacks parameter information such as sampling rate, calibration information and the like related to the acquisition channel, and the analyzable information is limited in the subsequent processing process of the data. When the data is continuously stored, a control scheme with good real-time performance is lacked when a plurality of storage modules store the data, and the continuity of the data storage in the process of switching the storage units on line is controlled.
The invention provides a multi-stage data stream framing scheme, which adopts three-stage data (file, data segment and data block) to group frame data streams, wherein the file, the data segment header and the data block header contain rich header information, the data analyzability is good, the data searching can be flexibly carried out according to various lengths, and the data query efficiency is improved; the data stream framing and data storage processing are completed by adopting a fully parallel FPGA method, so that the real-time performance is high and the speed is high; the control scheme of the high-speed data continuous storage processing is designed aiming at the high-speed data storage processing, so that the online switching of the storage modules can be conveniently realized, and the continuity of the data in the switching process of the storage modules is ensured.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a high-speed continuous data stream storage processing device and method based on an FPGA, which are reasonable in design, overcome the defects in the prior art and have good effects.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a high-speed continuous data stream storage processing device based on an FPGA comprises the FPGA, a DDR data buffer module, a high-speed transmission port and four data storage boards;
the FPGA is configured to complete data stream framing and a high-speed port data transmission control algorithm;
a DDR data cache module configured to complete a cache of data to adapt to varying data storage speeds of the memory board;
a high-speed transmission port, including Aurora1, aurora2, aurora3, aurora4, configured to realize high-speed transmission of the data stream from the data stream framing end to the data stream storage end;
and the data storage board is configured for completing data stream storage.
In addition, the invention also provides a high-speed continuous data stream storage processing method based on the FPGA, which adopts the high-speed continuous data stream storage processing device based on the FPGA and comprises the following steps:
step 1: third-stage framing of data streams;
step 2: caching data;
step 3: AURORA data is stored and processed continuously at a high speed.
Preferably, in step 1, firstly, a data stream is framed according to a preset file format by using a file header, a data segment header, a data block header and multiple paths of parallel data; the data stream is framed according to three stages, namely a file, a data segment and a data block; the data stream is composed of a plurality of files; a single file contains file header data and file data; the file data consists of M data segments, wherein a single data segment comprises a data segment head and data segment data; the single data segment data consists of N data blocks, wherein the single data block comprises a data block head and data block data; the data block is composed of Q data;
setting the number of data segments in a single file as M, the number of data blocks in the single data segment as N, the number of data blocks in the single data block as Q, analyzing time data and geographic information data by using an FPGA algorithm after framing, registering acquisition channel parameter data, sequentially caching the data started at the moment into a FIFO with 256 bits wide, framing a file head according to the format of the file head, filling data which is expected to be stored in filling positions, filling the filling positions with the filling data uniformly as 1, wherein the number of the data bits of the file head after filling is an integral multiple of 256, and temporarily setting the transmission of 256 and 1 period; then generating a data segment head, wherein the filling position of the data segment head is filled with data which is expected to be stored, the filling position is used for supplementing the data, the unified filling is all 1, the number of bits of the data segment head after supplementing is an integer multiple of 256, the number of bits of the data segment head after supplementing is temporarily set to 512, and the data is transmitted in 2 clock cycles; next, a data block header is generated, the data block header valid flag is its core data, and the flag is in a stable sampling period, and has usability. Filling the data to be stored in the filling position of the data block head, wherein the filling position is used for filling the data, the filling position is uniformly filled with all 1, the number of the data bits of the data block head after filling is an integer multiple of 256, and the data block head is temporarily transmitted for 256 and 1 period; finally, 256-bit parallel data are read out from the cache FIFO, the data blocks are assembled, and the number of the completed data blocks is increased by 1; when the framing number of the data blocks does not reach N, continuing framing the data block head and the framing data blocks until the framing of a single data segment is completed, and adding 1 to the number of the data segments when the framing of the single data segment is completed; then, the process of framing the data segments is circulated; when the framing number of the data segments does not reach M, continuing framing the data segment heads and the framing data segments until framing of all the data segments in the file is completed, and adding 1 to the number of the completed file; and then start framing of the next file.
Preferably, in step 1, the data stream framing adopts a three-level structure, and hierarchical searching can be completed by searching the file header, the data segment header and the data block header; by searching file type data in the file header, namely realizing file-level data search, the file IDs identify different files; the data segment head type is searched, namely data search at the data segment level is realized, and different data segments in the file are identified by the data segment head ID; in the same data segment, different data blocks in the data segment can be distinguished by searching the data block ID data in the data segment header; based on this principle, data querying can be done at three levels of files, data segments and data blocks.
Preferably, in step 2, the framed data stream employs double DDR4 buffering.
Preferably, in step 3, the data stream is gated by the AURORA data transmission control module, and then is transmitted to the data storage board through the high-speed transmission ports AURORA1, AURORA2, AURORA3, AURORA4, respectively, to complete data storage.
Preferably, in step 3, the AURORA data transmission control module and the data storage board exchange information through a signal line to control continuous data storage.
Preferably, there are two signal lines per data storage board.
Preferably, only one data storage board is operated online within the same period of time, and the maximum data storage capacity is the sum of the storage spaces of four data storage boards.
Preferably, the data tails of the high-speed transmission ports Aurora1, aurora2, aurora3 and Aurora4 are respectively data tails 1,2,3 and 4, and when the data storage board receives the corresponding data tails, the data transmission process to the data storage board is completed; setting the maximum value of the available storage space of the data storage board, wherein the maximum value is smaller than the maximum storage space of the data storage board, reserving allowance, and ensuring that the rest unfinished transmitted data in the high-speed port Aurora can be completely stored in the switching process of the data storage board; and if the available storage space of the currently working data storage board reaches the maximum value of the available storage space, switching data storage to the next data storage board.
The invention has the beneficial technical effects that:
(1) The data query efficiency is high: the three-level data stream framed by the method can flexibly search data according to three sizes of files, data segments and data blocks, and has high data query efficiency;
(2) The data stream processing speed is high: the method adopts a full parallel mode to process data, and the data is realized by adopting a full FPGA parallel scheme in the processes of framing, data caching and data storage, so that the storage processing speed is high;
(3) The multiple storage modules in the system can switch work on line: the data transmission control scheme of the high-speed transmission port is used for controlling the on-line switching of the data storage plate, so that the working efficiency is high, and the continuous storage of data is ensured.
Drawings
FIG. 1 is a schematic diagram of a prior art FPGA-based data stream framing;
FIG. 2 is a general block diagram of the method of the present invention;
fig. 3 is a block diagram of a file header, a data segment, and a data block header.
Detailed Description
The invention is described in further detail below with reference to the attached drawings and detailed description:
the invention provides a high-speed continuous data stream storage processing method based on an FPGA, which sequentially completes three-stage framing of data streams, data caching and AURORA data high-speed continuous storage processing as shown in figure 2. The parallel data sampling rate is 200MSPS, the parallel data is 8 paths, the single path data bit width is 32b at maximum, and the total data bit width is 256 at maximum. The DDR4 user clock adopts 250MHz, the user data processing width is more than 256, and the data processing speed of 51.2Gb/s can be ensured by considering the data processing efficiency. In the method, the total of 8 single Aurora and the single Lane transmission speed is 8Gb/s, and the transmission efficiency is considered, so that the data transmission speed of 51.2Gb/s can be ensured.
Firstly, framing a data stream by using a file header, a data segment header, a data block header and multiple paths of parallel data according to a preset file format. It can be seen from fig. 2 that the data stream is framed in three levels, file, data segment and data block. The data stream is made up of a plurality of files. A single file contains file header data and file data; the file data consists of M data segments, wherein a single data segment comprises a data segment head and data segment data; the single data segment data consists of N data blocks, wherein the single data block comprises a data block head and data block data; the data block is in turn composed of Q data. And caching the parallel data stream generated according to the preset format into the DDR. The data output by the DDR cache is controlled by an AURORA data transmission control module to be continuously stored in four data storage boards. The AURORA data transmission control module utilizes eight signal lines (two of each data storage board) to interact information with the data storage module so as to control the continuous storage of data.
According to the preset format frame data stream of fig. 2, the number of data segments in a single file is set to be M, the number of data blocks in a single data segment is set to be N, and the number of data blocks in a single data block is set to be Q. The format of the header, the data segment and the data block header is shown in fig. 3 (a), 3 (b) and 3 (c). After the framing is started, time data and geographic information data are analyzed by using an FPGA algorithm, data of acquisition channel parameters (such as sampling rate, reference level and the like) are registered, and the data started at the moment are sequentially cached in a FIFO with 256-bit width. The file header is framed according to the format of the file header, the filling position can be used for adding data which is expected to be stored, the data format data is used for supplementing data, and the filling position is used for uniformly filling all 1. The number of bits of the header data after the supplement is an integer multiple of 256, and is tentatively 256, 1-cycle transmission. And then generating a data segment head, wherein filling time data, geographical position data, reference level data, sampling rate and the like of filling positions of the data segment head are used for supplementing data, and the filling positions are uniformly filled to be all 1. The number of bits of the header data of the data segment after supplementation is an integer multiple of 256, which is temporarily designated 512 here, and is transmitted in 2 clock cycles. Next, a data block header is generated, the data block header valid flag is its core data, and the flag is in a stable sampling period, and has usability. The use of padding and reserved locations in the data block header is the same as the data header. The number of bits of the data block header after supplementation is an integer multiple of 256, and is tentatively 256, 1-cycle transmission. And finally, reading 256-bit parallel data from the cache FIFO, framing data blocks, and adding 1 to the number of the completed data blocks. And when the framing number of the data blocks does not reach N, continuing to frame the data block head and the data block until the framing of a single data segment is completed, and adding 1 to the number of the data segments when the framing of the single data segment is completed. And then the process of assembling the frame data segments is looped. And when the framing number of the data segments does not reach M, continuing framing the data segment header and the framing data segments until framing of all the data segments in the file is completed, and adding 1 to the number of the completed file. And then start framing of the next file. Obviously, the bit width of the data stream completed by framing is unified to be 256.
The data stream framing adopts a three-level structure, and hierarchical searching can be completed by searching the file header, the data segment header and the data block header. By searching the file type data (set to FB 746572) in the header, file-level data searching can be achieved, with the file IDs identifying the different files. By searching for the type of the data segment header (set to FB 746572), data searching at the data segment level can be achieved, and the data segment header ID identifies different data segments in the file. In the same data segment, different data blocks in the data segment can be distinguished by searching the data block ID data in the data segment header. Based on this principle, data queries can be done at three levels, file, data segment and data block.
The data flow completed by framing adopts double DDR4 buffer, and the data output by buffer is unified into the clock domain of a high-speed transmission port Aurora 1. Four data storage boards 1,2,3 and 4 are arranged in the system, the data flow is gated by an AURORA data transmission control module, and then the data flow is transmitted to the data storage boards by high-speed transmission ports AURORA1, AURORA2, AURORA3 and AURORA4 to finish data storage. Note that each of fifo_1, fifo_2, fifo_3, and fifo_4 operates in a mode of writing 256 bits wide and reading 512 bits wide.
The collected data is transmitted to the data storage board through the high-speed transmission port, and the transmission bit width of the high-speed transmission port is 512. Only one data storage board works on line within the same period, the working sequence is 1,2,3 and 4, the cycle is realized, and the maximum data storage capacity is the sum of the storage spaces of four data storage boards. The data tails 1,2,3 and 4 of the high-speed transmission ports Aurora1, aurora2, aurora3 and Aurora4 are set to be 16 pieces of 32-bit data 32'h1D, 32' h2D, 32'h3D and 32' h4D spliced together, and the bit width of a single data tail is 512 bits. When a data storage board receives a corresponding data tail, it indicates that the data transfer process to the data storage board is complete. And setting the maximum value of the available storage space of the data storage board, wherein the maximum value is smaller than the maximum storage space of the data storage board, reserving allowance, and ensuring that the rest data which are not transmitted in the high-speed port Aurora in the switching process of the data storage board can be stored completely. And if the available storage space of the currently working data storage board reaches the maximum value of the available storage space, switching data storage to the next data storage board. The continuity of data stream storage processing in the process of switching the data storage boards can be ensured by utilizing the principle.
The specific working principle is as follows: the data stream framing buffer end and the data storage end ensure continuous storage processing of data through information interaction control. The data storage board 1 is connected with a data stream framing buffer end through a signal line 11 and a signal line 12; the data storage board 2 is connected with a data stream framing buffer end through a signal line 21 and a signal line 22; the data storage board 3 is connected with a data stream framing buffer end through a signal line 31 and a signal line 32; the data storage board 4 is connected with a data stream framing buffer through a signal line 41 and a signal line 42. The signal lines 11, 21, 31, 41 are switching signals, and the signal lines 12, 22, 32, 42 are on-line signals.
When the power is applied for the first time, the online data storage board controls the online signals to generate a rising edge, and monitors the rising edges of the four online signals at the data stream framing buffer end, for example, monitors that the signal line 12 generates the rising edge, so that the current online data storage board 1 can be judged, and the control data is transmitted and stored to the data storage board 1 through Aurora1, and the other same principle. Here, the first online working data storage board after power-up by default is the data storage board 1. At this time, the line signal line 12 is 1, and the switching signal line 11 is 0. With the continuous use of fifo_1, data is transmitted to the data storage board 1 through the high-speed transmission port Aurora1 and stored, and when the data stream satisfies the switching condition from the data storage board 1 to the data storage board 2, the data storage board 1 side immediately controls the switching signal line 11 to generate a rising edge. At the position of the rising edge of the switching signal line 11, the AURORA data selector is controlled by the AURORA data transmission control module to transmit control data to the data storage board 2 through the high-speed transmission port AURORA2, and the on-line signal line 22 is set to 1. At the same time, from the occurrence of the rising edge of the switching signal line 11, the empty flag 1 of fifo_1 is monitored, and when the empty flag 1 of fifo_1 is valid, the data tail 1 is inserted in the data stream. Thus, a data tail is inserted into the data transmitted to the data storage board 1 through the aurora 1. At the data storage board 1 side, the monitor data tail 1 is received, and when the data is monitored, the on-line signal line 12 is set to 0 (note that the data tail 1 is not data in the data stream that needs to be stored, and thus the data storage board 1 does not store the data), indicating that the data storage board 1 is finished working. The data storage board 1 is operated for a period from when the on-line signal line 12 is set to 1 to when the end of the data tail 1 is monitored. The same applies to the operation principle of switching data from storage to data storage board 2 to data storage board 3, from storage to data storage board 3 to data storage board 4, and from storage to data storage board 4 to data storage board 1. With the above control scheme, the system can store data to a plurality of data storage boards continuously for storing high-speed data, and can switch the data storage boards on line.
The method is used for finishing three-level framing of parallel data streams in the face of multipath parallel high-speed data, and the data after framing can finish data searching according to three sizes of files, data segments and data blocks; the method adopts an FPGA full parallel scheme to group frame data streams to finish the data stream processing process; the method provides a high-speed transmission port data transmission control scheme for the high-speed data stream after framing, completes on-line switching of the data storage board through interactive control between the data stream framing buffer end and the data storage end, and ensures continuous data storage processing.
It should be understood that the above description is not intended to limit the invention to the particular embodiments disclosed, but to limit the invention to the particular embodiments disclosed, and that the invention is not limited to the particular embodiments disclosed, but is intended to cover modifications, adaptations, additions and alternatives falling within the spirit and scope of the invention.
Claims (5)
1. A high-speed continuous data stream storage processing method based on FPGA is characterized in that: the device comprises an FPGA, a DDR data buffer module, a high-speed transmission port and four data storage boards;
the FPGA is configured to complete data stream framing and a high-speed port data transmission control algorithm;
a DDR data cache module configured to complete a cache of data to adapt to varying data storage speeds of the memory board;
a high-speed transmission port, including Aurora1, aurora2, aurora3, aurora4, configured to realize high-speed transmission of the data stream from the data stream framing end to the data stream storage end;
a data storage board configured to complete data stream storage; the method comprises the following steps:
step 1: third-stage framing of data streams;
step 2: caching data;
step 3: AURORA data high-speed continuous storage processing;
in step 1, firstly, framing a data stream by using a file header, a data segment header, a data block header and multiple paths of parallel data according to a preset file format; the data stream is framed according to three stages, namely a file, a data segment and a data block; the data stream is composed of a plurality of files; a single file contains file header data and file data; the file data consists of M data segments, wherein a single data segment comprises a data segment head and data segment data; the single data segment data consists of N data blocks, wherein the single data block comprises a data block head and data block data; the data block is composed of Q data;
setting the number of data segments in a single file as M, the number of data blocks in the single data segment as N, the number of data blocks in the single data block as Q, analyzing time data and geographic information data by using an FPGA algorithm after framing, registering acquisition channel parameter data, sequentially caching the data started at the moment into a FIFO with 256 bits wide, framing a file head according to the format of the file head, filling data which is expected to be stored in filling positions, filling the filling positions with the filling data uniformly as 1, wherein the number of the data bits of the file head after filling is an integral multiple of 256, and temporarily setting the transmission of 256 and 1 period; then generating a data segment head, wherein the filling position of the data segment head is filled with data which is expected to be stored, the filling position is used for supplementing the data, the unified filling is all 1, the number of bits of the data segment head after supplementing is an integer multiple of 256, the number of bits of the data segment head after supplementing is temporarily set to 512, and the data is transmitted in 2 clock cycles; then, generating a data block head, wherein a data block head effective mark is core data of the data block head effective mark, and the data is marked to be in a stable sampling period and has usability; filling the data to be stored in the filling position of the data block head, wherein the filling position is used for filling the data, the filling position is uniformly filled with all 1, the number of the data bits of the data block head after filling is an integer multiple of 256, and the data block head is temporarily transmitted for 256 and 1 period; finally, 256-bit parallel data are read out from the cache FIFO, the data blocks are assembled, and the number of the completed data blocks is increased by 1; when the framing number of the data blocks does not reach N, continuing framing the data block head and the framing data blocks until the framing of a single data segment is completed, and adding 1 to the number of the data segments when the framing of the single data segment is completed; then, the process of framing the data segments is circulated; when the framing number of the data segments does not reach M, continuing framing the data segment heads and the framing data segments until framing of all the data segments in the file is completed, and adding 1 to the number of the completed file; then starting framing of the next file;
in the step 1, the data stream framing adopts a three-level structure, and hierarchical searching can be completed by searching a file header, a data segment header and a data block header; by searching file type data in the file header, namely realizing file-level data search, the file IDs identify different files; the data segment head type is searched, namely data search at the data segment level is realized, and different data segments in the file are identified by the data segment head ID; in the same data segment, different data blocks in the data segment can be distinguished by searching the data block ID data in the data segment header; based on the principle, the data query can be completed at three levels of files, data segments and data blocks;
in the step 2, the data stream after framing is buffered by adopting double DDR 4;
in step 3, the data stream is gated by the AURORA data transmission control module, and then is transmitted to the data storage board through the high-speed transmission ports AURORA1, AURORA2, AURORA3 and AURORA4 respectively to finish data storage.
2. The FPGA-based high-speed continuous data stream storage processing method according to claim 1, wherein: in step 3, the AURORA data transmission control module and the data storage board interact information through a signal line to control the continuous storage of data.
3. The FPGA-based high-speed continuous data stream storage processing method according to claim 1, wherein: each data storage board has two signal lines.
4. The FPGA-based high-speed continuous data stream storage processing method according to claim 1, wherein: only one data storage board works on line in the same period, and the maximum data storage capacity is the sum of storage spaces of four data storage boards.
5. The FPGA-based high-speed continuous data stream storage processing method according to claim 1, wherein: the data tails of the high-speed transmission ports Aurora1, aurora2, aurora3 and Aurora4 are respectively data tails 1,2,3 and 4, and when the data storage board receives the corresponding data tails, the data transmission process to the data storage board is completed; setting the maximum value of the available storage space of the data storage board, wherein the maximum value is smaller than the maximum storage space of the data storage board, reserving allowance, and ensuring that the rest unfinished transmitted data in the high-speed port Aurora can be completely stored in the switching process of the data storage board; and if the available storage space of the currently working data storage board reaches the maximum value of the available storage space, switching data storage to the next data storage board.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111448601.5A CN114390117B (en) | 2021-12-01 | 2021-12-01 | High-speed continuous data stream storage processing device and method based on FPGA |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111448601.5A CN114390117B (en) | 2021-12-01 | 2021-12-01 | High-speed continuous data stream storage processing device and method based on FPGA |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114390117A CN114390117A (en) | 2022-04-22 |
| CN114390117B true CN114390117B (en) | 2023-08-22 |
Family
ID=81195710
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111448601.5A Active CN114390117B (en) | 2021-12-01 | 2021-12-01 | High-speed continuous data stream storage processing device and method based on FPGA |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114390117B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116483916B (en) * | 2023-02-14 | 2024-06-21 | 中国人民解放军63811部队 | High-frequency real-time code stream storage and data synchronous retrieval system based on sub-segment splitting |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2328230A1 (en) * | 1995-07-14 | 1997-02-06 | Oracle Corporation | Method for creating a digital data stream |
| CN103078727A (en) * | 2013-01-06 | 2013-05-01 | 中国电子科技集团公司第十研究所 | Method for improving frame synchronization performance of data transmission receiver |
| CN103873874A (en) * | 2014-02-19 | 2014-06-18 | 同观科技（深圳）有限公司 | Full search motion estimation method based on programmable parallel processor |
| CN105138476A (en) * | 2015-08-26 | 2015-12-09 | 广东创我科技发展有限公司 | Data storage method and system based on hadoop heterogeneous storage |
| CN111163077A (en) * | 2019-12-26 | 2020-05-15 | 国家计算机网络与信息安全管理中心 | A system and method for realizing multi-dimensional continuous mask based on network processor |
| CN112100119A (en) * | 2020-08-18 | 2020-12-18 | 中国科学院声学研究所 | High-speed Ethernet frame reconstruction system based on FPGA |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8725707B2 (en) * | 2009-03-26 | 2014-05-13 | Hewlett-Packard Development Company, L.P. | Data continuous SQL process |
| US20160275146A1 (en) * | 2013-10-04 | 2016-09-22 | Koutin Technology Inc. Ltd | Use a parallel hardware search device to implement big databases efficiently |
-
2021
- 2021-12-01 CN CN202111448601.5A patent/CN114390117B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2328230A1 (en) * | 1995-07-14 | 1997-02-06 | Oracle Corporation | Method for creating a digital data stream |
| CN103078727A (en) * | 2013-01-06 | 2013-05-01 | 中国电子科技集团公司第十研究所 | Method for improving frame synchronization performance of data transmission receiver |
| CN103873874A (en) * | 2014-02-19 | 2014-06-18 | 同观科技（深圳）有限公司 | Full search motion estimation method based on programmable parallel processor |
| CN105138476A (en) * | 2015-08-26 | 2015-12-09 | 广东创我科技发展有限公司 | Data storage method and system based on hadoop heterogeneous storage |
| CN111163077A (en) * | 2019-12-26 | 2020-05-15 | 国家计算机网络与信息安全管理中心 | A system and method for realizing multi-dimensional continuous mask based on network processor |
| CN112100119A (en) * | 2020-08-18 | 2020-12-18 | 中国科学院声学研究所 | High-speed Ethernet frame reconstruction system based on FPGA |
Non-Patent Citations (1)
| Title |
|---|
| 高速串行数据处理模块的设计与实现;徐健;侯振龙;龚东磊;方明;;计算机工程(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114390117A (en) | 2022-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103885887B (en) | User data storage method, read method and system | |
| CN114390117B (en) | High-speed continuous data stream storage processing device and method based on FPGA | |
| CN107832343B (en) | A method for fast data retrieval based on bitmap-based MBF data index structure | |
| CN113839835A (en) | A Top-k Stream Accurate Monitoring Architecture Based on Small Stream Filtering | |
| CN106649627A (en) | Log searching method and device | |
| CN102053923A (en) | Storage method and storage device for logbook data | |
| CN104407998B (en) | High-speed ADC (Analog to Digital Converter) sampled data receiving and buffering method and system based on FPGA (Field Programmable Gate Array) | |
| CN102843298A (en) | Method and system for achieving priority of Openflow switchboard chip flow tables | |
| CN107094064B (en) | Viterbi demodulation system and method for octal continuous phase frequency shift keying | |
| CN102508803A (en) | Matrix transposition memory controller | |
| CN103957139A (en) | Ten-Gigabit Ethernet frame analyzing method based on FPGA | |
| CN105224485A (en) | A kind of method of reseptance of pervasive serial data and device | |
| CN101162919A (en) | Data caching circuit | |
| CN111813709A (en) | High-speed parallel storage method based on FPGA (field programmable Gate array) storage and calculation integrated framework | |
| CN107943738A (en) | Clock data recovery circuit and implementation method | |
| CN116048453B (en) | Multichannel triggering asynchronous scheduling system and method based on FPGA | |
| CN113726499A (en) | Digital sampling data high-speed interpolation synchronization method and system | |
| CN115113820B (en) | A system and method for implementing a multi-target long-delay DRFM storage component | |
| CN105446699A (en) | Data frame queue management method | |
| CN111611180A (en) | A Dynamic Shared Buffer Supporting Multiple Protocols | |
| CN101494506B (en) | Circuit for capturing running water structure underwater sound spread-spectrum communication bandpass signal | |
| CN103309981A (en) | A system and method for organizing ADC data with high storage efficiency | |
| CN109726158A (en) | A kind of multi-channel serial data recovery method and device | |
| WO2025044545A1 (en) | Data selection device for sram array, and storage system and system-on-chip | |
| CN103646056B (en) | Method for storing and extracting historical data based on characteristic value storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |