CN115346605A - Gene data processing method and device under high-throughput sequencing background and related equipment - Google Patents

Abstract

The present application discloses a genetic data processing method, device and related equipment under the background of high-throughput sequencing. In the reserved area; based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain the gene data compression block, and the gene data compression block is retained in the second reserved area of the first memory, and from Release gene data blocks in the first memory; when the number of gene data compression blocks in the first memory reaches J, output J gene data compression blocks to the second memory and release them from the first memory. This application processes the gene data blocks output by the sequencing platform in parallel in a streaming manner, which not only saves waiting time, but also improves the utilization efficiency of computing resources, and improves the overall speed from the off-machine of the sequencing platform data to the analysis of biological information.

Description

Gene data processing method, device and related equipment under the background of high-throughput sequencing

technical field

This application relates to the field of sequencing-based technologies, and more specifically, relates to a genetic data processing method, device and related equipment under the background of high-throughput sequencing.

Background technique

As an important means to explore the mysteries of life, gene sequencing technology has become an important branch of bioinformatics research. It has been widely used in species identification, gene detection, and disease diagnosis. The rapid development of gene sequencing technology has laid a solid foundation for precision medicine. The basics.

With the development of gene sequencing technology, the cost of sequencing is getting lower and lower, resulting in a wider and wider scale of sequencing business. With the continuous increase of the sequencing business scale, the equipment types, business capacity, and network structure of the networking are becoming more and more complex. How to process more sequencing services in a timely manner and improve the delivery efficiency of biological information sequencing under the existing computing resources and storage resources is a technical issue worth exploring.

Contents of the invention

In view of this, the present application provides a gene data processing method, device and related equipment under the background of high-throughput sequencing, which realizes highly concurrent stream processing of gene sequencing data, thereby improving the delivery efficiency of biological information sequencing.

In order to achieve the above purpose, the first aspect of the present application provides a genetic data processing method in the context of high-throughput sequencing, including:

When the available space in the first reserved area of the first memory reaches a preset capacity, acquiring a genetic data block of a preset size, and inputting the genetic data block into the first reserved area of the first memory , wherein the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

Based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compression block, retain the gene data compression block in the second reserved area of the first memory, and releasing the gene data block from the first memory, and the second reserved area has the ability to accommodate M gene data compression blocks;

After the number of gene data compression blocks in the first memory reaches J, output J gene data compression blocks to the second memory, and release the J gene data compression blocks from the first memory;

Wherein, N, M, and J are all preset natural numbers, and J is not greater than M.

Preferably, the short sequence includes metadata, base data and quality data; based on the data characteristics of each short sequence in the genetic data block, the process of compressing the genetic data block includes:

comparing the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the comparison result;

Compressing the metadata in the genetic data block by using incremental encoding technology or run-length encoding technology;

determining the complexity of the quality data through a preset adaptive model, and determining a context statistical model of a first target order based on the complexity;

Compressing the quality data by using the context statistical model of the first target order to obtain a first intermediate compression result;

The first intermediate compression result is compressed by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

Preferably, the process of comparing the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the comparison result includes:

Divide the base data in the gene data block into multiple subsequences;

Comparing each subsequence with a preset reference genome using a hash comparison method to obtain matching information for each subsequence, the matching information including a mismatch value;

For a subsequence whose mismatch value is less than or equal to a preset threshold, compress the subsequence based on the matching information of the subsequence;

For subsequences with mismatch values greater than a preset threshold:

determining the complexity of the subsequence through a preset adaptive model, and determining a context statistical model of a second target order based on the complexity;

compressing the subsequence by using the context statistical model of the second target order to obtain a second intermediate compression result;

The second intermediate compression result is compressed by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

Preferably, the process of comparing each subsequence with a preset reference genome using the hash comparison method to obtain the matching information of each subsequence includes:

Using the hash value of each subsequence as the query condition, query in the preset hash table to obtain the matching information of each subsequence;

Wherein, the preset hash table records the hash value of each reference subsequence in the reference genome and the position information of each reference subsequence in the reference genome, and each reference subsequence is obtained from the derived from the reference genome.

Preferably, the process of obtaining each reference subsequence from the division of the reference genome includes:

A plurality of overlapping reference subsequences of length K are divided from the reference genome with a preset step size, wherein K is a preset length value.

Preferably, the process of comparing the base data in the gene data block with a preset reference genome includes:

The base data in the gene data block is compared with a preset reference genome using a BWT algorithm.

Preferably, it also includes:

Obtaining the processing speed of the first operation, the first operation comprising: obtaining a genetic data block of a preset size, and inputting it into a first reserved area of the first memory;

Obtaining the processing speed of the second operation, the second operation comprising: compressing the gene data block based on the data characteristics of each short sequence in the gene data block;

Based on the processing speed of the first operation and the processing speed of the second operation, computing resources allocated to the first operation and the second operation are determined.

The second aspect of the present application provides a genetic data processing device under the background of high-throughput sequencing, including:

A data block acquiring unit, configured to acquire a gene data block of a preset size when the available space in the first reserved area of the first memory reaches a preset capacity, and input the gene data block into the first memory In the first reserved area, the genetic data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N genetic data blocks, and the preset size is not greater than said preset capacity;

A data block processing unit, configured to compress the gene data block based on the data characteristics of each short sequence in the gene data block to obtain a gene data compression block, and retain the gene data compression block in the first memory. In the second reserved area, and release the gene data block from the first memory, the second reserved area has the ability to accommodate M gene data compression blocks;

A data block deriving unit, configured to output J gene data compression blocks to the second memory after the number of gene data compression blocks in the first memory reaches J, and release the J gene data from the first memory compressed block;

Wherein, N, M, and J are all preset natural numbers, and J is not greater than M.

The third aspect of the present application provides a genetic data processing device in the context of high-throughput sequencing, including: a memory and a processor;

The memory is used to store programs;

The processor is configured to execute the program to realize each step of the above-mentioned gene data processing method under the background of high-throughput sequencing.

The fourth aspect of the present application provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, various steps of the above-mentioned gene data processing method under the background of high-throughput sequencing are realized.

It can be seen from the above-mentioned technical solution that when the available space in the first reserved area of the first memory reaches the preset capacity, the present application acquires a genetic data block of a preset size, and inputs the genetic data block into the first In the first reserved area of a memory. Wherein, the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size is not greater than the preset capacity. Next, based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compression block, and reserve the gene data compression block in the second reserved area of the first memory , and release the gene data block from the first memory at the same time. Wherein, the second reserved area is capable of accommodating M compressed blocks of genetic data. It can be understood that the process of acquiring and placing the gene data block and the process of compressing the gene data block are performed in a cycle. That is, the gene data blocks are continuously delivered from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks; at the same time, the first reserved area Each gene data block in is subjected to compression processing, and the obtained gene data compression block is reserved in the second reserved area of the first memory. When the number of compressed genetic data blocks in the first memory reaches J, output J compressed genetic data blocks to the second memory, and release the J compressed genetic data blocks from the first memory. Wherein, N, M, and J are all preset natural numbers, and J is not greater than M. This application realizes the parallel processing of each genetic data block output by the sequencing platform in a streaming manner, without waiting for all the genetic data to be collected before processing. While saving waiting time, it improves the utilization efficiency of computing resources and improves the efficiency of data from the sequencing platform. The overall speed from off-machine to bioinformatics analysis helps to improve the delivery efficiency of bioinformatics sequencing.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

Fig. 1 is a schematic diagram of the genetic data processing method under the background of high-throughput sequencing disclosed in the embodiment of the present application;

FIG. 2 illustrates a schematic diagram of parallel processing of the steps disclosed in the embodiment of the present application;

Fig. 3 is a schematic diagram of the genetic data processing method under the background of high-throughput sequencing disclosed in the embodiment of the present application;

Fig. 4 is a schematic diagram of a genetic data processing device under the background of high-throughput sequencing disclosed in the embodiment of the present application;

Fig. 5 is a schematic diagram of a genetic data processing device under the background of high-throughput sequencing disclosed in the embodiment of the present application;

Fig. 6 is a schematic diagram of genetic data processing equipment under the background of high-throughput sequencing disclosed in the embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application. The following first related nouns are explained:

High-throughput sequencing: Also known as "Next-generation" sequencing technology, it can sequence hundreds of thousands to millions of DNA molecules in parallel and compare the general read length. Short and so on as a sign. Sequencing refers to the analysis of the base sequence of a specific DNA fragment, that is, the arrangement of adenine (A), thymine (T), cytosine (C) and guanine (G). The emergence of rapid DNA sequencing methods has greatly promoted the research and development of biology and medicine.

Base Calling: Identify the base type (DNA sequence) from the original image (row images) through computer vision, write the result to the cal file, and finally generate the sequencing report and FastQ data.

Sequence Alignment: Aligning two or more sequences together to indicate their similarities. Gaps (usually indicated by a dash "-") can be inserted in the sequence. Corresponding identical or similar symbols (A, T (or U), C, G in nucleic acids, single-letter representations of amino acid residues in proteins) are arranged in the same column. Often used to study sequences that have evolved from a common ancestor, especially biological sequences such as protein sequences or DNA sequences. In the alignment, mismatches correspond to mutations, while gaps correspond to insertions or deletions.

Reference genome: refers to the genome sequence of the species, which is the complete genome sequence that has been assembled, and is often used as a standard reference for the species, such as the human genome reference sequence (fasta format).

K-mer: A subsequence of length K in a short sequence.

Networking technology: Networking technology is network building technology, which is divided into Ethernet networking technology and ATM LAN networking technology. Ethernet networking is very flexible and simple. It can use a variety of physical media and network with different topologies. It is the most widely used network at home and abroad, and has become the mainstream of network technology. Ethernet is divided into 10Mb/s, 100Mb/s, and 1000Mb/s according to its transmission rate.

The inventors of the present application found that the existing process from off-board sequencing to bioinformatics analysis includes:

1) The sequencer completes the biochemical and imaging to obtain the base image data, and converts the base image data into a CAL file containing the DNA sequence through Base Calling, and converts the CAL file into a FASTQ text file containing the sequence information through data processing. Data quality control and use compression software to compress FASTQ and store backup to disk.

2) Read the compressed FASTQ file from the disk, compare the sequence with the sequence in the reference genome by sequence comparison software (such as BWA), find the position of each sequence on the reference genome, and then arrange them in order.

3) Subsequent work such as mutation detection and interpretation will be carried out according to the needs of the research. The purpose of variant detection analysis is to accurately detect the set of variants in the genome of each sample (such as human beings), that is, those DNA sequences that differ from person to person.

From the above, we can see that the following problems exist in the current genetic testing process:

1) After Base Calling is completed, the CAL file needs to be transferred to the GPU server for data conversion. After the data conversion, the converted FASTQ file needs to be transferred to the data quality control server for data quality control, and then the quality-controlled data is transferred to The secondary storage server backs up and stores the data, and at the same time transmits it to the computing server for subsequent calculation and analysis. The mixed networking mode of various special equipment is complicated and the cost is extremely high.

2) In the process of data processing and conversion of CAL files into FASTQ, it is necessary to output a complete FASTQ file to the disk after all processing of a CAL file before subsequent data transmission, quality control, backup and analysis, but the whole A CAL file has a large amount of data, and the conversion takes a long time.

3) After the CAL file conversion is completed, the FASTQ file needs to be written to the disk, and then the compression software reads it from the disk for compression; when performing sequence alignment, the comparison software needs to read the FASTQ file from the disk first. Every input and output of data requires I/O on the disk. I/O is frequent and slow, and relies heavily on storage performance.

Since the sequencing files of genetic data range from a few gigabytes to tens of gigabytes to hundreds of gigabytes, if you wait for all the data to be downloaded and then perform subsequent data compression and bioinformatics analysis, computing resources will be idle and not properly utilized The situation, which in turn affects the efficiency of student letter delivery. This application takes the data block as the unit, adopts the pipeline method, and uses the similarity that the gene data compression and sequence comparison process must be compared with the reference genome, and merges the data export and data compression from independent steps. , can improve overall efficiency and make full use of computing resources.

The genetic data processing method under the background of high-throughput sequencing provided by the embodiment of the present application is introduced below. Please refer to Figure 1, the genetic data processing method under the background of high-throughput sequencing provided by the embodiment of the present application may include the following steps:

Step S101, acquiring a genetic data block of a preset size and inputting it into the first reserved area of the first memory.

Among them, the gene data block is a collection of short sequences imported from the sequencing platform in real time, and the first reserved area has the ability to accommodate N gene data blocks. N is a preset natural number

Exemplarily, the first memory may be a memory, especially, the memory here may be a DDR memory. Due to the large genetic data and the limited memory of computing devices, it can usually be expanded to DDR memory. DDR memory transmits data once on the rising and falling edges of the clock signal, which makes the data transmission speed of DDR memory twice that of traditional SDRAM. Moreover, since only the falling edge signal is used more, it does not cause an increase in energy consumption. As for the addressing and control signals, they are the same as traditional SDRAM and are only transmitted on the rising edge of the clock.

The short sequence collection contains multiple short sequences (read), wherein the size of the gene data block can be defined by the number of short sequences, for example, the preset size of the gene data block can be set based on the memory capacity of the computing device 40,000 to 500,000 short sequences.

Step S102, based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compressed block.

Among them, each short sequence in the gene data block can contain the following four lines:

The first line of metadata begins with @, followed by a sequence identifier and an optional description.

The second line of base data is essentially a sequence of letters consisting of five letters: A, C, G, T, and N. This is also the DNA sequence we really care about. N represents those that cannot be identified during sequencing bases.

The third line begins with a + character, sometimes followed by the same sequence identifier as line 1.

The fourth line is the quality data, which represents the quality score and describes the reliability of each base, which is a string of characters composed of ASCII codes.

It should be noted that the lengths of the second row and the fourth row must be equal, and each element corresponds to each other. From the composition of the short sequence, we can know that the metadata in the first line is a general description, which can be compressed by a general compression method; the base data in the second line is rich in biological characteristics, which can be used to compare the reference genome The quality data in the fourth line has a relatively high internal correlation, and can be compressed by run-length coding and other methods.

Step S103, keeping the gene data compressed block in the second reserved area of the first memory, and releasing the gene data block from the first memory.

Wherein, the second reserved area is capable of accommodating M compressed blocks of genetic data, where M is a preset natural number. It should be noted that, generally, the space size of the gene data compression block will be smaller than the space size of the relative gene data block. However, considering that the speed of pulling the gene data block may not be consistent with the calculated speed based on the data compression block Therefore, the size relationship between the capacity N of the first reserved area and the size of the second reserved area M must be further set according to the aforementioned two speeds, so as to avoid wasting space.

Step S104, when the number of compressed genetic data blocks in the first memory reaches J, output J compressed genetic data blocks to the second memory, and release the J compressed genetic data blocks from the first memory.

Wherein, J is a preset natural number, and J is not greater than M. Exemplarily, the second storage may be a hard disk in the computing device, that is, each J data-based compression block is output as a file slice, and finally a complete sequencing file is restored from each file slice.

It can be understood that, referring to FIG. 2, step S101 to step S103 are executed cyclically, that is, step S101 will continuously transfer gene data blocks from the sequencing platform to the first reserved area of the first memory until the first A reserved area cannot accommodate more gene data blocks; step S102 will compress each gene data block in the first reserved area, and retain the obtained gene data compressed blocks in the first memory through step S103 In the second reserved area, the space in the first reserved area released in step S102 and the space in the second reserved area released in step S103 are used to receive new gene data blocks and gene data compression blocks. In this way, for each gene data block, step S101 to step S103 can perform the compression of the sequencing data in a streaming manner; meanwhile, steps S101 to step S103 of each gene data block can be executed in parallel to a high degree.

In this application, when the available space in the first reserved area of the first storage reaches a preset capacity, a genetic data block of a preset size is obtained, and the genetic data block is input into the first reserved area of the first storage. In the area, wherein, the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size is not greater than the preset capacity. Next, based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a gene data compression block, and the gene data compression block is reserved in the second reserved area of the first memory , and release the gene data block from the first memory at the same time. Wherein, the second reserved area is capable of accommodating M compressed blocks of genetic data. It can be understood that the process of acquiring and placing the gene data block and the process of compressing the gene data block are performed in a cycle. That is, the gene data blocks are continuously delivered from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks; at the same time, the first reserved area Each gene data block in is subjected to compression processing, and the obtained gene data compression block is reserved in the second reserved area of the first memory. When the number of compressed genetic data blocks in the first memory reaches J, output J compressed genetic data blocks to the second memory, and release the J compressed genetic data blocks from the first memory. Wherein, N, M, and J are all preset natural numbers, and J is not greater than M. This application realizes the parallel processing of each gene data block output by the sequencing platform in a streaming manner, without waiting for all the gene data to be collected before processing, while saving waiting time, it improves the utilization efficiency of computing resources, and improves the efficiency of data from the sequencing platform. The overall speed from off-machine to bioinformatics analysis helps to improve the delivery efficiency of bioinformatics sequencing.

In some embodiments of the present application, the process of compressing the gene data block based on the data characteristics of each short sequence in the above step S102 may include:

S1, compressing the metadata in the gene data block by using incremental coding technology or run-length coding technology.

Among them, the metadata in the gene data blocks of different sequencing platforms may have different formats. Typically, metadata starts with the symbol "@", followed by a sequence identifier and other optional information, such as instrument name, flow cell channel, block number within the flow cell channel, "x" coordinates of clusters within a block, The 'y' coordinate of the clusters within the tile, the sample number in pooled multisamples, the paired-end identity, and the sequence length.

For the metadata in the genetic data block, they can be parsed into different data segments first using delimiters (punctuation marks). Delimiters generally include dots, spaces, underscores, hyphens, slashes, equal signs and colons; and then Different data segments use identification algorithms to classify the data segments, and do not encode and compress fixed information such as instrument names, sequence lengths, and double-ended identifiers. Other data segments use different compression encoding methods, including incremental encoding, Run-length coding and context hybrid compression algorithms.

For a data segment that only contains continuous integers, use incremental encoding to record the difference between the two data segments before and after; for a data segment that contains letters and numbers, use the improved swim-length encoding to record the characters that appear repeatedly in the data segment before and after, Incremental encoding is used to record the difference between the two data segments before and after.

S2, combining the context statistical model, run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology to compress the quality data in the gene data block.

Specifically, it may include:

S21. Determine the complexity of the quality data in the gene data block through a preset adaptive model, and determine a context statistical model of the first target order based on the complexity.

Wherein, the complexity of the quality score may be 4 quality values, 8 quality values or 40 quality values, etc., based on different quality values, the first target order may be 1st order, 2nd order, 3rd order, 4th order, 5th order order or 6th order. After counting the complexity of the quality data through the preset adaptive model, the order of the context statistical model suitable for the quality data is determined, so that a better compression effect can be obtained.

S22. Compress the quality data by using the context statistical model of the first target order to obtain a first intermediate compression result.

Wherein, the first intermediate compression result is only an intermediate output, and will not form a file to be output to a storage such as a disk.

S23. Compress the first intermediate compression result by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

Among them, ANS (Asymmetric Numeral System, asymmetric number system) + FSE (Finite State Entropy, finite state entropy) is a lossless compression technology. Compressing the first intermediate compression result by any one of these lossless compression techniques can further increase the compression ratio.

S3, using the preset reference genome to compare the base data in the gene data block, and compress the base data in the gene data block according to the comparison result.

It should be noted that the reference genome here refers to the genome sequence of the sequenced species, that is, the complete genome sequence that has been assembled, and the complete genome sequence is often used as a standard reference for the species.

Exemplarily, when the present application compares the base data in the gene data block, it can compare the bases in the base data in the gene data block with the reference genome, and find each piece of base data in the reference The position on the gene, so as to obtain the comparison result.

For the matched base data, it can be characterized by its position in the reference genome, that is, it is recorded through this position, instead of retaining the original base data, the recorded matching information is sorted and then the matching information is added. Quantity coding compression, so as to achieve the effect of data compression; for the unmatched base data, it can be compressed by general compression method.

Based on this, in some embodiments of the present application, the above-mentioned S3 uses the preset reference genome to compare the base data in the gene data block, and compresses the base data in the gene data block according to the comparison result process, which can include:

S31. Divide the base data in the gene data block into multiple subsequences.

Exemplarily, the base data can be divided into multiple subsequences with a step size of 20.

S32. Using a hash comparison method to compare each subsequence with a preset reference genome to obtain matching information of each subsequence.

Wherein, the matching information includes a mismatch value (Mismatch).

S33. For a subsequence whose mismatch value is less than or equal to a preset threshold, compress the subsequence based on the matching information of the subsequence.

For example, information such as position and mismatch on the reference genome in the sequence alignment is recorded, instead of retaining the original subsequence, the recorded matching information is sorted and then the matching information is incrementally encoded and compressed.

S34. For a subsequence whose mismatch value is greater than a preset threshold, compress the subsequence in combination with a context hybrid compression algorithm, an incremental encoding technique, and a run-length encoding technique.

Specifically, it may include:

S341. Determine the complexity of the subsequence through a preset adaptive model, and determine a context statistical model of a second target order based on the complexity.

Wherein, the second target order may be 0 order, 1 order or 2 order. After the complexity of the subsequence is counted through the preset adaptive model, the order of the context statistical model suitable for the subsequence is determined, so that a better compression effect can be obtained.

Specifically, the character types of the subsequence (base sequence) include A/T/C/G/N, and if the GC content is greater than 80% or less than 20%, the 0-order context statistical model is selected for compression through prediction; If the GC content is between 60% and 80%, or between 20% and 40%, the 1st-order context statistical model is used for compression through prediction; if the GC content is between 40% and 60%, the compression is performed by The prediction is compressed using a 2nd-order contextual statistical model.

S342. Compress the subsequence by using the context statistical model of the second target order to obtain a second intermediate compression result.

Wherein, the second intermediate compression result is only an intermediate output, and will not form a file to be output to a storage such as a disk.

S343. Compress the second intermediate compression result by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

Exemplarily, run-length coding is used for preprocessing to eliminate redundancy, and then the complexity of base data and quality data in short sequences is understood through an adaptive statistical model, and then dynamic context mixing algorithms of different orders are used according to different complexity The data modeling is carried out, and finally the combination scheme of arithmetic coding and Huffman coding is used for compression.

In some embodiments of the present application, the above S32 uses the hash comparison method to compare each subsequence with a preset reference genome, and the process of obtaining the matching information of each subsequence may include:

Using the hash value of each subsequence as a query condition, the query is performed in a preset hash table to obtain the matching information of each subsequence.

Wherein, the preset hash table records the hash value of each reference subsequence in the reference genome and the position information of each reference subsequence in the reference genome, and each reference subsequence is divided from the reference genome.

In some embodiments of the present application, the above-mentioned process of obtaining each reference subsequence from the reference genome division may include:

Multiple overlapping reference subsequences of length K are divided from the reference genome with a preset step size. Wherein, K is a preset length value. Exemplarily, K may take a value of 20.

Specifically, first, the sequences in the reference genome R are segmented into overlapping k-mer subsequences. Then, a hash table HR is established, and the k-mer long subsequence and the source position mapping of each subsequence in the reference genome are stored in the hash table HR through a hash function. Next, the short sequence in the gene data block is also divided into k-mer long subsequences. Finally, data matching is performed by querying the hash table composed of subsequences with long k-mer length in the genome, and the mismatch value Mismatch is calculated at the same time.

If the Mismatch in the short sequence is less than or equal to the predefined threshold P, it means that all subsequences of the short sequence are adjacent to each other in the correct order and completely match the subsequences of the reference genome. The short sequence can be located on the reference genome and record the matching information.

Wherein, the matching information may include a matching position MATCHpos, a matching length MATCHlen, a matching type MATCHtype, a mismatching position MISpos, and a mismatching base MISalt.

If the Mismatch in the short sequence is less than or equal to the predefined threshold P, the matching information includes the matching position, matching length and matching type "P(Perfect Match)"; otherwise, the matching information includes matching position, matching length and matching type-"I(Insertion ), D(Deletion)", mismatch position and mismatch base. Specifically, CIGAR information in the sequence alignment results can be referred to. The matching information is then sorted according to the matching position. The sorted matching position is conducive to further compression of the matching information. Finally, the incremental encoding scheme is used to record the difference between the two matching information before and after.

It should be noted that when the short sequence contains SNP/INDEL, the mismatch must be located in one of the subsequences of the short sequence, but the other subsequences can completely match the subsequences of the reference genome.

In this way, the fully matched seed sequence position can be used as the anchor point, and the region next to the anchor point can be accurately compared with the remaining incompletely matched subsequences (possibly mismatched subsequences) through the classical alignment algorithm to find the final matching position of the sequence Click to record matching information.

In some other embodiments of the present application, the process of comparing the base data in the gene data block with the preset reference genome in the above S2 may include:

The BWT (Burrows-Wheeler Transform) algorithm is used to compare the base data in the gene data block with the preset reference genome.

Among them, the BWT algorithm is a data conversion algorithm, which puts similar characters in a string in adjacent positions for subsequent compression. Specifically, the index is constructed based on FM, and the backward search and backtracking are realized based on the Burrows-Wheeler matrix (BWM) and Last-First (LF) algorithm. Among them, BWT is a tree structure containing all suffixes and prefixes in the string, which can realize fast search of strings. The process of backtracking the base data in the gene data block is the process of finding the optimal alignment position of the aligned base data.

In some embodiments of the present application, please refer to Figure 3, the genetic data processing method in the context of high-throughput sequencing may also include:

Step S105, acquiring the processing speed of the first operation.

Wherein, the first operation corresponds to the above-mentioned step S101, including: acquiring a genetic data block of a preset size, and inputting it into the first reserved area of the first memory.

Step S106, acquiring the processing speed of the second operation.

Wherein, the second operation corresponds to the above step S102, including: compressing the gene data block based on the data characteristics of each short sequence in the gene data block.

Step S107, based on the processing speed of the first operation and the processing speed of the second operation, determine computing resources allocated to the first operation and the second operation.

Specifically, if the processing speed of the first operation is lower than the processing speed of the second operation, there are fewer genetic data blocks in the first reserved area, and computing resources are relatively sufficient, one genetic data block can simultaneously enable base data comparison compression, Metadata compression and quality data compression are three processes, and appropriate computing resources are allocated to the corresponding processes.

When the metadata or quality data compression of each gene data block is completed, the result is stored in the second reserved area, and the computing resources are released; when the base data comparison task is completed, the comparison result is stored in In the second reserved area, follow-up compression processing is performed according to the comparison result of the base data. After completion, it is stored in the second reserved area and computing resources are released.

If the processing speed of the first operation is greater than the processing speed of the second operation, there will be multiple lingering gene data blocks in the first reserved area. At this time, fewer computing threads can be allocated to a gene data block, and the gene data block is prioritized. For the data comparison of data blocks, during the comparison process, if there are resources that can be used for the compression of metadata and quality data, the corresponding resources will be allocated, otherwise, the data will be compressed and stored again after the comparison of genetic data blocks is completed.

Through the above-mentioned resource adaptive dynamic allocation algorithm, computing resources can be dynamically provided according to the number of genetic data blocks in the first reserved area, as far as possible to ensure first-in-first-out, and at the same time realize the task management and scheduling capabilities of automatically adjusting computing resources. Elastic scaling can ensure that there will be sufficient resources after the resource demand rises in the next period of time, so that there will be no resource shortage and resource supply lag. When the resource demand surges, the elastically released resources can effectively buffer .

The genetic data processing device under the background of high-throughput sequencing provided by the embodiment of the present application is described below. The genetic data processing device under the background of high-throughput sequencing described below is the same as the genetic data processing device under the background of high-throughput sequencing described above. The methods can be referred to each other.

Please refer to Figure 4, the genetic data processing device under the background of high-throughput sequencing provided by the embodiment of the present application may include:

A data block acquiring unit 21, configured to acquire a genetic data block of a preset size when the available space in the first reserved area of the first memory reaches a preset capacity, and input the genetic data block into the first In the first reserved area of the memory, the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size not greater than said preset capacity;

The data block processing unit 22 is used for compressing the gene data block based on the data characteristics of each short sequence in the gene data block to obtain a gene data compression block, and retaining the gene data compression block in the first memory In the second reserved area, and release the gene data block from the first memory, the second reserved area has the ability to accommodate M gene data compression blocks;

The data block deriving unit 23 is used to output J gene data compression blocks to the second memory when the number of gene data compression blocks in the first memory reaches J, and release the J gene data from the first memory data compression block;

Wherein, N, M, and J are all preset natural numbers, and J is not greater than M.

In some embodiments of the present application,

The short sequence includes metadata, base data and quality data; based on the data characteristics of each short sequence in the gene data block, the process of compressing the gene data block includes:

comparing the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the comparison result;

Compressing the metadata in the genetic data block by using incremental encoding technology or run-length encoding technology;

determining the complexity of the quality data through a preset adaptive model, and determining a context statistical model of a first target order based on the complexity;

Compressing the quality data by using the context statistical model of the first target order to obtain a first intermediate compression result;

The first intermediate compression result is compressed by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

In some embodiments of the present application, the data block processing unit 22 uses a preset reference genome to compare the base data in the genetic data block, and compares the base data in the genetic data block according to the comparison result. The process of data compression can include:

Divide the base data in the gene data block into multiple subsequences;

Comparing each subsequence with a preset reference genome using a hash comparison method to obtain matching information for each subsequence, the matching information including a mismatch value;

For a subsequence whose mismatch value is less than or equal to a preset threshold, compress the subsequence based on the matching information of the subsequence;

For subsequences with mismatch values greater than a preset threshold:

determining the complexity of the subsequence through a preset adaptive model, and determining a context statistical model of a second target order based on the complexity;

compressing the subsequence by using the context statistical model of the second target order to obtain a second intermediate compression result;

The second intermediate compression result is compressed by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

In some embodiments of the present application, the data block processing unit 22 uses a hash comparison method to compare each subsequence with a preset reference genome, and the process of obtaining the matching information of each subsequence may include:

Using the hash value of each subsequence as the query condition, query in the preset hash table to obtain the matching information of each subsequence;

Wherein, the preset hash table records the hash value of each reference subsequence in the reference genome and the position information of each reference subsequence in the reference genome, and each reference subsequence is obtained from the derived from the reference genome.

In some embodiments of the present application, the process of dividing the data block processing unit 22 into each reference subsequence from the reference genome may include:

A plurality of overlapping reference subsequences of length K are divided from the reference genome with a preset step size, wherein K is a preset length value.

In some embodiments of the present application, the process of comparing the base data in the genetic data block by the data block processing unit 22 using a preset reference genome may include:

The base data in the gene data block is compared with a preset reference genome using a BWT algorithm.

In some embodiments of the present application, please refer to FIG. 5 , the genetic data processing device in the context of high-throughput sequencing may also include a computing resource allocation unit 24 for:

Obtaining the processing speed of the first operation, the first operation comprising: obtaining a genetic data block of a preset size, and inputting it into a first reserved area of the first memory;

Obtaining the processing speed of the second operation, the second operation comprising: compressing the gene data block based on the data characteristics of each short sequence in the gene data block;

Based on the processing speed of the first operation and the processing speed of the second operation, computing resources allocated to the first operation and the second operation are determined.

The genetic data processing device in the context of high-throughput sequencing provided in the embodiments of the present application can be applied to genetic data processing equipment in the context of high-throughput sequencing, such as computers. Optionally, FIG. 6 shows a block diagram of the hardware structure of the genetic data processing device in the context of high-throughput sequencing. Referring to FIG. 6, the hardware structure of the genetic data processing device in the context of high-throughput sequencing may include: at least one processor 31 , at least one communication interface 32 , at least one memory 33 and at least one communication bus 34 .

In the embodiment of the present application, there are at least one processor 31, communication interface 32, memory 33, and communication bus 34, and the processor 31, communication interface 32, and memory 33 complete mutual communication through the communication bus 34;

The processor 31 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application;

The memory 33 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory;

Wherein, the memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, and the program is used for:

When the available space in the first reserved area of the first memory reaches a preset capacity, acquiring a genetic data block of a preset size, and inputting the genetic data block into the first reserved area of the first memory , wherein the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

Based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compression block, retain the gene data compression block in the second reserved area of the first memory, and releasing the gene data block from the first memory, and the second reserved area has the ability to accommodate M gene data compression blocks;

After the number of gene data compression blocks in the first memory reaches J, output J gene data compression blocks to the second memory, and release the J gene data compression blocks from the first memory;

Wherein, N, M, and J are all preset natural numbers, and J is not greater than M.

Optionally, the detailed functions and extended functions of the program can refer to the above description.

The embodiment of the present application also provides a storage medium, which can store a program suitable for execution by a processor, and the program is used for:

When the available space in the first reserved area of the first memory reaches a preset capacity, acquiring a genetic data block of a preset size, and inputting the genetic data block into the first reserved area of the first memory , wherein the gene data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

Based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compression block, retain the gene data compression block in the second reserved area of the first memory, and releasing the gene data block from the first memory, and the second reserved area has the ability to accommodate M gene data compression blocks;

After the number of gene data compression blocks in the first memory reaches J, output J gene data compression blocks to the second memory, and release the J gene data compression blocks from the first memory;

Wherein, N, M, and J are all preset natural numbers, and J is not greater than M.

Optionally, the detailed functions and extended functions of the program can refer to the above description.

In summary:

In this application, when the available space in the first reserved area of the first storage reaches a preset capacity, a genetic data block of a preset size is obtained, and the genetic data block is input into the first reserved area of the first storage. In the area, wherein, the genetic data block is a collection of short sequences imported from the sequencing platform in real time, the first reserved area has the ability to accommodate N genetic data blocks, and the preset size is not greater than the preset capacity. Next, based on the data characteristics of each short sequence in the gene data block, compress the gene data block to obtain a gene data compression block, and reserve the gene data compression block in the second reserved area of the first memory , and release the gene data block from the first memory at the same time. Wherein, the second reserved area is capable of accommodating M compressed blocks of genetic data. It can be understood that the process of acquiring and placing the gene data block and the process of compressing the gene data block are performed in a cycle. That is, the gene data blocks are continuously delivered from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks; at the same time, the first reserved area Each gene data block in is subjected to compression processing, and the obtained gene data compression block is reserved in the second reserved area of the first memory. When the number of compressed genetic data blocks in the first memory reaches J, output J compressed genetic data blocks to the second memory, and release the J compressed genetic data blocks from the first memory. Wherein, N, M, and J are all preset natural numbers, and J is not greater than M. This application realizes the parallel processing of each genetic data block output by the sequencing platform in a streaming manner, without waiting for all the genetic data to be collected before processing. While saving waiting time, it improves the utilization efficiency of computing resources and improves the efficiency of data from the sequencing platform. The overall speed from off-machine to bioinformatics analysis helps to improve the delivery efficiency of bioinformatics sequencing.

Finally, it should also be noted that in this text, relational terms such as first and second etc. are only used to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations, any such actual relationship or order exists. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the difference from other embodiments. The various embodiments can be combined as needed, and the same and similar parts can be referred to each other. .

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A method of processing genetic data in the context of high throughput sequencing, comprising:

when available space in a first reserved area of a first memory reaches a preset capacity, acquiring a gene data block with a preset size, and inputting the gene data block into the first reserved area of the first memory, wherein the gene data block is a short sequence set transmitted from a sequencing platform in real time, the first reserved area has the capacity of accommodating N gene data blocks, and the preset size is not larger than the preset capacity;

compressing the gene data block based on the data characteristics of each short sequence in the gene data block to obtain a gene data compression block, reserving the gene data compression block in a second reserved area of the first memory, and releasing the gene data block from the first memory, wherein the second reserved area has the capacity of containing M gene data compression blocks;

outputting the J gene data compressed blocks to a second memory and releasing the J gene data compressed blocks from the first memory after the number of the gene data compressed blocks in the first memory reaches J;

n, M, J are all preset natural numbers, and J is not more than M.

2. The method of claim 1, wherein the short sequence comprises metadata, base data, and quality data; a process for compressing the genetic data block based on data characteristics of each short sequence in the genetic data block, comprising:

comparing the base data in the gene data block by using a preset reference gene group, and compressing the base data in the gene data block according to a comparison result;

compressing the metadata in the gene data block by adopting an increment coding technology or a run length coding technology;

determining the complexity of quality data in the gene data block through a preset self-adaptive model, and determining a context statistical model of a first target order based on the complexity;

compressing the quality data by using the context statistical model of the first target order to obtain a first intermediate compression result;

and compressing the first intermediate compression result by adopting a run length coding technology, an ANS + FSE coding technology, an arithmetic coding technology or a Huffman coding technology.

3. The method of claim 2, wherein the comparing the base data in the gene data block with a preset reference gene group and compressing the base data in the gene data block according to the comparison result comprises:

dividing base data in the gene data block into a plurality of subsequences;

comparing each subsequence with a preset reference genome by adopting a Hash comparison method to obtain matching information of each subsequence, wherein the matching information comprises a mismatch value;

compressing the subsequence with the mismatching value less than or equal to a preset threshold value based on the matching information of the subsequence;

for subsequences with mismatch value greater than a preset threshold:

determining the complexity of the subsequence through a preset self-adaptive model, and determining a context statistical model of a second target order based on the complexity;

compressing the subsequence by using the context statistical model of the second target order to obtain a second intermediate compression result;

and compressing the second intermediate compression result by adopting a run length coding technology, an ANS + FSE coding technology, an arithmetic coding technology or a Huffman coding technology.

4. The method according to claim 3, wherein the step of obtaining the matching information of each subsequence by comparing each subsequence with a predetermined reference genome using a hash comparison method comprises:

inquiring in a preset hash table by using the hash value of each subsequence as an inquiry condition to obtain matching information of each subsequence;

the preset hash table carries hash values of all reference subsequences in the reference genome and position information of all the reference subsequences in the reference genome, and all the reference subsequences are obtained by dividing the reference genome.

5. The method of claim 4, wherein the step of obtaining each reference subsequence from the reference genome partition comprises:

and dividing a plurality of overlapping reference subsequences with the length of K from the reference genome by a preset step length, wherein K is a preset length value.

6. The method of claim 2, wherein the comparing the base data in the gene data block with the preset reference genome comprises:

and comparing the base data in the gene data block with a preset reference genome by adopting a BWT algorithm.

7. The method of any one of claims 1 to 7, further comprising:

acquiring a processing speed of a first operation, wherein the first operation comprises: acquiring a gene data block with a preset size, and inputting the gene data block into a first reserved area of the first memory;

acquiring a processing speed of a second operation, wherein the second operation comprises: compressing the gene data block based on the data characteristics of each short sequence in the gene data block;

determining, based on the processing speed of the first operation and the processing speed of the second operation, a computing resource allocated to the first operation and the second operation.

8. A genetic data processing apparatus in the context of high throughput sequencing, comprising:

the data block acquisition unit is used for acquiring a gene data block with a preset size when an available space in a first reserved area of a first memory reaches a preset capacity, and inputting the gene data block into the first reserved area of the first memory, wherein the gene data block is a short sequence set transmitted from a sequencing platform in real time, the first reserved area has the capacity of accommodating N gene data blocks, and the preset size is not larger than the preset capacity;

the data block processing unit is used for compressing the gene data block based on the data characteristics of each short sequence in the gene data block to obtain a gene data compression block, reserving the gene data compression block in a second reserved area of the first memory, and releasing the gene data block from the first memory, wherein the second reserved area has the capacity of containing M gene data compression blocks;

a data block derivation unit for outputting the J gene data compressed blocks to the second memory and releasing the J gene data compressed blocks from the first memory after the number of gene data compressed blocks in the first memory reaches J;

n, M, J are all preset natural numbers, and J is not more than M.

9. A genetic data processing apparatus in the context of high throughput sequencing, comprising: a memory and a processor;

the memory is used for storing programs;

the processor, for executing the program, to implement the steps of the method for gene data processing in the context of high throughput sequencing according to any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of genetic data processing in the context of high throughput sequencing according to any one of claims 1 to 7.

Images