CN118677703A - Data transmission system, method and storage medium based on internal and external network isolation - Google Patents
Data transmission system, method and storage medium based on internal and external network isolation Download PDFInfo
- Publication number
- CN118677703A CN118677703A CN202411155008.5A CN202411155008A CN118677703A CN 118677703 A CN118677703 A CN 118677703A CN 202411155008 A CN202411155008 A CN 202411155008A CN 118677703 A CN118677703 A CN 118677703A
- Authority
- CN
- China
- Prior art keywords
- data
- transmission
- configuration
- module
- service unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 263
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000002955 isolation Methods 0.000 title claims abstract description 30
- 239000000725 suspension Substances 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0263—Rule management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Computer And Data Communications (AREA)
Abstract
本申请涉及网络管理的技术领域，尤其是涉及一种基于内外网隔离的数据传输系统、方法及存储介质，其包括用于将内网部分和外网部分进行隔离的隔离装置，包括外部传输服务单元、内部接收服务单元和内网数据使用单元，其中，外部传输服务单元基于配置要求生成配置项；外部传输服务单元基于数据查询用的配置项在目标位置的数据源中读取相应的数据，并基于数据传输用的配置项将数据传输至内部接收服务单元；内部接收服务单元接收数据，内网数据使用单元中存在若干注册连接于内部接收服务单元中的服务接口，内部接收服务单元基于数据传输用的配置项将数据传输至相应传输要求对应的服务接口中。本申请具有提高数据传输过程中的可配置灵活性的效果。
The present application relates to the technical field of network management, and in particular to a data transmission system, method and storage medium based on the isolation of internal and external networks, which includes an isolation device for isolating an internal network part from an external network part, including an external transmission service unit, an internal receiving service unit and an internal network data use unit, wherein the external transmission service unit generates configuration items based on configuration requirements; the external transmission service unit reads corresponding data in a data source at a target location based on the configuration items for data query, and transmits the data to the internal receiving service unit based on the configuration items for data transmission; the internal receiving service unit receives the data, and there are several service interfaces registered and connected to the internal receiving service unit in the internal network data use unit, and the internal receiving service unit transmits the data to the service interface corresponding to the corresponding transmission requirement based on the configuration items for data transmission. The present application has the effect of improving the configurable flexibility in the data transmission process.
Description
Technical Field
The present application relates to the field of network management technologies, and in particular, to a data transmission system, method and storage medium based on internal and external network isolation.
Background
Based on the security requirement, a considerable amount of enterprises adopt network structures with internal and external network isolation, namely, the internal network and the external network are physically or logically isolated in a network environment, so that the security and the stability of the internal network are ensured. Such isolation is typically achieved through firewall, router, network isolation devices, etc. techniques. The main purpose is to prevent the threat and damage of external network bad factors (such as malicious software, network attack, etc.) to internal network.
In the prior art, a direct mode of jdbc is basically adopted for data transmission under the condition of internal and external network isolation, jdbc is specifically identified as a Java database connection, and is used for providing an application program interface for standardizing how a client program accesses the database through Java language, and providing methods such as inquiring and updating data in the database. The service stores the data into the target database, and the receiver directly acquires the data from the target database through jdbc.
However, in jdbc, the screening conditions in the data transmission process are all fixed, and cannot be set, which results in that relevant parameters cannot be set and updated in the transmission process, all the parameter contents are fixed in the system, and only the program can be modified for setting, so that the flexibility of data transmission is lacking.
Disclosure of Invention
In order to improve the configurable flexibility in the data transmission process, the application provides a data transmission system, a method and a storage medium based on the inter-network and inter-network isolation.
In a first aspect, the present application provides a data transmission system based on the isolation between an internal network and an external network, which adopts the following technical scheme:
The data transmission system based on the internal and external network isolation comprises an isolation device for isolating an internal network part and an external network part, an external transmission service unit corresponding to the external network part, an internal receiving service unit corresponding to the internal network part and an internal network data using unit,
The external transmission service unit generates a configuration item based on a configuration requirement, wherein the configuration item comprises a plurality of configuration columns, each configuration column stores a corresponding configuration characteristic label, and the type of the configuration characteristic label comprises data query and data transmission;
The external transmission service unit reads corresponding data from a data source of a target position based on the configuration item for data query, and transmits the data to the internal receiving service unit based on the configuration item for data transmission;
The internal receiving service unit receives the data, a plurality of service interfaces registered and connected in the internal receiving service unit exist in the intranet data using unit, and the internal receiving service unit transmits the data to the service interfaces corresponding to corresponding transmission requirements based on the configuration items for data transmission.
In some embodiments thereof, the external transmission service unit specifically includes:
The configuration management module is used for reading the configuration requirements to generate a configuration item, wherein the configuration feature tag for data query in the configuration item comprises a data source, a data table name, an execution increment sql and a reading number, and the configuration feature tag also comprises an encryption algorithm;
The data query module is used for selecting the target position based on the data source, querying corresponding data in the data source corresponding to the target position based on the data table name and reading the data, wherein the number of the data read each time is determined based on the number of the data read when the data source is read, and whether the reading process is completed is also confirmed according to the execution increment sql;
a data encryption module that encrypts the data read from the data source based on the encryption algorithm;
And the data trans-regional transmission module acquires the encrypted data and transmits the data to the internal receiving service unit.
In some embodiments, the configuration feature tag for data query further includes an excess time, and the data query module is further configured to obtain a corresponding read time when reading data from the data source, compare the read time with the excess time, and interrupt reading the data and generate an error record when the read time is greater than the excess time.
In some of these embodiments, the configuration feature tag for the data query further includes an execution table number and a total amount sql, the execution table number being characterized by a number of transmission data tables that are allowed to be executed simultaneously;
The data query module is further configured to generate a plurality of independent query task threads based on the execution table number, each of the query task threads corresponding to one of the transmission data tables and each of the query task threads corresponding to a set of the configuration feature tags for data query;
And after each inquiry task thread confirms that the reading process is finished based on the execution increment sql, the data inquiry module is further used for generating summary data based on the total amount sql and sending the summary data to the internal receiving service unit through the trans-regional transmission module.
In some embodiments, the configuration feature tag for data transmission in the configuration item generated by the configuration management module includes a transmission address and a transmission number, the configuration feature tag further includes a decryption algorithm, and the internal receiving service unit specifically includes:
The interface registration module is used for acquiring registration requests of a plurality of service interfaces in the intranet data use unit, completing registration connection with each service interface based on the registration requests, wherein the registration requests contain interface addresses corresponding to each service interface;
the data receiving module is used for acquiring the data transmitted by the data cross-region transmission module and extracting the transmission address and the transmission number corresponding to the data;
The data decryption module decrypts the data acquired by the data receiving module based on the decryption algorithm;
and the data sending module is used for inquiring the corresponding service interface in the intranet data using unit based on the transmission address and transmitting the decrypted data based on the number of transmission lines.
In some embodiments, after the intranet data usage unit receives the data, it is further configured to:
judging the data type of the data, wherein the data type comprises single data and summary data;
If the data is the single data, judging whether the data exists or not, and generating a return result to the internal receiving service unit, wherein the return result comprises: if the data exists, updating the data existing at present according to the data, and if the data does not exist, adding the data;
And if the data is the summary data, comparing the transmission number with the total amount sql corresponding to the summary data to generate a transmission result, and generating a return result to the internal receiving service unit.
In some embodiments, the returned result includes a transmission state, and the internal receiving service unit further includes an automatic push-up module, where the automatic push-up module is configured to generate a timing query task, and the timing query task includes a timing time, a query object, and target data;
The automatic push-up module selects the query object in the timing time, queries the transmission state of the target data in the query object, automatically returns a push-up instruction to resend the target data if the transmission state is failed, and marks the target data to return manual processing information if the resending frequency exceeds a preset frequency.
In some embodiments, the data sending module further comprises a data ranking module and a task allocation module;
The data ranking module is used for inquiring the transmission progress corresponding to the data currently transmitted by each service interface and ranking the service interfaces from high to low based on the transmission progress;
The task allocation module is configured to determine an average transmission speed corresponding to the data currently transmitted by each service interface, and when the average transmission speed is lower than a preset speed, generate a preset number of priority transmission instructions and transmission suspension instructions, sequentially send the preset number of priority transmission instructions to the corresponding service interfaces according to the ranks to continue transmission, and send the transmission suspension instructions to the remaining service interfaces to stop transmission;
The task allocation module is further configured to sequentially convert the transmission suspension instructions of the corresponding number of service interfaces into the priority transmission instructions after the transmission of the service interfaces corresponding to the priority transmission instructions is completed.
In a second aspect, the present application provides a data transmission method based on internal and external network isolation, which adopts the following technical scheme:
a data transmission method based on internal and external network isolation comprises the following steps:
The external transmission service unit generates a configuration item based on a configuration requirement, wherein the configuration item comprises a plurality of configuration columns, each configuration column stores a corresponding configuration characteristic label, and the type of the configuration characteristic label comprises data query and data transmission;
The external transmission service unit reads corresponding data from a data source of a target position based on the configuration item for data query, and transmits the data to the external transmission service unit based on the configuration item for data transmission;
And the internal receiving service unit receives the data, and transmits the data to a corresponding registration in the intranet data using unit based on the configuration item for data transmission, and the registration is connected to a service interface in the internal receiving service unit.
In a third aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:
A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor realizes the steps of the above-mentioned method.
By the technical scheme provided by the embodiment of the application, the following technical effects are achieved:
The data generation and transmission processes are separated, each process corresponds to a modifiable and settable configuration item corresponding to a configuration requirement input by a user, different transmission states in the data generation and data transmission processes are adjusted in real time through the configuration item, and corresponding configuration can be carried out on the read sql, the number of the read sql and the number of the transmitted sql, and the flexibility is extremely high.
Drawings
Fig. 1 is a block diagram of an intranet-based and extranet-based data transmission system in some embodiments of the present application.
Fig. 2 is a block diagram of a data transmission system based on an intranet and an extranet in some embodiments of the application.
Fig. 3 is a flowchart corresponding to the data query generation process in the embodiment of the present application.
Fig. 4 is a flowchart corresponding to a data transmission process in the embodiment of the present application.
Fig. 5 is a flowchart corresponding to the data automatic push process in the embodiment of the present application.
Fig. 6 is a schematic diagram of steps of a data transmission method based on an intranet and extranet in an embodiment of the present application.
Detailed Description
The present application will be described and illustrated with reference to the accompanying drawings and examples for a clearer understanding of the objects, technical solutions and advantages of the present application. However, it will be apparent to one of ordinary skill in the art that the present application may be practiced without these specific details. In some instances, well known methods, procedures, systems, components, and/or circuits have been described at a high-level so as not to obscure aspects of the present application with unnecessary description. It will be apparent to those having ordinary skill in the art that various changes can be made to the disclosed embodiments of the application and that the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the application. Thus, the present application is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the scope of the application as claimed.
The description of these embodiments is provided to assist understanding of the present invention, but is not intended to limit the present invention. In addition, the technical features of the embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
In the description of the present application, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is only for the purpose of distinguishing between technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present application, the descriptions of the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic line representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The embodiment of the application discloses a data transmission system based on internal and external network isolation.
As shown in fig. 1, a data transmission system based on the isolation of an internal network and an external network comprises an isolation device for isolating the internal network part and the external network part. In the embodiment of the application, the isolation device is a physical device for isolating the internal network from the external network.
The system also comprises an external transmission service unit corresponding to the external network part, an internal receiving service unit corresponding to the internal network part and an internal network data using unit. The external network transmission service unit is mainly responsible for data query, generation, encryption and other works, and safely and quickly transmits external data to the internal network. The internal receiving service unit is mainly used for receiving the data transmitted by the external network part, carrying out relevant processing and transmitting the data to a data requiring party. The intranet data use unit stores a plurality of data demand parties, and each demand party is connected with the internal receiving service unit through the service interface and is used for receiving the data.
The external transmission service unit generates configuration items based on the configuration requirements, wherein each configuration item comprises a plurality of configuration columns. In the initial state, each configuration field is blank, and each configuration field is used for storing configuration feature labels with different configuration requirements. The content of the configuration feature tag is associated with a configuration requirement issued by the user, which is a user-selected transmission requirement, such as what data to transmit, the destination location of the transmission, how much data to transmit simultaneously, and so forth. The type of the configuration feature tag comprises data query and data transmission. The configuration feature tag for data query is characterized by configuration parameters required for querying and acquiring corresponding data from corresponding data sources based on certain configuration requirements, and the configuration feature tag for data transmission is characterized by configuration parameters required for transmitting the data to corresponding positions based on certain configuration requirements.
The external transmission service unit reads corresponding data from the data source of the target position based on the configuration item for data query, and transmits the data to the external transmission service unit based on the configuration item for data transmission.
The data sources are characterized as sources of data, and different data correspond to different data sources.
The method comprises the steps of acquiring information such as sources, data names, query amounts and the like of data to be queried based on configuration items for data query, and sending the data to an internal receiving service unit according to configuration requirements based on the configuration items for data transmission, such as transmission speed and the like after the required data is read from the data sources.
The internal receiving service unit receives the data and is used for sending the data to the connected intranet data using unit. And a plurality of service interfaces exist in the intranet data using unit, the service interfaces are all corresponding to a data requiring party, and the service interfaces are all registered and connected with the internal receiving service unit.
After the internal receiving service unit receives the data, the data is sent to the service interface meeting the transmission configuration requirement according to the configuration item for data transmission, and the service interface transmits the data to the server corresponding to the subsequent demand party so as to complete the data transmission work.
According to the technical scheme, the data generation process and the data transmission process are separated, each process corresponds to a configuration item which can be modified and set and corresponds to the configuration requirement input by a user, different transmission states in the data generation and data transmission processes are adjusted in real time through the configuration item, and the corresponding configuration can be carried out on the read sql, the number of the read pieces each time and the number of the transmitted pieces each time, so that the method has extremely high flexibility.
As shown in fig. 2 and 3, in other embodiments, the external transmission service unit specifically includes:
and the configuration management module is used for reading the configuration requirements to generate configuration items. The configuration feature tag for data query in the configuration item comprises a data source, a data table name, an execution increment sql and a reading number. The configuration feature tag also includes an encryption algorithm.
The configuration management module is configured to generate data, transmit the data and carry out subsequent encryption and decryption of configuration items, and specifically, the data source is characterized as a specific data source corresponding to the data required by configuration, for example, the data source is a human resource library; the data table name table is the configured table name of the data to be transmitted, and when the data is transmitted, the acquired and transmitted data are stored in the corresponding table; executing the delta sql characterizes the sql required for the configured daily generation data; the read stripe number is characterized as the number of data stripes each time a query is made in the data source.
The data query module is an executor of the sql and is used for executing corresponding statement instructions of the sql to query corresponding data in the data source. Which determines a target location based on the data sources, the target location corresponding to one or more of the data sources. The corresponding data is then queried in the corresponding database based on the data indication and read, with how many pieces of data are read at a time depending on the number of reads configured by the user. After the corresponding data is read, judging whether all the required data are read according to the increment sql, and when the number of the read data is matched with the number of the increment sql, considering that the reading process is finished.
The data encryption module is used for encrypting the read data by a preset encryption algorithm.
And the data cross-region transmission module is used for calling the internal receiving service unit after the data encryption is finished so as to send the encrypted data to the internal receiving service unit, and storing the data after the internal receiving service unit receives the corresponding data so as to finish the data query and cross-region transmission work.
In other embodiments, the configuration feature tag for data querying further includes an excess time characterized as a maximum time for reading data configured by a user.
The data query module is also used for acquiring the corresponding reading time when the data is read from the data source and comparing the reading time with the exceeding time. If the reading time is longer than the exceeding time, the time spent for inquiring and reading certain data exceeds the longest time for reading the data required by the user, at this time, the reading work of the table corresponding to the data needs to be stopped, error records are carried out, and the user can know the abnormal problems in the data reading process according to the error records.
In other embodiments, the configuration feature tag for data queries further includes an execution table number characterized as the number of concurrently allowed transmit data tables and a total amount sql characterized as the sum of all sql needed in the query process configured based on the data table name.
The data query module is further configured to generate a plurality of independent query task threads based on the number of execution tables, each query task thread corresponding to a transmission data table and each query task thread corresponding to a set of configuration feature tags for data query.
Therefore, a plurality of independent task threads are generated according to the configuration items, each independent task inquiring thread corresponds to one table to respectively perform data generation tasks, and the method is different from serial transmission of the related art, in the serial transmission, the data quantity of each transmission data table is different, so that the corresponding consumed time of each table is different, and data cannot be transmitted in a specified time.
When each inquiry task thread confirms that the reading process is finished based on the execution increment sql, the data inquiry module generates summary data based on the total amount sql and sends the summary data to the internal receiving service unit through the transregional transmission module, wherein the summary data is summary information which is generated for the data transmitted at this time after each data transmission is finished, and the summary information mainly comprises the following steps: the data transmission method comprises the steps of (a) data source name, (a table name), (how much data is transmitted this time), and (a) total data in a table after transmission is finished, wherein the total data is used for overall data transmission integrity judgment when the subsequent data is transmitted to an intranet data use unit.
As shown in fig. 2 and fig. 4, in other embodiments, the configuration feature tag for data transmission in the configuration item generated by the configuration management module includes a transmission address and a transmission stripe number, where the transmission address is characterized by an interface address corresponding to a service interface that needs to receive the data, the transmission stripe number is characterized by a data stripe number included in a transmission message corresponding to each transmission, and the configuration feature tag further includes a decryption algorithm.
The internal receiving service unit specifically includes:
the interface registration module is used for acquiring registration requests of a plurality of service interfaces in the intranet data use unit, completing registration connection with each service interface based on the registration requests, wherein the registration requests contain interface addresses corresponding to each service interface.
All the required parties needing to receive the data need to register the service interfaces corresponding to the required parties into the intranet interface service unit, and corresponding interface addresses are correspondingly generated during registration so as to be used for data transmission in the later period.
The data receiving module is used for acquiring the data transmitted by the data cross-region transmission module and extracting the transmission address and the transmission number corresponding to the data.
The internal receiving service unit receives the data sent by the external network transmission service unit and acquires the configuration information corresponding to the data.
And the data decryption module is used for decrypting the acquired data based on a decryption algorithm, and the decryption algorithm and the encryption algorithm are related to each other.
And the data sending module is used for inquiring a corresponding service interface in the intranet data using unit based on the transmission address and transmitting the decrypted data based on the number of transmission lines.
And the data transmission module is used for transmitting the stored data to the corresponding service interface according to the configuration requirement, and the number of the transmission lines corresponds to the corresponding transmission rate.
In other embodiments, after the intranet data using unit receives the data, the intranet data using unit is further configured to:
The data type of the data is judged, wherein the data type comprises single data and summary data.
The data received by the intranet data using unit comprises common single data and final summary data, and the received summary data is characterized in that all data in the transmission task are acquired by the corresponding server. In order to realize real-time monitoring in the process of transmitting data, corresponding data judgment is needed to be carried out after the data is acquired, the data transmission problem is found in time, and meanwhile, the processing methods corresponding to different data types are different.
If the data is single data, judging whether the data exists or not, and generating a return result to the internal receiving service unit.
When single data is acquired, whether the data exists in a data requiring party or not needs to be judged, if the data exists, the existing old data is subjected to coverage updating according to the newly received data, and if the data does not exist in the requiring party, the data is used as new data. Meanwhile, after the corresponding processing is performed, a return result needs to be generated and sent to the internal receiving service unit for the user to check.
If the data is summary data, comparing the number of transmission with the total amount sql corresponding to the summary data to generate a transmission result, and generating a return result to the internal receiving service unit.
If the acquired data is summary data, it indicates that the data transmission process from the external network to the internal network is completed, and the corresponding data is theoretically completely transmitted to the corresponding data demander, and then it needs to be verified whether an abnormality exists in the complete transmission process, where the abnormality includes data loss.
Therefore, the number of the transmission lines monitored in real time needs to be integrated to calculate the total amount, and the calculated total amount is compared with the total amount sql configured in the summarized data to verify whether the whole data transmission is correct or not, if the number is not matched, the repeated transmission, the error transmission or the data loss of the data may occur in the transmission process. And after the judgment is finished, generating a corresponding return result to the internal receiving service unit.
By the method, the data is counted after each transmission, and the data is checked with a data receiver, so that the accuracy of each transmission is ensured.
In other embodiments, as shown in fig. 2 and fig. 5, the returned result includes a transmission state, and the internal receiving service unit further includes an automatic push-up module, where the automatic push-up module is configured to generate a timing query task, and the timing query task includes a timing time, a query object, and target data. The timing time is characterized by a set execution frequency, for example, the timing time is operated once every three minutes, the query object is characterized by a service interface corresponding to a requiring party needing to query, and the target data is characterized by specific data needing to query.
The automatic push-up module selects a query object in a timing time and queries the transmission state of target data in the query object, wherein the transmission state comprises transmission success and transmission failure. If the transmission failure is detected, a push-up instruction is automatically generated, and the instruction enables the internal receiving service unit to send the data with the failed transmission to the intranet data using unit again.
And if the number of times of retransmission exceeds the preset number of times, marking the target data to return manual processing information. The preset times are generally 5 times, and when the retransmission is unsuccessful for five times, the data transmission failure is identified, and a manual processing link is performed.
In other embodiments, the data transmission module further includes a data ranking module and a task allocation module.
The data ranking module is used for inquiring the transmission progress corresponding to the data currently transmitted by each service interface and ranking the service interfaces from high to low based on the transmission progress.
The transmission progress is characterized by the ratio of the acquired data quantity of the data demand side corresponding to the current service interface to the total quantity sql, and the higher the transmission progress is, the closer the demand data of the demand side is to complete the transmission.
The service interfaces are ordered according to the transmission progress, and the more front the ordering is, the closer the transmission task is to be completed.
The task allocation module is used for judging the average transmission speed corresponding to the data currently transmitted by each service interface, and generating a preset number of priority transmission instructions and transmission suspension instructions when the average transmission speed is lower than a preset speed.
And then judging the average transmission speed corresponding to all data transmitted from the internal receiving service unit to the intranet data using unit through the task allocation module, wherein generally, when more data are simultaneously transmitted and more demand sides corresponding to the simultaneously transmitted data are simultaneously transmitted, the average transmission speed of the whole is slower, otherwise, when the amount of the data which are currently simultaneously transmitted is not more or the demand sides corresponding to the simultaneously transmitted data are not more, the average transmission speed of the whole is faster, and meanwhile, the transmitted data are also influenced by factors such as network environment, server response speed and the like.
When the average transmission speed of the current whole is low, the transmission states corresponding to the service interfaces are allocated through certain processing.
The priority transmission instruction is characterized by an instruction which can lead the transmission task corresponding to part of service ports to be performed preferentially under the condition that the transmission speed is slow; the transmission suspension instruction is characterized by an instruction to cause a service port on which a transmission task is partially performed to suspend the transmission task when there is a slow transmission speed.
The task allocation module is used for sequentially sending the preset number of priority transmission instructions to the corresponding service interfaces according to the ranking to continue transmission, and sending the transmission suspension instructions to the rest service interfaces to transmit.
When transmission allocation is needed, a certain preset amount is set first, and a priority transmission instruction with a certain preset amount is generated to be respectively sent to a plurality of service interfaces with higher transmission progress according to queuing, and the rest service interfaces correspond to transmission suspension instructions. Therefore, the number of transmission lines and the number of data transmission which are simultaneously carried out are reduced, and a certain transmission pressure is slowed down to yield a plurality of service interfaces which are relatively close to completing transmission tasks, so that the transmission speed corresponding to part of the service interfaces can be effectively improved, and the integral reduction of the transmission efficiency caused by the occurrence of transmission blockage of more service interfaces is avoided.
The task allocation module is further configured to sequentially convert transmission suspension instructions of a corresponding number of service interfaces into priority transmission instructions after transmission of the service interfaces corresponding to the priority transmission instructions is completed.
And finally, after the transmission task of the service interface corresponding to the priority transmission instruction is completed, the corresponding number of service interfaces under the transmission suspension instruction can be restored to the priority transmission instruction, so that the transmission task is restarted. For example, when 7 transmission threads exist, 4 corresponding priority transmission instructions and 3 corresponding transmission suspension instructions exist, after a period of time, a service interface in which one thread corresponding to the data transmission task is completed correspondingly becomes the priority transmission instruction.
Meanwhile, if the transmission task allocation process is performed, all the tasks in suspension are restarted at regular time so as to re-judge the overall average transmission speed, and if the average transmission speed is recovered to the normal condition, the transmission allocation is suspended so as to start threads corresponding to all the transmission tasks.
As shown in fig. 6, the application also discloses a data transmission method for the internal and external network isolation, which comprises the following steps:
S100, the external transmission service unit generates a configuration item based on the configuration requirement, wherein the service interface configuration item comprises a plurality of configuration columns, each service interface configuration column stores corresponding configuration feature labels, and the types of the service interface configuration feature labels comprise data query and data transmission.
S200, the external transmission service unit reads corresponding data from a data source of a target position based on a service interface configuration item for service interface data query, and transmits service interface data to the service interface external transmission service unit based on the service interface configuration item for service interface data transmission.
S300, the internal receiving service unit receives service interface data, and the internal receiving service unit transmits the service interface data to the corresponding register in the intranet data using unit based on the service interface configuration item for transmitting the service interface data, and is connected to the service interface in the internal receiving service unit of the service interface.
The application also discloses a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the above method.
The implementation principle is as follows:
The data generation and transmission processes are separated, each process corresponds to a modifiable and settable configuration item corresponding to a configuration requirement input by a user, different transmission states in the data generation and data transmission processes are adjusted in real time through the configuration item, and corresponding configuration can be carried out on the read sql, the number of the read sql and the number of the transmitted sql, and the flexibility is extremely high.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein.
The above embodiments are not intended to limit the scope of the present application, so: all equivalent changes in structure, shape and principle of the application should be covered in the scope of protection of the application.
Claims (10)
1. The data transmission system based on the internal and external network isolation comprises an isolation device for isolating an internal network part and an external network part, and is characterized by comprising an external transmission service unit corresponding to the external network part, an internal receiving service unit corresponding to the internal network part and an internal network data using unit,
The external transmission service unit generates a configuration item based on a configuration requirement, wherein the configuration item comprises a plurality of configuration columns, each configuration column stores a corresponding configuration characteristic label, and the type of the configuration characteristic label comprises data query and data transmission;
The external transmission service unit reads corresponding data from a data source of a target position based on the configuration item for data query, and transmits the data to the internal receiving service unit based on the configuration item for data transmission;
The internal receiving service unit receives the data, a plurality of service interfaces registered and connected in the internal receiving service unit exist in the intranet data using unit, and the internal receiving service unit transmits the data to the service interfaces corresponding to corresponding transmission requirements based on the configuration items for data transmission.
2. The data transmission system based on the intranet and extranet isolation according to claim 1, wherein the external transmission service unit specifically comprises:
The configuration management module is used for reading the configuration requirements to generate a configuration item, wherein the configuration feature tag for data query in the configuration item comprises a data source, a data table name, an execution increment sql and a reading number, and the configuration feature tag also comprises an encryption algorithm;
The data query module is used for selecting the target position based on the data source, querying corresponding data in the data source corresponding to the target position based on the data table name and reading the data, wherein the number of the data read each time is determined based on the number of the data read when the data source is read, and whether the reading process is completed is also confirmed according to the execution increment sql;
a data encryption module that encrypts the data read from the data source based on the encryption algorithm;
And the data trans-regional transmission module acquires the encrypted data and transmits the data to the internal receiving service unit.
3. The data transmission system based on intranet and extranet isolation according to claim 2, wherein the configuration feature tag for data query further comprises an excess time, the data query module is further configured to obtain a corresponding read time when data is read from the data source, compare the read time with the excess time, and interrupt reading of the data and generate an error record when the read time is greater than the excess time.
4. The data transmission system based on the intranet and extranet isolation according to claim 2, wherein the configuration feature tag for the data query further includes an execution table number and a total amount sql, the execution table number being characterized by the number of transmission data tables allowed to be executed simultaneously;
The data query module is further configured to generate a plurality of independent query task threads based on the execution table number, each of the query task threads corresponding to one of the transmission data tables and each of the query task threads corresponding to a set of the configuration feature tags for data query;
And after each inquiry task thread confirms that the reading process is finished based on the execution increment sql, the data inquiry module is further used for generating summary data based on the total amount sql and sending the summary data to the internal receiving service unit through the trans-regional transmission module.
5. The data transmission system based on the intranet and extranet isolation according to claim 1, wherein the configuration feature tag for the data transmission in the configuration item generated by the configuration management module includes a transmission address and a transmission number, the configuration feature tag further includes a decryption algorithm, and the internal receiving service unit specifically includes:
The interface registration module is used for acquiring registration requests of a plurality of service interfaces in the intranet data use unit, completing registration connection with each service interface based on the registration requests, wherein the registration requests contain interface addresses corresponding to each service interface;
the data receiving module is used for acquiring the data transmitted by the data cross-region transmission module and extracting the transmission address and the transmission number corresponding to the data;
The data decryption module decrypts the data acquired by the data receiving module based on the decryption algorithm;
and the data sending module is used for inquiring the corresponding service interface in the intranet data using unit based on the transmission address and transmitting the decrypted data based on the number of transmission lines.
6. The data transmission system based on the intranet and extranet isolation according to claim 5, wherein after the intranet data using unit receives the data, the intranet data using unit is further configured to:
judging the data type of the data, wherein the data type comprises single data and summary data;
If the data is the single data, judging whether the data exists or not, and generating a return result to the internal receiving service unit, wherein the return result comprises: if the data exists, updating the data existing at present according to the data, and if the data does not exist, adding the data;
And if the data is the summary data, comparing the transmission number with the total amount sql corresponding to the summary data to generate a transmission result, and generating a return result to the internal receiving service unit.
7. The data transmission system based on the intranet and extranet isolation according to claim 6, wherein the returned result comprises a transmission state, the internal receiving service unit further comprises an automatic push-up module, and the automatic push-up module is used for generating a timing inquiry task, wherein the timing inquiry task comprises timing time, an inquiry object and target data;
The automatic push-up module selects the query object in the timing time, queries the transmission state of the target data in the query object, automatically returns a push-up instruction to resend the target data if the transmission state is failed, and marks the target data to return manual processing information if the resending frequency exceeds a preset frequency.
8. The data transmission system based on the intranet and extranet isolation according to claim 1, wherein the data sending module further comprises a data ranking module and a task allocation module;
The data ranking module is used for inquiring the transmission progress corresponding to the data currently transmitted by each service interface and ranking the service interfaces from high to low based on the transmission progress;
The task allocation module is configured to determine an average transmission speed corresponding to the data currently transmitted by each service interface, and when the average transmission speed is lower than a preset speed, generate a preset number of priority transmission instructions and transmission suspension instructions, sequentially send the preset number of priority transmission instructions to the corresponding service interfaces according to the ranks to continue transmission, and send the transmission suspension instructions to the remaining service interfaces to stop transmission;
The task allocation module is further configured to sequentially convert the transmission suspension instructions of the corresponding number of service interfaces into the priority transmission instructions after the transmission of the service interfaces corresponding to the priority transmission instructions is completed.
9. The data transmission method based on the internal and external network isolation is characterized by comprising the following steps of:
The external transmission service unit generates a configuration item based on a configuration requirement, wherein the configuration item comprises a plurality of configuration columns, each configuration column stores a corresponding configuration characteristic label, and the type of the configuration characteristic label comprises data query and data transmission;
The external transmission service unit reads corresponding data from a data source of a target position based on the configuration item for data query, and transmits the data to the external transmission service unit based on the configuration item for data transmission;
And the internal receiving service unit receives the data, and transmits the data to a corresponding registration in the intranet data using unit based on the configuration item for data transmission, and the registration is connected to a service interface in the internal receiving service unit.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411155008.5A CN118677703B (en) | 2024-08-22 | 2024-08-22 | A data transmission system, method and storage medium based on internal and external network isolation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411155008.5A CN118677703B (en) | 2024-08-22 | 2024-08-22 | A data transmission system, method and storage medium based on internal and external network isolation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118677703A true CN118677703A (en) | 2024-09-20 |
| CN118677703B CN118677703B (en) | 2024-10-22 |
Family
ID=92732953
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411155008.5A Active CN118677703B (en) | 2024-08-22 | 2024-08-22 | A data transmission system, method and storage medium based on internal and external network isolation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118677703B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006089386A1 (en) * | 2005-02-22 | 2006-08-31 | Nextair Corporation | Simulating an application for subsequent deployment to a device |
| CN104363165A (en) * | 2014-11-14 | 2015-02-18 | 华东电网有限公司 | Information interactive system under internal and external network isolation environment and data integrating method |
| CN108881470A (en) * | 2018-07-06 | 2018-11-23 | 山东中瑞新能源科技有限公司 | A kind of data remote monitoring system and its implementation |
| CN109698837A (en) * | 2019-02-01 | 2019-04-30 | 重庆邮电大学 | A kind of tertiary-structure network based on one-way transmission physical medium and DEU data exchange unit and method |
| US20190266091A1 (en) * | 2018-02-28 | 2019-08-29 | Imagination Technologies Limited | Memory Interface Having Multiple Snoop Processors |
| CN115481924A (en) * | 2022-10-11 | 2022-12-16 | 国电南瑞科技股份有限公司 | Service configuration method and system in power grid dispatching automation system |
| US11569997B1 (en) * | 2020-03-09 | 2023-01-31 | Amazon Technologies, Inc. | Security mechanisms for data plane extensions of provider network services |
| WO2023201927A1 (en) * | 2022-04-23 | 2023-10-26 | 东南大学 | Blockchain cross-chain regulation method for governance of chain by chain |
-
2024
- 2024-08-22 CN CN202411155008.5A patent/CN118677703B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006089386A1 (en) * | 2005-02-22 | 2006-08-31 | Nextair Corporation | Simulating an application for subsequent deployment to a device |
| CN104363165A (en) * | 2014-11-14 | 2015-02-18 | 华东电网有限公司 | Information interactive system under internal and external network isolation environment and data integrating method |
| US20190266091A1 (en) * | 2018-02-28 | 2019-08-29 | Imagination Technologies Limited | Memory Interface Having Multiple Snoop Processors |
| CN108881470A (en) * | 2018-07-06 | 2018-11-23 | 山东中瑞新能源科技有限公司 | A kind of data remote monitoring system and its implementation |
| CN109698837A (en) * | 2019-02-01 | 2019-04-30 | 重庆邮电大学 | A kind of tertiary-structure network based on one-way transmission physical medium and DEU data exchange unit and method |
| US11569997B1 (en) * | 2020-03-09 | 2023-01-31 | Amazon Technologies, Inc. | Security mechanisms for data plane extensions of provider network services |
| WO2023201927A1 (en) * | 2022-04-23 | 2023-10-26 | 东南大学 | Blockchain cross-chain regulation method for governance of chain by chain |
| CN115481924A (en) * | 2022-10-11 | 2022-12-16 | 国电南瑞科技股份有限公司 | Service configuration method and system in power grid dispatching automation system |
Non-Patent Citations (1)
| Title |
|---|
| 冯姗姗;施苗根;杨侃;: "电力客户网上服务中心内外网安全隔离技术", 科技传播, no. 24, 23 December 2012 (2012-12-23) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118677703B (en) | 2024-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7328366B2 (en) | Method and system for reciprocal data backup | |
| US8424095B2 (en) | Method and equipment for verifying propriety of system management policies to be used in a computer system | |
| US7657509B2 (en) | System to manage and store backup and recovery meta data | |
| US7822779B2 (en) | Method and apparatus for scalable transport processing fulfillment system | |
| US7996493B2 (en) | Framework for managing client application data in offline and online environments | |
| US9026679B1 (en) | Methods and apparatus for persisting management information changes | |
| US20100070764A1 (en) | Transfer data management system for internet backup | |
| US20130318178A1 (en) | Systems and Methods for Processing Emails | |
| US20120284770A1 (en) | System, method and program for managing firewalls | |
| US20050278395A1 (en) | Remotely identifying software on remote network nodes by discovering attributes of software files and comparing software file attributes to a unique signature from an audit table | |
| CN111061685B (en) | Log query method and device, node equipment and storage medium | |
| US20050038888A1 (en) | Method of and apparatus for monitoring event logs | |
| US10616055B2 (en) | Hash based device configuration management | |
| CN112764956A (en) | Database exception handling system, and database exception handling method and device | |
| CN111782456A (en) | Anomaly detection method and device, computer equipment and storage medium | |
| CN117076409B (en) | File sharing method, device, system, electronic equipment and storage medium | |
| WO2020232892A1 (en) | Method and apparatus for key management for blockchain, device, and storage medium | |
| CN118677703A (en) | Data transmission system, method and storage medium based on internal and external network isolation | |
| CN111343101B (en) | Server current limiting method, device, electronic equipment and readable storage medium | |
| WO2024164719A1 (en) | Monitoring method and apparatus, and electronic device and readable storage medium | |
| CN117633112A (en) | System event processing method, device and storage medium | |
| CN109933351A (en) | A kind of method and apparatus of reparation and upgrading linux system | |
| US9996870B2 (en) | Method, system, and computer readable medium for utilizing job control orders in an order management system | |
| CN114860814A (en) | Credit investigation processing system, method, credit investigation system, device and medium | |
| JPH08249279A (en) | Online system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |