CN111600957A - File transmission method, device and system and electronic equipment - Google Patents

Abstract

The disclosure provides a file transmission method, a file transmission device, a file transmission system and electronic equipment. The method comprises the following steps: acquiring a file to be processed, wherein the file to be processed comprises at least one file with a file format; converting the file to be processed into a specified file format to obtain a file to be transmitted; and sending the file information of the file to be transmitted to the second server side, so that the third server side can obtain the file to be transmitted from the first server side based on the obtained path information, wherein the obtained path information is determined by the third server side based on the file information obtained from the second server side.

Description

File transmission method, device and system and electronic equipment

Technical Field

The present disclosure relates to the field of internet technologies, and in particular, to a method, an apparatus, a system, and an electronic device for file transmission.

Background

With the continuous development of internet technology and information technology, the amount of traffic that various organizations need to handle is increasing dramatically. Financial institutions such as exchanges, registrars, security dealers, fund companies, commercial banks, etc., interact with each other to process a large number of different types of documents to be processed each day due to the difference in technical platforms. The process of data acquisition, data arrangement, checking and the like is involved in the process of importing the file to be processed, and the file to be processed can adopt a serial import mode.

In the course of implementing the disclosed concept, the inventors found that the related art has at least the following problems: in the process of automatically importing files to be processed, due to the existence of various file formats, different import schemes need to be designed respectively for different file formats, and the file import efficiency cannot meet the requirements.

Disclosure of Invention

In view of this, the present disclosure provides a file transmission method, device, system and electronic device that can effectively improve file import efficiency.

One aspect of the present disclosure provides a file transmission method executed by a first server side, including obtaining a file to be processed, where the file to be processed includes a file in at least one file format; converting the file to be processed into a specified file format to obtain a file to be transmitted; and sending the file information of the file to be transmitted to the second server side, so that the third server side can obtain the file to be transmitted from the first server side based on the obtained path information, wherein the obtained path information is determined by the third server side based on the file information obtained from the second server side.

According to the file transmission method provided by the embodiment of the disclosure, the first server side converts the file to be processed into the uniform file format, so that the third server side can process the file to be transmitted acquired from the first server side in the same manner, the file processing efficiency is improved, and the system complexity is reduced. After the file information of the file to be transmitted is stored in the second server, the second server can conveniently distribute a file processing task to the third server based on the file information, and if at least one file to be transmitted is distributed to at least one third server node, the concurrent processing of the file is realized, and the file importing and processing efficiency is improved.

One aspect of the present disclosure provides a file transmission method performed by a second server, including receiving file information of a file to be transmitted from a first server, and receiving third server information from a third server; distributing a file to be transmitted to at least one third server node based on the file information and the third server side information to generate a scheduling task; and adding the scheduling task into the task scheduling queue, so that the third server side determines to obtain the path information based on the task scheduling queue, and obtains the file to be transmitted from the first server side based on the obtained path information, so that each third server node can generate a command to be executed based on the file to be transmitted, and the file to be transmitted is stored in a database.

One aspect of the present disclosure provides a file transmission method performed by a third server side, including, for each third server node, determining file information of a file to be transmitted in response to a scheduling task from a task scheduling queue of the second server side; determining the obtaining path information of the file to be transmitted based on the file information; and acquiring the file to be transmitted from the first server side based on the acquisition path information so as to execute the instruction generated based on the file to be transmitted.

One aspect of the present disclosure provides a file transfer method performed by a file transfer system, where the file transfer system includes a first server, a second server, and a third server connected to each other, and the third server includes at least one third server node, where the method includes: the method comprises the steps that a first server side converts an acquired file to be processed into an appointed file format to obtain a file to be transmitted, file information of the file to be transmitted is sent to a second server side, and the file to be processed comprises at least one file with a file format; the second server side distributes a file to be transmitted to at least one third server node based on the file information and the third server side information so as to generate a task scheduling queue, and the third server side information represents the service state of the at least one third server node; and the third server side responds to the scheduling tasks of the task scheduling queue, determines the file information of the file to be transmitted, and determines the obtaining path information of the file to be transmitted based on the file information so as to obtain the file to be transmitted based on the obtaining path information.

One aspect of the present disclosure provides a file transfer apparatus including: the device comprises a first file acquisition module, a second file acquisition module and a first file sending module. The first file acquisition module is used for acquiring files to be processed, and the files to be processed comprise files in at least one file format; the second file acquisition module is used for converting the file to be processed into a specified file format to obtain the file to be transmitted; and the first file sending module is used for sending the file information of the file to be transmitted to the second server side, so that the third server side can obtain the file to be transmitted from the first server side based on the obtained path information, and the obtained path information is determined by the third server side based on the file information obtained from the second server side.

One aspect of the present disclosure provides a file transfer apparatus including: the device comprises a first file receiving module, a first file distributing module and a scheduling module. The first file receiving module is used for receiving file information of a file to be transmitted from a first server and receiving third server information from a third server; the first file distribution module is used for distributing files to be transmitted to at least one third server node based on the file information and the third server side information so as to generate a scheduling task; and the scheduling module is used for adding the scheduling task into the task scheduling queue so that the third server side can determine to obtain the path information based on the task scheduling queue and obtain the file to be transmitted from the first server side based on the obtained path information.

One aspect of the present disclosure provides a file transfer apparatus including: the system comprises a task corresponding module, an acquisition path information determining module and a third file acquisition module. The task corresponding module is used for responding to a scheduling task from a task scheduling queue of the second server end and determining file information of a file to be transmitted; the acquisition path information determining module is used for determining the acquisition path information of the file to be transmitted based on the file information; and the third file acquisition module is used for acquiring the file to be transmitted from the first server side based on the acquisition path information so as to execute the instruction generated based on the file to be transmitted.

One aspect of the present disclosure provides a file transfer system including: the server comprises a first server side, a second server side and a third server side which are connected with each other, wherein the third server side comprises at least one third server node. Wherein the first server side, the second server side and the third server side are configured to: the method comprises the steps that a first server side converts an acquired file to be processed into an appointed file format to obtain a file to be transmitted, and transmits file information of the file to be transmitted to a second server side, wherein the file to be processed comprises at least one file format; the second server side distributes files to be processed to at least one third server node based on the file information and third server side information to generate a task scheduling queue, and the third server side information represents the service state of the at least one third server node; and the third server side responds to the scheduling tasks of the task scheduling queue, determines file information, and determines the obtaining path information of the file to be transmitted based on the file information, so that the instruction generated based on the file to be transmitted is executed after the file to be transmitted is obtained based on the obtaining path information.

Another aspect of the present disclosure provides an electronic device comprising at least one processor and a storage, wherein the storage is configured to store executable instructions, which when executed by the processor, implement the method as described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.

Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario of a file transfer method, apparatus, system and electronic device according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates an exemplary system architecture to which the file transfer method, apparatus, system, and electronic device may be applied, according to an embodiment of the present disclosure;

fig. 3 schematically shows a flow chart of a file transfer method performed by a first server side according to an embodiment of the present disclosure;

FIG. 4 schematically shows a schematic diagram of file information according to an embodiment of the disclosure;

fig. 5 schematically shows a flow chart of a file transfer method performed by the second server side according to an embodiment of the present disclosure;

FIG. 6 schematically shows a schematic diagram of allocating files to be transmitted according to an embodiment of the present disclosure;

fig. 7 schematically shows a flow chart of a file transfer method performed by a third server side according to an embodiment of the present disclosure;

fig. 8 schematically shows a schematic diagram of updating third server-side information according to an embodiment of the present disclosure;

FIG. 9 schematically illustrates a flow chart of a file transfer method performed by a file transfer system according to an embodiment of the present disclosure;

FIG. 10 schematically illustrates a data flow diagram of a file transfer method performed by a file transfer system according to an embodiment of the present disclosure;

FIG. 11 schematically illustrates a structural schematic view of a document transport apparatus according to an embodiment of the disclosure;

fig. 12 schematically shows a structural view of a document transporting apparatus according to another embodiment of the present disclosure;

fig. 13 schematically shows a structural view of a document transporting apparatus according to another embodiment of the present disclosure;

FIG. 14 schematically illustrates a structural schematic of a file transfer system according to an embodiment of the disclosure; and

FIG. 15 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. At least one embodiment may be practiced without these specific details. In the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of at least one other feature, step, operation, or component.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art and are to be interpreted as having a meaning that is consistent with the context of this specification and not in an idealized or overly formal sense expressly so defined herein.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). The terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one feature.

For organizations (such as financial institutions and online trading platforms) needing to process a large number of files with various file formats, how to utilize own technical means to quickly respond and process the files to form data required by the organizations and analyze the data to obtain information with commercial value becomes core competitiveness among the organizations, and is convenient for providing better and better services for customers.

The embodiment of the disclosure provides a file transmission method, a file transmission device, a file transmission system and electronic equipment. The file transmission method comprises a file preprocessing process executed by a first server side, a file distribution process executed by a second server side and a file transmission process executed by a third server side. In the file preprocessing process, firstly, files to be processed are obtained, the files to be processed comprise files in at least one file format, then the files to be processed are converted into the specified file format, the files to be transmitted are obtained, and then the file information of the files to be transmitted is sent to the second server side. After the file preprocessing process is finished, a file allocation process is started, firstly, a second server receives file information of a file to be transmitted from a first server and third server information from a third server, then the file to be transmitted is allocated to at least one third server node based on the file information and the third server information to generate a scheduling task, and then the scheduling task is added into a task scheduling queue. After the file allocation process is completed, a file transmission process is started, firstly, the file information of the file to be transmitted is determined in response to the scheduling task of the task scheduling queue from the second server side, then, the obtaining path information of the file to be transmitted is determined based on the file information, then, the file to be transmitted is obtained from the first server side based on the obtaining path information, so that an instruction generated based on the file to be transmitted is executed, and operations such as importing the file into a database and the like are completed.

Fig. 1 schematically illustrates an application scenario of a file transfer method, device, system and electronic equipment according to an embodiment of the disclosure.

As shown in FIG. 1, the first server has nodes 1-4, etc., and the third server has nodes a-d, etc. Different from the prior art in which at least one file to be processed is directly received and stored by a database, each node of the first server end can respectively receive files to be processed from different main bodies and respectively perform preprocessing (such as format conversion, compression, backup and the like), thereby realizing thread-level concurrent processing. The first server side does not directly send each file to be processed to the application database, but sends file information of the file to be processed to a second server side (such as a database server), the file information can comprise file identification, node identification, path information and the like of the file to be processed, then the second server side can distribute the file to be processed based on the file information and server node information of a third server side to determine a proper application server (process level concurrence is realized), a scheduling task and a task scheduling queue are formed, and therefore each server node of the third server side can conveniently determine the obtaining path information of the file to be processed based on the scheduling task, obtain the file to be processed based on the obtaining path information, and execute operation related to the file to be transmitted. The file processing efficiency and reliability are effectively improved.

Fig. 2 schematically illustrates an exemplary system architecture to which the file transfer method, apparatus, system, and electronic device may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 2 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 2, the system architecture 200 according to this embodiment may include terminal devices 201, 202, a network 203, and servers 204, 205, 206. The network 203 may include at least one gateway, hub, network line, etc. to provide a medium for communication links between the terminal devices 201, 202 and the servers 204, 205, 206. Network 203 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the terminal device 201, 202 to interact with other terminal devices and servers 204, 206 via the network 203 to receive or send information and the like, such as sending a service request, sending an information request, receiving processing results and the like. The terminal devices 201 and 202 may be installed with various communication client applications, for example, a bank application, an operation and maintenance application, a web browser application, a search application, an office application, an instant messaging tool, a mailbox client, social platform software, and other applications (examples are only) installed on a website terminal.

The terminal devices 201, 202 include, but are not limited to, kiosk machines, smart phones, virtual reality devices, augmented reality devices, tablets, laptop computers, and the like.

The server 204 may receive a request, such as a service request (e.g., processing a specific object with an associated file), the server 204 sends file information to the server 205, and the server 205 may perform file allocation according to the file information of the file to be processed and the requested state of the server, so as to improve processing efficiency and generate a scheduling task. The server 206 may obtain the pending file from the server 204 storing the pending file according to the scheduling task to perform the business process. For example, the servers 204, 205, 206 may be background management servers, server clusters, and the like. The background management server may analyze and process the received service request, information request, database update instruction, and the like, and feed back a processing result (such as requested information, a processing result, and the like) to the terminal device.

It should be noted that the file transmission method provided by the embodiments of the present disclosure may be generally executed by the servers 204, 205, 206. The file transfer method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster different from the servers 204, 205, 206 and capable of communicating with the terminal devices 201, 202 and/or the servers 204, 205, 206.

It should be understood that the number of terminal devices, networks, and servers are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 3 schematically shows a flowchart of a file transfer method performed by the first server side according to an embodiment of the present disclosure. If the first server end is composed of at least one file server node, each file server node is connected with the database server to run the timing task. Specifically, a JAVA third-party timed task framework Quartz (including file acquisition, file backup, file processing, and the like) may be adopted to send the processed file information to the second server to be written into the database, and each operation in fig. 3 corresponds to a different thread, so as to maximally decouple the operations.

As shown in fig. 3, the file transfer method includes operations S301 to S305.

In operation S301, a file to be processed is acquired, where the file to be processed includes a file in at least one file format.

In this embodiment, the first server end is connected to the second server end and the third server end, respectively, and the first server end may include at least one first server node, and receive the file to be processed in a server cluster form.

For example, the file server cluster is externally connected with a third-party system, and receives files to be processed (such as data files) in different channels by using multi-thread concurrent processing, and then collects and stores the received files after landing. The formats of the files to be processed in different channels may be different, for example, the original file formats include but are not limited to: dbf, excel, csv, xml, and the like. When the application server processes the files, different processing flows can be respectively adopted, which brings inconvenience to the application server for processing the files, and different programs need to be respectively developed for different formats.

In operation S303, the file to be processed is converted into a specified file format, so as to obtain the file to be transmitted.

In this embodiment, various file formats may be converted into specified file formats, so that the third server can read and process the file information. For example, the specified format includes, but is not limited to, any of: txt, dbf, excel, csv, xml, and the like. The file format conversion process may be implemented locally by the first server side, or may be implemented by calling a third-party application interface.

For example, a file to be processed is read and analyzed through a third-party application interface (API), the file content can be returned by a read () method in a public interface FileReader in different file formats, the file content is written into a new txt file, files with the same interface are summarized and written into the same txt file, and the influence of a plurality of small files on the subsequent processing efficiency due to the fact that threads are occupied by the small files is prevented. The information of each processing unit can be converted into the information of the file with the specified format from high to low according to the grade of the processing unit of the file to be processed, wherein the processing unit with the high grade comprises at least one processing unit with the low grade. For example, the DBF file includes processing units such as a field array, a record array, and a field, and first reads the field array and determines field information of the record array included in each field data. Also for example, an EXCEL file includes processing units such as a workbook, a page, and a line of records. As another example, a CSV file characterizes a processing unit with delimiters.

The following reading and parsing of a file in excel format may include the following operations: the method comprises the steps of obtaining a file input stream, reading workbook information, reading page information and reading record lines, so that the content of a file in an excel format can be read from high to low step by step, and the file to be transmitted can be written into the file according to a specified file format.

The following describes an example of conversion of a file in an excel format.

In operation S305, file information of the file to be transmitted is sent to the second server, so that the third server obtains the file to be transmitted from the first server based on the obtaining path information, where the obtaining path information is determined by the third server based on the file information obtained from the second server.

In this embodiment, the file information of the file to be transmitted may be written into the database, so that the third server may perform matching based on an Internet Protocol (IP) address of the third server in the database, and determine the obtained path information of the file to be transmitted.

Specifically, the file information includes acquisition path information and at least one of: the file management system comprises file identification information, file size information and file state information, wherein the acquired path information comprises first server node address information and file path information. For example, the file information includes: and writing the values of the information such as the file name, the path, the size, the state, the IP of the first server node and the like into a database.

Fig. 4 schematically shows a schematic diagram of file information according to an embodiment of the present disclosure.

As shown in fig. 4, the file information sent by the first server to the second server may include a file name, a file path, and an IP address of the first server node, so that the third server can determine the IP address and the file path of the first server node where the file to be transmitted is located based on the identification information (such as the file name) of the allocated file to be transmitted, so as to request the file to be transmitted from the first server. In the process of file allocation, a mapping relationship between the IP address of the third server node of the third server and the file information may be established in the database.

For example, if the second server assigns file 2 to the third server node 3, the IP address of the third server node 3 may be written in the row of file 2 in fig. 4. Therefore, the third server node can acquire the acquisition path information of the file to be transmitted in a table look-up mode. In addition, if the file is transmitted in a manner of generating a scheduling task, the established mapping relationship may further include a time parameter. In addition, the second server may record data transmission operations, such as when the scheduling task is triggered, or what time the third server starts to execute the import task and completes the scheduling task or the import task, such as recording an import start time and an import end time. The file information after the mapping relationship is established in the second server may be as shown in table 1.

TABLE 1

In another embodiment, some files with large data volume are inconvenient to transmit, the compression efficiency of the text files is high, network resources occupied during transmission can be effectively reduced after compression, and the subsequent processing efficiency is increased. Accordingly, the above method may further include the following operations. After the file to be processed is converted into the specified file format, the file to be processed in the specified file format is compressed.

Specifically, in order to improve the compression efficiency, the files to be processed may be compressed in a multi-thread concurrent processing manner. The compression method can adopt a lossless compression mode and a lossy compression mode which does not prevent the third server side from executing the file to be processed.

In another embodiment, in order to facilitate historical data query, exception handling and the like, the method may further include the operation of backing up the file to be processed after the file to be processed is acquired. In particular, multithreading concurrent processing may be used to backup the original files of the files to be processed to ensure the integrity and consistency of the files.

For example, the backup operation may be performed according to configured backup task events and trigger relationships computed by the functions. The task function calculation processes the backup task in an elastic mode, and performs packing, compression, duplicate removal and other processing according to preset rules (such as according to sources, time, business and the like), wherein the backup file can be stored in a local or cloud terminal, and is transferred to media such as a tape and the like for long-term reservation.

Fig. 5 schematically shows a flowchart of a file transfer method performed by the second server side according to an embodiment of the present disclosure.

As shown in fig. 5, the method may include operations S501 to S505.

In operation S501, file information of a file to be transmitted from a first server is received, and third server information from a third server is received.

In this embodiment, the second server may be a database server, the first server may be a file server cluster, and the third server may be an application server cluster. The database server is respectively connected with the file server cluster and the application server cluster so as to receive and store file information from the file server cluster and server node state information from the application server cluster. The third server side information may include IP address information of the third server node, so that a mapping relationship is conveniently established between the file information and the IP address information of the third server node, and the third server side determines the obtaining path information of the file to be transmitted based on the mapping relationship.

Specifically, the timing task is run, and the file import task is distributed to an available application server node by using, for example, an ORACLE database timing task DBMS _ JOB.

In operation S503, a file to be transmitted is allocated to at least one third server node based on the file information and the third server-side information to generate a scheduling task.

For example, the file allocation may be performed by polling, random algorithm, hash modulo, etc.

In one embodiment, allocating the file to be transmitted to the at least one third server node based on the file information and the third server-side information may include the following operations.

First, a load balancing policy is determined based on file information (such as size, allocated files to be processed and occupied third server nodes thereof, etc.). And a proper load balancing algorithm is adopted to uniformly distribute tasks so as to improve the processing efficiency of the files to be processed. Load balancing algorithms that may be employed include, but are not limited to: polling, random algorithms, hash modulo, etc. In addition, a customized load balancing algorithm can be adopted: such as a file number balancing policy and a file size balancing policy. For example, the number of files is balanced as: the file number of the files currently processed by each node (which may be compressed files compressed by the first server side) is queried, and the node with the least number of compressed files is allocated. The algorithm can balance the number of tasks being processed by each node, and the problems of low processing efficiency and the like caused by uneven distribution are prevented. For another example, the file size is balanced as: and inquiring the sum of the sizes of the files currently processed by each node, and allocating the node with the smallest sum of the sizes of the files. The algorithm can balance the size of the processed file of each server, and prevent risks of memory overflow of a certain node and the like caused by overlarge individual files.

And then, distributing the file to be transmitted to at least one third server node based on the load balancing strategy and the third server side information. In particular, reference may be made to the relevant description part of fig. 4 to implement the allocation process of the file to be transmitted. According to the embodiment, an applicable load balancing strategy can be selected according to the characteristics of the file to be transmitted. For example, the original file sizes are averaged, and a file number balancing policy may be selected. For another example, for a file to be transmitted with a large amount of data, and other files are relatively even, a file size balancing policy may be selected.

Fig. 6 schematically shows a schematic diagram of allocating files to be transmitted according to an embodiment of the present disclosure.

As shown in fig. 6, the first server node 1 stores the files a and b waiting for transmission, the first server node 2 stores the files c and d waiting for transmission, and the first server node 3 stores the files e and f waiting for transmission. The second server side is used for transmitting the file: and the files a to f and the like are distributed to the application server nodes so that the application server nodes can process the files to be transmitted conveniently. For example, according to the state of each third server node of the third server, the characteristics of the files a to f, and the like, the application service nodes are respectively allocated to the files a to f. For example, in fig. 6, the hash modulo comparisons of the file b and the file d are similar, the third server node 3 is in a normal working state, and the available computing resources can meet the computing requirements for the file b and the file d, then the file b and the file d may be allocated to the third server node 3. For another example, if the file a is a large-data-volume file and requires a large number of computing resources, the third server node 2 may be separately allocated to the file a. For another example, if the file c and the file e are small-data-size files and can be allocated to the same third server node for processing, the file c and the file e may be allocated to the third server node 1. For another example, if the third server nodes 1 to 3 have already been allocated with files, the third server node 4 has not been allocated with files currently, and the computing resources are sufficient, the file f may be determined to be allocated to the third server node 4 based on a polling method.

In operation S505, a scheduling task is added to a task scheduling queue, so that a third server determines to obtain path information based on the task scheduling queue, and obtains a file to be transmitted from a first server based on the obtained path information

In another embodiment, the method may further include the following operations. And updating the third server side information of the third server side in response to the registration request and/or the logout request from the third server side.

In a specific embodiment, taking a file to be transmitted after the first server performs file compression as an example for description, first, compressed file information from the first server is received, and each piece of compressed file information is recorded. And receiving the application server information from the third server, inserting the information of each normal application server node, and deleting the information of each abnormal application server node. Then, each compressed file is distributed to each application server. And then, adding the task into a scheduling queue, updating the file state in the file information table, and enabling the file to be transmitted to enter a state to be imported.

Fig. 7 schematically shows a flowchart of a file transfer method performed by the third server side according to an embodiment of the present disclosure. The third server end is respectively connected with the first server end and the second server end, and comprises at least one third server node. The third server node may be an application server node, each application server node is connected to a database (second server side), and the application server node registers its own information (such as an IP address) at the second server side. And running a timing task, for example, adopting a JAVA third-party timing task framework Quartz, acquiring a task allocated by the database to process a file to be transmitted (for example, importing the file to be transmitted to a specified object), and the like.

As shown in fig. 7, the file transfer method may include operations S701 to S705 for each third server node.

In operation S701, file information of a file to be transmitted is determined in response to a scheduling task from a task scheduling queue of the second server.

For example, the scheduling task may include file information of the allocated file to be transmitted, and the file information may include an IP address and a file path of a first server node where the file to be transmitted is located, so that the third server node may obtain the file to be transmitted based on the IP address and the file path of the first server node. The file to be transmitted can be compressed to reduce the occupation of network resources.

For example, the database compressed file information is obtained from the second server side. The state information of all compressed files is recorded in the database compressed file information table, and the information of the compressed files to be processed is obtained from a specified server (such as a file server).

In operation S703, acquisition path information of a file to be transmitted is determined based on the file information. Such as the IP address and file path of the first server node.

In operation S705, a file to be transmitted is acquired from the first server based on the acquisition path information, so as to execute an instruction generated based on the file to be transmitted.

For example, the compressed file is copied from a first server node (e.g., a file server) based on the acquisition path information. Different Transfer protocols, such as File Transfer Protocol (FTP), hypertext Transfer Protocol (HTTP), etc., or a shared disk manner may be used.

In another embodiment, the file to be transmitted is a compressed file to be processed. Accordingly, the obtaining of the file to be transmitted from the first server based on the obtaining path information to execute the instruction generated based on the file to be transmitted may include the following operations. First, a compressed file to be processed is acquired and decompressed. Then, the to-be-executed instruction is generated based on the to-be-processed file. Then, the instruction to be executed is executed to implement, for example, transferring the file to be processed into a database, or executing a specific operation.

For example, the file to be transmitted after the compression processing is decompressed. And decompressing the compressed file to obtain a file to be transmitted in a specified format, such as a file to be processed in a txt format. Then, a control file is generated. The control file is generated, e.g., according to the file interface definition. Next, the sqlldr command is executed. SQL load is an ORACLE data loading tool that is typically used to migrate operating system files (data) into an ORACLE database. And after the application server decompresses the txt file to generate a control file, updating the import starting time in the file information table, calling an sqlldr command to import the sqlldr command into an ORACLE database, and updating the file state and the import ending time in the file information table after the import is finished, so that the import is finished.

For example, the following operation instructions may be executed.

sqlldr userid user/pass @ orcl// database username, password, instance name

control// control file/control/xxx

data/XXX. txt// data file

log/xxx. txt. log// log file

bad// error file

errors＝200000rows＝1000bindsize＝655360silent＝header,feedback

In another embodiment, the method may further include at least one of the following operations.

And updating the third server side information, wherein the third server side information represents the service state of at least one third server node.

And after the third server node meets the service providing condition, sending a registration request to the second server to register the address of the third server node.

And after the third server node meets the service stopping condition, sending a logout request to the second server side to logout the address of the third server node.

For example, after the application server is started, the application server registers the IP in the database. When the application server is started, except for loading the components required by the operation of the application server, the registration component is started at the same time, the current server IP is added into the database cluster information table and is used as a node of the unique server identifier and the load balancing strategy, and dynamic capacity expansion and hot plug are supported. The cluster information table is shown in table 2.

TABLE 2

Server type

Application server IP

For another example, when the application server is stopped, such as an abnormal situation that the application server goes down, the IP address of the application server may be logged out of the database. For example, when the application server stops, a logout component is started, the IP address of the current application server is deleted from the database cluster information table and is no longer used as a node of a load balancing strategy, and dynamic capacity reduction and fault isolation are supported. And if the server is abnormally shut down in the task execution process to cause the task interruption, reallocating the interrupted task to the server in batches at regular time and adding the server to a scheduling queue.

Fig. 8 schematically shows a schematic diagram of updating third server-side information according to an embodiment of the present disclosure.

As shown in fig. 8, after the third server node 1 is started, if the service provision condition is satisfied, the third server node 1 may perform IP registration with the second server, so as to add the IP address of the third server node 1 in the database cluster information table. And if the third server node 2 is currently in a normal working state, the IP address of the third server node 2 in the database cluster information table is maintained. If the third server node 3 is currently in an abnormal state, the IP address of the third server node 3 in the database cluster information table is deleted. If the third server node 4 needs to be shut down, IP logout may be performed to the second server side to delete the IP address of the third server node 4 in the database cluster information table.

In another embodiment, the method further comprises: and after the file instruction to be executed is executed, sending the scheduling task state updating information to the second server side. For example, the status of the scheduling task, whether the scheduling task is completed, the import start time, the import completion time, which IP address server node executes the scheduling task, and the like are updated.

The file transmission method provided by the embodiment of the disclosure at least partially solves the problem that in the traditional file import process, single files need to be imported one by one. Meanwhile, the defects that the file format is limited to diversification, a plurality of importing schemes need to be designed, and the importing efficiency, accuracy and expansibility are poor are at least partially solved. The file transmission method provided by the embodiment of the disclosure combines the characteristic of importing a large-data-volume file into a database, provides a set of universal automatic rapid importing method, dynamically expands and contracts the capacity, has good expansibility, and supports mechanisms such as high concurrency and high availability, soft load balancing, fault isolation and the like.

Fig. 9 schematically shows a flowchart of a file transfer method performed by the file transfer system according to an embodiment of the present disclosure. The file transmission system comprises a first server side, a second server side and a third server side which are connected with each other, wherein the third server side comprises at least one third server node.

As shown in fig. 9, the method may include operations S901 to S905.

In operation S901, the first server converts the acquired to-be-processed file into a specified file format to obtain a to-be-transmitted file, and sends file information of the to-be-transmitted file to the second server, where the to-be-processed file includes at least one file in a file format.

In operation S903, the second server allocates a file to be transmitted to the at least one third server node based on the file information and the third server information, so as to generate a task scheduling queue, where the third server information represents a service state of the at least one third server node.

In operation S905, the third server determines file information of a file to be transmitted in response to a scheduling task of the task scheduling queue, and determines acquisition path information of the file to be transmitted based on the file information, so as to acquire the file to be transmitted based on the acquisition path information.

Fig. 10 schematically illustrates a data flow diagram of a file transfer method performed by a file transfer system according to an embodiment of the present disclosure.

As shown in fig. 10, the first server stores compressed file information to the second server after the timing task processing (such as file format conversion, file backup, and file compression) is completed, and the second server enters the third server after the timing task processing (such as collecting compressed file information, collecting third server information, and performing compressed file allocation) is completed. And the third server side stores the third server side information to the second server side by a timing task, in addition, the file information of the compressed file to be transmitted is determined based on the scheduling queue, so that the compressed file to be transmitted is conveniently acquired from the first server side based on the file information, and finally, a Control file generated based on the compressed file executes an sqlldr command to guide the decompressed file into a database of the second server side. Wherein, the file server file can be used for synchronous copy of the server after being compressed. After the application server logs in the IP, the IP enters the information record of the database server, and after the application server logs out the IP, the logged-out IP enters the information of the database server so as to redistribute the IP.

The file transmission method provided by the embodiment of the disclosure is based on the Quartz timing task, the file reading and writing, the file transmission, the ORACLE timing task and the SQL load import technology, and according to the existing server information and the file information, a user-defined load balancing strategy is designed, a high-concurrency high-availability rapid file import scheme is realized, hot plug, soft load balancing and fault tolerance mechanisms are supported, fault isolation is improved, tasks are automatically recovered, and the crash of one node does not affect other nodes.

In the file transmission method provided by the embodiment of the present disclosure, when a system expands capacity, only file servers with the same configuration need to be deployed at a first server end, and application servers with the same configuration need to be deployed at a third server end, and are respectively connected to a database for processing, that is, the system expands capacity horizontally.

According to the file transmission method provided by the embodiment of the disclosure, when the system is scaled down, only the file server and the application server need to be offline, namely, the system is scaled down horizontally.

According to the file transmission method provided by the embodiment of the disclosure, when part of the file servers and the application servers of the system are abnormal, the fault can be isolated by only stopping and taking off the line, and other servers are not affected. Furthermore, due to the system architecture and the information as recorded in reference to table 1, the interrupt tasks can be reinitiated, support idempotent operations, and achieve high availability.

The file transmission method provided by the embodiment of the disclosure supports high concurrency in terms of file import efficiency, on one hand, achieves a theoretically maximum load balancing effect through horizontal expansion, reasonably adjusts the number of file server clusters and application server clusters according to the size and the number of different files, and actually realizes process-level concurrency. On the other hand, the concurrency number of each node is reasonably increased by combining the hardware configuration of each node to improve the import efficiency, and the thread-level concurrency is actually realized. The file import efficiency is effectively improved.

Fig. 11 schematically shows a structural diagram of a document transportation apparatus according to an embodiment of the present disclosure.

As shown in fig. 11, the file transfer apparatus 110 includes: a first file obtaining module 1110, a second file obtaining module 1120, and a first file sending module 1130.

The first file obtaining module 1110 is configured to obtain a file to be processed, where the file to be processed includes a file in at least one file format.

The second file obtaining module 1120 is configured to convert the file to be processed into a specified file format, so as to obtain the file to be transmitted.

The first file sending module 1130 is configured to send file information of a file to be transmitted to the second server, so that the third server obtains the file to be transmitted from the first server based on the obtained path information, where the obtained path information is determined by the third server based on the file information obtained from the second server.

Fig. 12 schematically shows a structural diagram of a document transportation apparatus according to another embodiment of the present disclosure.

As shown in fig. 12, the file transmission apparatus 1200 may include a first file receiving module 1210, a first file allocating module 1220, and a scheduling module 1230.

The first file receiving module 1210 is configured to receive file information of a file to be transmitted from a first server and receive third server information from a third server.

The first file allocation module 1220 is configured to allocate a file to be transmitted to at least one third server node based on the file information and the third server-side information, so as to generate a scheduling task.

The scheduling module 1230 is configured to add the scheduling task to the task scheduling queue, so that the third server determines to obtain the path information based on the task scheduling queue, and obtains the file to be transmitted from the first server based on the obtained path information.

Fig. 13 schematically shows a structural diagram of a document transportation apparatus according to another embodiment of the present disclosure.

As shown in fig. 13, the file transfer apparatus 1300 includes: a task responding module 1310, an acquisition path information determining module 1320, and a third file acquiring module 1330.

The task responding module 1310 is configured to determine file information of a file to be transmitted in response to a scheduling task from a task scheduling queue of the second server.

The acquisition path information determining module 1320 is configured to determine acquisition path information of a file to be transmitted based on the file information.

The third file obtaining module 1330 is configured to obtain the file to be transmitted from the first server based on the obtaining path information, so as to execute an instruction generated based on the file to be transmitted.

Fig. 14 schematically shows a structural diagram of a file transfer system according to an embodiment of the present disclosure.

As shown in fig. 14, the file transfer system 1400 includes: a first server side 1410, a second server side 1420 and a third server side 1430 connected to each other, the third server side comprising at least one third server node.

The file transfer system shown in fig. 14 may implement the high concurrency and high availability import of files into the ORACLE database. The solid arrows in fig. 14 may characterize the data flow induced by the server-side data change, and the dashed arrows may characterize the data synchronization between the server sides.

For example, the first server side 1410, the second server side 1420, and the third server side 1430 are configured as follows.

The first server 1410 converts the acquired file to be processed into a specified file format to obtain a file to be transmitted, and sends file information of the file to be transmitted to the second server 1420, where the file to be processed includes at least one file format.

The second server 1420 allocates the pending file to the at least one third server node based on the file information and the third server information, which characterizes a service status of the at least one third server node, to generate a task scheduling queue.

The third server 1430 determines file information in response to a scheduling task of the task scheduling queue, and determines acquisition path information of the file to be transmitted based on the file information, so as to execute an instruction generated based on the file to be transmitted after acquiring the file to be transmitted based on the acquisition path information.

The file transmission method executed by the file transmission system may refer to a relevant part of the file transmission method executed by the first server, the second server and/or the third server, and is not described herein again.

It should be noted that the implementation, solved technical problems, implemented functions, and achieved technical effects of each module and the like in the apparatus part embodiment are respectively the same as or similar to the implementation, solved technical problems, implemented functions, and achieved technical effects of each corresponding step in the method part embodiment, and are not described in detail herein.

Any of the modules according to embodiments of the present disclosure, or at least part of the functionality of any of them, may be implemented in one module. Any one or more of the modules according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules according to the embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging the circuit, or in any one of three implementations, or in any suitable combination of any of the software, hardware, and firmware. Alternatively, one or more of the modules according to embodiments of the disclosure may be implemented at least partly as computer program modules which, when executed, may perform corresponding functions. For example, any plurality of the first file acquiring module 1110, the second file acquiring module 1120, and the first file transmitting module 1130 may be combined or implemented separately, implemented in the form of hardware or software, or the like.

Another aspect of the present disclosure provides an electronic device.

FIG. 15 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure. The electronic device shown in fig. 15 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 15, an electronic device 1500 according to an embodiment of the present disclosure includes a processor 1501 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1502 or a program loaded from a storage section 1508 into a Random Access Memory (RAM) 1503. Processor 1501 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset(s) and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), and so forth. The processor 1501 may also include on-board memory for caching purposes. Processor 1501 may include a single processing unit or multiple processing units for performing different acts of a method flow in accordance with embodiments of the present disclosure.

In the RAM1503, various programs and data necessary for the operation of the electronic apparatus 1500 are stored. The processor 1501, the ROM 1502, and the RAM1503 are communicatively connected to each other by a bus 1504. The processor 1501 executes various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 1502 and/or RAM 1503. Note that the program may also be stored in one or more memories other than the ROM 1502 and the RAM 1503. The processor 1501 may also execute various operations of the method flows according to embodiments of the present disclosure by executing programs stored in one or more memories.

According to an embodiment of the present disclosure, electronic device 1500 may also include input/output (I/O) interface 1505, input/output (I/O) interface 1505 also being connected to bus 1504. The electronic device 1500 may also include one or more of the following components connected to the I/O interface 1505: an input portion 1506 including a keyboard, a mouse, and the like; an output portion 1507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1508 including a hard disk and the like; and a communication section 1509 including a network interface card such as a LAN card, a modem, or the like. The communication section 1509 performs communication processing via a network such as the internet. A drive 1510 is also connected to the I/O interface 1505 as needed. A removable medium 1511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1510 as necessary, so that a computer program read out therefrom is mounted into the storage section 1508 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1509, and/or installed from the removable medium 1511. The computer program, when executed by the processor 1501, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 1502 and/or RAM1503 described above and/or one or more memories other than the ROM 1502 and RAM 1503.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims (19)

1. A file transmission method performed by a first server, where the first server is connected to a second server and a third server, respectively, the method comprising:

acquiring a file to be processed, wherein the file to be processed comprises at least one file with a file format;

converting the file to be processed into a specified file format to obtain a file to be transmitted; and

and sending the file information of the file to be transmitted to a second server so that a third server can acquire the file to be transmitted from the first server based on acquisition path information, wherein the acquisition path information is determined by the third server based on the file information acquired from the second server.

2. The method of claim 1, wherein the first server side comprises at least one first server node;

the file information includes acquisition path information and at least one of: the file management system comprises file identification information, file size information and file state information, wherein the acquired path information comprises first server node address information and file path information.

3. The method of claim 1, wherein the converting the file to be processed into a specified file format comprises:

and converting the information of each processing unit into the information of the specified format file according to the grade of the processing unit of the file to be processed from high to low, wherein the processing unit of high grade comprises at least one processing unit of low grade.

4. The method of claim 1, further comprising: and after the file to be processed is converted into the specified file format, compressing the file to be processed in the specified file format.

5. The method of claim 1, further comprising: and after the file to be processed is obtained, backing up the file to be processed.

6. A file transmission method executed by a second server, where the second server is connected to a first server and a third server respectively, and the third server includes at least one third server node, the method comprising:

receiving file information of a file to be transmitted from the first server side and receiving third server side information from the third server side; and

distributing a file to be transmitted to at least one third server node based on the file information and the third server side information to generate a scheduling task; and

and adding the scheduling task into a task scheduling queue so that the third server side determines to acquire path information based on the task scheduling queue, and acquiring the file to be transmitted from the first server side based on the acquired path information.

7. The method of claim 6, further comprising:

and updating the third server side information of the third server side in response to the registration request and/or the logout request from the third server side.

8. The method of claim 6, wherein the allocating the file to be transmitted to at least one of the third server nodes based on the file information and the third server-side information comprises:

determining a load balancing policy based on the file information; and

and distributing the file to be transmitted to at least one third server node based on the load balancing strategy and the third server side information.

9. The method of claim 8, wherein the load balancing policy comprises a compressed file number balancing policy and a file size balancing policy.

10. A file transmission method executed by a third server, where the third server is connected to a first server and a second server respectively, and the third server includes at least one third server node, where the method includes: for each of the third server nodes it is,

determining file information of a file to be transmitted in response to a scheduling task from a task scheduling queue of a second server;

determining the obtaining path information of the file to be transmitted based on the file information;

and acquiring the file to be transmitted from the first server side based on the acquisition path information so as to execute an instruction generated based on the file to be transmitted.

11. The method of claim 10, wherein the file to be transmitted is a compressed file to be processed;

the obtaining the file to be transmitted from the first server based on the obtaining path information to execute the instruction generated based on the file to be transmitted includes:

acquiring and decompressing a compressed file to be processed;

generating a to-be-executed instruction based on the to-be-processed file; and

and executing the instruction to be executed.

12. The method of claim 11, further comprising: and after the instruction to be executed is executed, sending scheduling task state updating information to the second server side.

13. The method of claim 10, further comprising:

updating third server-side information, wherein the third server-side information represents the service state of at least one third server node;

and/or

After a third server node meets the service providing condition, sending a registration request to the second server end to register the address of the third server node;

and/or

And after the third server node meets the service stopping condition, sending a logout request to the second server side to logout the address of the third server node.

14. A file transfer method performed by a file transfer system, the file transfer system comprising a first server side, a second server side and a third server side connected to each other, the third server side comprising at least one third server node, the method comprising:

the first server side converts the acquired file to be processed into an appointed file format to obtain a file to be transmitted, and transmits file information of the file to be transmitted to the second server side, wherein the file to be processed comprises at least one file with a file format;

the second server side distributes a file to be transmitted to at least one third server node based on the file information and third server side information to generate a task scheduling queue, wherein the third server side information represents the service state of the at least one third server node; and

and the third server side responds to the scheduling tasks of the task scheduling queue, determines the file information of the file to be transmitted, and determines the obtaining path information of the file to be transmitted based on the file information so as to obtain the file to be transmitted based on the obtaining path information.

15. A file transfer device comprising:

the file processing device comprises a first file acquisition module, a second file acquisition module and a file processing module, wherein the first file acquisition module is used for acquiring files to be processed, and the files to be processed comprise files in at least one file format;

the second file acquisition module is used for converting the file to be processed into a specified file format to obtain a file to be transmitted; and

the first file sending module is configured to send file information of the file to be transmitted to a second server, so that the third server obtains the file to be transmitted from the first server based on obtained path information, where the obtained path information is determined by the third server based on the file information obtained from the second server.

16. A file transfer device comprising:

the first file receiving module is used for receiving file information of a file to be transmitted from a first server and receiving third server information from a third server;

the first file allocation module is used for allocating a file to be transmitted to at least one third server node based on the file information and the third server side information so as to generate a scheduling task; and

and the scheduling module is used for adding the scheduling task into a task scheduling queue so that the third server side determines to acquire path information based on the task scheduling queue, and acquires the file to be transmitted from the first server side based on the acquired path information.

17. A file transfer device comprising:

the task corresponding module is used for responding to a scheduling task of a task scheduling queue from the second server and determining file information of a file to be transmitted;

the acquisition path information determining module is used for determining the acquisition path information of the file to be transmitted based on the file information; and

and the third file acquisition module is used for acquiring the file to be transmitted from the first server based on the acquisition path information so as to execute an instruction generated based on the file to be transmitted.

18. A file transfer system comprising: a first server side, a second server side and a third server side connected to each other, the third server side comprising at least one third server node, wherein the first server side, the second server side and the third server side are configured to:

the first server side converts the acquired file to be processed into an appointed file format to obtain a file to be transmitted, and transmits file information of the file to be transmitted to the second server side, wherein the file to be processed comprises at least one file format;

the second server side distributes a file to be processed to at least one third server node based on the file information and third server side information to generate a task scheduling queue, wherein the third server side information represents the service state of the at least one third server node; and

and the third server side responds to the scheduling tasks of the task scheduling queue, determines file information, and determines the obtaining path information of the file to be transmitted based on the file information, so that after the file to be transmitted is obtained based on the obtaining path information, an instruction generated based on the file to be transmitted is executed.

19. An electronic device, comprising:

at least one processor;

a storage device for storing executable instructions which, when executed by the processor, implement a method according to any one of claims 1 to 14.

Images