CN106603729A - Distributed-file-system multi-client synchronization method and system thereof - Google Patents

Abstract

The invention discloses a multi-client synchronization method of a distributed file system, which includes: judging whether the number of clients is 1, and if so, granting exclusive rights to the above-mentioned clients; The client assigns the above-mentioned write permission; the client with the above-mentioned write permission completes writing to the file, and then updates the corresponding metadata; judges whether there are other clients with read-write permission to operate the above-mentioned file, if not, end the operation, if so , the updated metadata is sent to the corresponding client. It can be seen that the present invention manages the read and write permissions of clients sequentially; at the same time, delay synchronization is adopted for small files, and the previous metadata is overwritten after aggregation, and common synchronization strategies are used for management after aggregated files are placed on the disk, which improves the management efficiency of multiple clients. The invention also discloses a distributed file system multi-client synchronization system, which has the same technical effect as the above method.

Description

A distributed file system multi-client synchronization method and system

technical field

The present invention relates, in particular, to a multi-client synchronization method and system of a distributed file system.

Background technique

For a distributed file system, since its clients are distributed on different physical machines, the operations of different clients on file system metadata and data are mutually invisible. So there is a problem that one client's modification of the file system is not visible to another client, that is, there is a problem of cache consistency among multiple clients. In the prior art, two methods are adopted for data synchronization: 1. Each client can operate files online, but due to the simultaneous operation of different clients, multiple data will be overwritten at the same time during data synchronization. At this time, the synchronized The result is that only the last overwritten file takes effect, or the system reports an error; 2. The client operates the file offline and then synchronizes to the server. At this time, the same file will synchronize multiple results. When other users access it again, it is impossible to determine which file is the latest file. .

To sum up, how to improve the management efficiency of multi-client synchronization is a technical problem to be solved by those skilled in the art.

Contents of the invention

In view of this, the object of the present invention is to provide a distributed file system multi-client synchronization method and system, which can significantly improve the management efficiency of multi-client synchronization. The specific plan is as follows:

A distributed file system multi-client synchronization method, comprising:

Step S1: determine whether the number of clients is 1, if so, give the client an exclusive right, if not, proceed to step S2;

Step S2: according to the application sequence of write permission, assign the write permission to the corresponding client;

Step S3: The client with the write permission finishes writing the file, and then updates the corresponding metadata;

Step S4: Determine whether there are other clients with read and write permissions to operate the file, if not, end the operation, and if so, send the updated metadata to the corresponding client.

Preferably, giving the client the exclusive authority includes:

Obtain the operation permission request sent by the client;

Reply to the operation authority request of the client, so as to assign the corresponding operation authority to the client;

Obtaining the permission release request sent by the client;

The reply to the client releases the permission result.

Preferably, the operation permissions include read permissions, write permissions and client local cache permissions.

Preferably, according to the application sequence of the write permission, assigning the write permission to the corresponding client further includes:

According to the application sequence of the read permission, the read permission is assigned to the corresponding client.

Preferably, according to the application sequence of the write permission, assigning the write permission to the corresponding client further includes:

Remove the local cache permission of the client with the write permission.

Preferably, the client with the write permission finishes writing the file, then updating the corresponding metadata includes:

Determine whether the size of the file requested by the client is smaller than the preset value, if so, use the aggregation mode to write,

Wherein, the aggregation mode is to write data edited by a client with write permission to a local cache, and if it is determined that the size of the local cache is greater than or equal to a threshold, then update corresponding metadata.

The invention also discloses a distributed file system multi-client synchronization system, including:

The client quantity judging module is used to judge whether the client quantity is 1;

The single-client authority allocation module is used to determine that the number of clients is 1, and then give the client exclusive authority;

The multi-client authority allocation module is used to assign the write authority to the corresponding client according to the order of application of the write authority when it is determined that the number of clients is not 1;

A metadata update module, configured to update the corresponding metadata when the client with the write permission finishes writing the file;

The client data update module is used to judge whether there are other clients with read and write permissions to operate the file, if not, end the operation, and if so, send the updated metadata to the corresponding client.

The present invention discloses a multi-client synchronization method of a distributed file system, including: step S1: judging whether the number of clients is 1, if so, giving the client an exclusive right, if not, proceeding to step S2; step S2 : According to the application sequence of write permission, assign the write permission to the corresponding client; step S3: the client with the write permission finishes writing the file, then update the corresponding metadata; step S4: judge whether there are other A client with read and write permissions operates the file, if not, the operation ends, and if yes, the updated metadata is sent to the corresponding client. It can be seen that the present invention adopts the unified management of the client cache, assigns read and write permissions in sequence according to the application order, ensures data consistency, and realizes concurrent reading and writing of multiple clients; at the same time, for the scenario of small file aggregation, this solution adopts delayed synchronization, and the last time of aggregation is After the data is placed on the disk, the previous metadata will be overwritten. After the aggregation file is placed on the disk, the normal synchronization strategy will be used for management. The logic of the method is simple and easy to implement, and at the same time realizes the concurrent operation of the distributed file system. Secondly, for the aggregated files, the multi-client synchronization is realized on the basis of ensuring the aggregation efficiency, thereby improving the management efficiency of the multi-client synchronization.

The present invention also discloses a distributed file system multi-client synchronization system, which has the same technical effect as the above-mentioned method and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

Fig. 1 is a schematic flow diagram of a distributed file system multi-client synchronization method also disclosed by the present invention;

FIG. 2 is a schematic structural diagram of a specific distributed file system multi-client synchronization system also disclosed in the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The embodiment of the present invention discloses a distributed file system multi-client synchronization method, as shown in Figure 1, including:

Step S1: Determine whether the number of clients is 1, if yes, give the client an exclusive right, if not, proceed to step S2.

When the number of clients is 1, that is, there is only a single client in the distributed system, so there is no data synchronization delay problem of other clients, and exclusive rights can be given to this client.

When the number of clients is not 1, but the number of clients is greater than or equal to 1, that is, the distributed system is a multi-client system. At this time, there is a problem of data synchronization delays accessed by multiple clients, and corresponding operations need to be performed.

In a specific embodiment, granting the exclusive authority to the single client includes: obtaining the operation authority request sent by the single client; replying to the operation authority request of the single client, so as to assign the corresponding operation authority to the single client; After the single client completes the editing of the corresponding file, obtain the permission release request sent by the single client; reply the permission release result to the single client.

It should be noted that the operation permission usually includes read permission, write permission of the file to be edited, and local cache permission of the client. The client reads the file to be edited, and needs to obtain the read permission of the file to be edited, the client edits the file to be edited, and the write permission of the file to be edited, of course, because the invention of the present invention lies in the online Synchronous update, when there is only a single client in the distributed system, there is no data synchronization delay problem for other clients, so the single client can be given local cache permissions.

It should be further explained that the present invention is also applicable to situations where the write permission in some existing systems includes the read permission.

Step S2: Assign the above-mentioned write permission to the corresponding client according to the application sequence of the write permission.

Since the read permission obtained by the client is only for the query and read permission of the corresponding file, it does not have the permission to edit the file and will not change the content of the file. Therefore, it is possible to assign more than one client to the same file. Read permission, that is, the same file supports being accessed by multiple clients at the same time, that is, the same file can be continuously accessed by more than one client within the same period of time.

Of course, in order to better manage each client, so that only one client can access and edit the same file at the same time, the above-mentioned read permission can also be assigned to the corresponding client according to the order of application for read permission.

In a specific implementation scheme, the metadata server in the distributed system obtains the write permission applied by client A, and after confirming that client A is a legal client, assigns the write permission and cache permission of the corresponding file to client A , and client A edits the file. At this time, the metadata server obtains the read and write permission requested by client B, and the metadata server assigns the read and write permission to client B after withdrawing the write permission from client A. Of course, the metadata server can also assign the read permission to the client B before withdrawing the write permission from the client A.

Since the inventive point of the present invention is to update the distributed system metadata online synchronously, when the distributed system includes multiple clients, that is, more than 1 client, there will be a problem of data synchronization delay for multiple clients. Therefore, in order to make the corresponding data If it is updated in real time on a distributed system, the local cache permission of the client with the above-mentioned write permission will be removed.

Step S3: The client with the above-mentioned write permission completes writing to the file, and then updates the corresponding metadata.

In order to improve the processing efficiency of small files, the present invention adopts an aggregation mode for small files. The small file here needs to be compared with the preset value in the metadata server in the distributed system. If it is smaller than the preset value, it can be determined that the file is a small file, and the aggregation mode is adopted accordingly.

The specific implementation method is to judge whether the size of the file written by the client application is smaller than the preset value, and if so, write it in the aggregation mode. The above aggregation mode is to write the data edited by the client with write permission to the local Cache, if it is judged that the size of the local cache is greater than or equal to the threshold, update the corresponding metadata.

If the data is not off the disk, the client will keep writing the data into the aggregation cache until the amount of data in the aggregation cache is greater than or equal to the threshold, then update the data in the aggregation cache to the metadata server.

It should be noted that the aggregation cache includes a local cache and a specific network cache. When the authority and the cache are specific network caches, the storage and processing speed of the network cache should be faster than that of the distributed system.

Step S4: Determine whether there are other clients with read and write permissions to operate the above file, if not, end the operation, and if so, send the updated metadata to the corresponding client.

If no other client accesses and edits the updated file, the file will be saved and the previous file will be overwritten. Of course, the historical compilation content of the file can also be saved in the order of revision.

If there are other clients accessing it, that is, there are other clients with read permission, the updated file is sent to the other client with read permission, and the local metadata of the corresponding client is updated.

The present invention also discloses a multi-client synchronization system of a distributed file system, as shown in FIG. Module 14 and client data updating module 15, wherein:

The client quantity judging module 11 is used to judge whether the client quantity is 1.

When the number of clients is 1, that is, there is only a single client in the distributed system, so there is no data synchronization delay problem of other clients, and the client can be given exclusive authority by the single-client authority allocation module 12 .

When the number of clients is not 1, and the number of clients is greater than or equal to 1, that is, the distributed system is a multi-client system. At this time, there is a problem of data synchronization delay for multiple clients to access, and a multi-client authority allocation module 13 is required. Do the corresponding operation.

It should be further explained that the present invention is also applicable to situations where the write permission in some existing systems includes the read permission.

The single-client authority allocation module 12 is used to grant exclusive authority to the above-mentioned client when it is determined that the number of the above-mentioned clients is 1.

In a specific embodiment, granting the exclusive authority to the single client includes: obtaining the operation authority request sent by the single client; replying to the operation authority request of the single client, so as to assign the corresponding operation authority to the single client; After the single client completes the editing of the corresponding file, obtain the permission release request sent by the single client; reply the permission release result to the single client.

It should be noted that the operation permission usually includes read permission, write permission of the file to be edited, and local cache permission of the client. The client reads the file to be edited, and needs to obtain the read permission of the file to be edited, the client edits the file to be edited, and the write permission of the file to be edited, of course, because the invention of the present invention lies in the online Synchronous update, when there is only a single client in the distributed system, there is no data synchronization delay problem for other clients, so the single client can be given local cache permissions.

The multi-client authority allocation module 13 is used to allocate the above-mentioned write authority to the corresponding clients according to the order of application for write authority when it is determined that the number of the above-mentioned clients is not 1.

Since the read permission obtained by the client is only for the query and read permission of the corresponding file, it does not have the permission to edit the file and will not change the content of the file. Therefore, it is possible to assign more than one client to the same file. Read permission, that is, the same file supports being accessed by multiple clients at the same time, that is, the same file can be continuously accessed by more than one client within the same period of time.

Of course, in order to better manage each client, so that only one client can access and edit the same file at the same time, the above-mentioned read permission can also be assigned to the corresponding client according to the order of application for read permission.

In a specific implementation scheme, the metadata server in the distributed system obtains the write permission applied by client A, and after confirming that client A is a legal client, assigns the write permission and cache permission of the corresponding file to client A , and client A edits the file. At this time, the metadata server obtains the read and write permission requested by client B, and the metadata server assigns the read and write permission to client B after withdrawing the write permission from client A. Of course, the metadata server can also assign the read permission to the client B before withdrawing the write permission from the client A.

Since the inventive point of the present invention is to update the distributed system metadata online synchronously, when the distributed system includes multiple clients, that is, more than 1 client, there will be a problem of data synchronization delay for multiple clients. Therefore, in order to make the corresponding data If it is updated in real time on a distributed system, the local cache permission of the client with the above-mentioned write permission will be removed.

The metadata update module 14 is configured to update the corresponding metadata when the client with the above-mentioned write permission completes writing to the file.

In order to improve the processing efficiency of small files, the present invention adopts an aggregation mode for small files. The small file here needs to be compared with the preset value in the metadata server in the distributed system. If it is smaller than the preset value, it can be determined that the file is a small file, and the aggregation mode is adopted accordingly.

The specific implementation method is to judge whether the size of the file written by the client application is smaller than the preset value, and if so, write it in the aggregation mode. The above aggregation mode is to write the data edited by the client with write permission to the local Cache, if it is judged that the size of the local cache is greater than or equal to the threshold, update the corresponding metadata.

If the data is not off the disk, the client will keep writing the data into the aggregation cache until the amount of data in the aggregation cache is greater than or equal to the threshold, then update the data in the aggregation cache to the metadata server.

It should be noted that the aggregation cache includes a local cache and a specific network cache. When the authority and the cache are specific network caches, the storage and processing speed of the network cache should be faster than that of the distributed system.

The client data update module 15 is used to judge whether there are other clients with read and write permissions to operate the above-mentioned files, if not, then end the operation, and if so, send the updated metadata to the corresponding client.

If no other client accesses and edits the updated file, the file will be saved and the previous file will be overwritten. Of course, the historical compilation content of the file can also be saved in the order of revision.

If there are other clients accessing it, that is, there are other clients with read permission, the updated file is sent to the other client with read permission, and the local metadata of the corresponding client is updated.

Finally, it should also be noted that in this text, relational terms such as first and second etc. are only used to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations, any such actual relationship or order exists. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

A kind of distributed file system multi-client synchronization method and system provided by the present invention has been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for To help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary, the content of this specification It should not be construed as limiting the invention.

Claims (7)

1. A distributed file system multi-client synchronization method, characterized in that, comprising: Step S1: determine whether the number of clients is 1, if so, give the client an exclusive right, if not, proceed to step S2; Step S2: according to the application sequence of write permission, assign the write permission to the corresponding client; Step S3: The client with the write permission finishes writing the file, and then updates the corresponding metadata; Step S4: Determine whether there are other clients with read and write permissions to operate the file, if not, end the operation, and if so, send the updated metadata to the corresponding client. 2. The distributed file system multi-client synchronization method according to claim 1, wherein giving the client the exclusive authority comprises: Obtain the operation permission request sent by the client; Reply to the operation authority request of the client, so as to assign the corresponding operation authority to the client; Obtaining the permission release request sent by the client; The reply to the client releases the permission result. 3. The multi-client synchronization method of a distributed file system according to claim 2, wherein the operation permissions include read permissions, write permissions, and client local cache permissions. 4. The distributed file system multi-client synchronization method according to claim 1, wherein, according to the application sequence of the write permission, assigning the write permission to the corresponding client also includes: According to the application sequence of the read permission, the read permission is assigned to the corresponding client. 5. The distributed file system multi-client synchronization method according to claim 4, wherein, according to the application sequence of the write permission, assigning the write permission to the corresponding client also includes: Remove the local cache permission of the client with the write permission. 6. The distributed file system multi-client synchronization method according to claim 1, wherein the client with the write authority finishes writing the file, and then updating the corresponding metadata includes: Determine whether the size of the file requested by the client is smaller than the preset value, if so, use the aggregation mode to write, Wherein, the aggregation mode is to write data edited by a client with write permission to a local cache, and if it is determined that the size of the local cache is greater than or equal to a threshold, then update corresponding metadata. 7. A distributed file system multi-client synchronization system, characterized in that it comprises: The client quantity judging module is used to judge whether the client quantity is 1; The single-client authority allocation module is used to determine that the number of clients is 1, and then give the client exclusive authority; The multi-client authority allocation module is used to assign the write authority to the corresponding client according to the order of application of the write authority when it is determined that the number of clients is not 1; A metadata update module, configured to update the corresponding metadata when the client with the write permission finishes writing the file; The client data update module is used to judge whether there are other clients with read and write permissions to operate the file, if not, end the operation, and if so, send the updated metadata to the corresponding client.